Description of problem: A customer is mapping multiple flat provider networks on a single bonded interface, each network with its separate bridge. According to our manual [1] this is not the way to go: ~~~ If there are multiple flat provider networks, each of them should have separate physical interface and bridge to connect them to external network. ~~~ However, the setup is expected to work. Upon reboot of the Compute node (running RHOS 10, ovs 2.5) the flows are being re-created only on __one bridge__ (out of many). The customer has reported that it's a random bridge: after a few reboots of the Compute node, the bridge __with__ flows changes to another one. [1] https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/networking-guide/#using_flat_provider_networks Version-Release number of selected component (if applicable): It happens in OSP10 (post upgrade from working OSP9, as well as on a fresh OSP10 installment). We tried downgrading openvswitch to 2.4: it didn't work. We tried also downgrading ovs to 2.4 and booting kernel 3.10.0-327.36.2.el7: no luck. The customer has tried upgrading ovs to 2.6: no change. How reproducible: Setup a Compute node with multiple flat provider networks on a single bonded interface in OSP10. Actual results: There are no ovs flows on the bridges. There is no connectivity on the provider networks. Expected results: There are ovs flows on the bridges. Instances are able to ping on the provider network(s). Workaround that brings back the flows (but there's still __no__ ping on the provider network): 1) Delete all bridges (ovs-vsctl list-br | xargs -I% ovs-vsctl del-br %) 2) Restart network (systemctl restart network &) 3) Restart neutron-openvswitch-agent (systemctl restart neutron-openvswitch-agent) 4) Look at your marvellous flows (ovs-vsctl list-br | xargs -I% ovs-ofctl dump-flows %) Additional info: At some point during a troubleshooting session we managed to get back the connectivity. However, after a reboot it was lost once again. Since then, we haven't been able to resume connectivity back. The multiple flat provider networks on a single interface has been discussed here: http://post-office.corp.redhat.com/archives/rhos-tech/2017-January/msg00130.html
It's mapped on a bonded interface using VLANs and each VLANs is mapped to a flat network within the openvswitch-agent. This is working alright but when rebooting the compute node, for some reasons, neutron-openvswitch-agent will recreate the flows only on the first bridge and skip the other ones. We have a workaround for that which is to delete everything and restart everything ... neutron-openvswitch-agent will then recreate the required flows for all bridges.
We're going to need an SOS report from a controller and a compute, as well as the OSPd files that were used to deploy the setup.
Hey Ajo, Any additional info needed? --Irina
fixed verified rpm -qa |grep python-tinyrpc python-tinyrpc-0.5-3.el7ost.noarch [root@controller-0 ~]# rpm -qa |grep python-ryu- python-ryu-common-4.9-2.1.el7ost.noarch python-ryu-4.9-2.1.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1587