having any sort of bond on the br-ex bridge causes neutron-ovs-cleanup to fail
So I actually wrote my initial fix against kilo. What I did there (functionally seemed to work fine) was: diff --git a/neutron/agent/common/ovs_lib.py b/neutron/agent/common/ovs_lib.py index 7b1d400..0b53f5b 100644 --- a/neutron/agent/common/ovs_lib.py +++ b/neutron/agent/common/ovs_lib.py @@ -313,6 +313,10 @@ class OVSBridge(BaseOVS): edge_ports = [] port_names = self.get_port_name_list() for name in port_names: + interfaces = self.db_get_val("Port", name, "interfaces", + check_error=True) + if not interfaces: + continue external_ids = self.db_get_val("Interface", name, "external_ids", check_error=True) ofport = self.db_get_val("Interface", name, "ofport", ---- The upstream Liberty fix here https://review.openstack.org/#/c/200223/ would also require upstream Neutron Git commit 4dc68ea88bf4f07b13253bf9eeedffe22b1f8013 (Read vif port information in bulk).
Sorry, The correct patch for kilo would be: diff --git a/neutron/agent/common/ovs_lib.py b/neutron/agent/common/ovs_lib.py index 7b1d400..1f428bd 100644 --- a/neutron/agent/common/ovs_lib.py +++ b/neutron/agent/common/ovs_lib.py @@ -313,6 +313,11 @@ class OVSBridge(BaseOVS): edge_ports = [] port_names = self.get_port_name_list() for name in port_names: + interfaces = self.db_get_val("Port", name, "interfaces", + check_error=True) + # skip ports w/ interfaces (bonds) + if interfaces: + continue external_ids = self.db_get_val("Interface", name, "external_ids", check_error=True) ofport = self.db_get_val("Interface", name, "ofport",
We'll definitely want to avoid backporting the 'vif port information in bulk' patch because that's a large patch with the potential for a lot of side effects. I'm already aware of one regression, so we'd need to backport the fix to that as well, and that's 3 patches now. I think we're better off with tailoring a simpler fix specially for Kilo and carrying that until the Liberty release.
tested openstack-neutron-2015.1.0-12.el7ost.noarch I still have errors , but don't get can;t see neutron-ovs-cleanup failed on pcs status ovs_version: "2.3.1-git3282e51" [root@overcloud-controller-0 heat-admin]# pcs status | grep -C 2 ovs Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup] Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ] cat /var/log/neutron/ovs-cleanup.log 2015-07-19 05:04:01.659 18360 ERROR neutron.agent.ovsdb.impl_vsctl [-] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'bond1']. 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Traceback (most recent call last): 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl log_fail_as_error=False).rstrip() 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl raise RuntimeError(m) 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl RuntimeError: 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'bond1'] 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Exit code: 1 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Stdin: 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Stdout: 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Stderr: ovs-vsctl: no row "bond1" in table Interface 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl 2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl
The error condition is still logged, but shouldn't cause a failure. I can amend the patch to not log the error either, but if it is just a logging issue I wouldn't think we should fail a blocker bug. Ofer: is this the case? Just showing up in the log?
The ovs-cleanup seems to be running by pacemaker, i can't tell if it worked or not Since i don't have old ports to be cleaned yet
seems to be working ( tested on computes ), do nothing on controlers Stdout: Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 15:00:35.017 20447 INFO neutron.cmd.ovs_cleanup [-] Deleting port: qvo4814d9c2-9e 2015-07-20 15:00:35.017 20447 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'qvoa9fb8e08-51'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84 2015-07-20 15:00:35.024 20447 DEBUG neutron.agent.linux.utils [-] Command: ['ip', '-o', 'link', 'show', 'qvoa9fb8e08-51'] Exit code: 0 Stdin: Stdout: 23: qvoa9fb8e08-51: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\ link/ether 1a:af:a8:1e:49:ca brd ff:ff:ff:ff:ff:ff Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 15:00:35.024 20447 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'link', 'delete', 'qvoa9fb8e08-51'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100 2015-07-20 15:00:35.043 20447 DEBUG neutron.agent.linux.utils [-] Command: ['ip', 'link', 'delete', u'qvoa9fb8e08-51'] Exit code: 0 Stdin: Stdout: Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 15:00:35.044 20447 INFO neutron.cmd.ovs_cleanup [-] Deleting port: qvoa9fb8e08-51 2015-07-20 15:00:35.044 20447 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'qvoccd9bf44-7a'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84 2015-07-20 15:00:35.048 20447 DEBUG neutron.agent.linux.utils [-] Command: ['ip', '-o', 'link', 'show', 'qvoccd9bf44-7a'] Exit code: 0 Stdin: Stdout: 20: qvoccd9bf44-7a: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\ link/ether fe:3e:fe:ba:63:a6 brd ff:ff:ff:ff:ff:ff Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 15:00:35.049 20447 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'link', 'delete', 'qvoccd9bf44-7a'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100 2015-07-20 15:00:35.075 20447 DEBUG neutron.agent.linux.utils [-] Command: ['ip', 'link', 'delete', u'qvoccd9bf44-7a'] Exit code: 0 Stdin: Stdout: Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 15:00:35.076 20447 INFO neutron.cmd.ovs_cleanup [-] Deleting port: qvoccd9bf44-7a 2015-07-20 15:00:35.076 20447 INFO neutron.cmd.ovs_cleanup [-] OVS cleanup completed successfully 2015-07-20 15:00:35.078 20447 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=20454 Does nothing on controllers Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 14:58:38.131 11457 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'vlan201'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100 2015-07-20 14:58:38.136 11457 DEBUG neutron.agent.linux.utils [-] Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'vlan201'] Exit code: 0 Stdin: Stdout: {"data":[[["map",[]]]],"headings":["external_ids"]} Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 14:58:38.136 11457 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'vlan202'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100 2015-07-20 14:58:38.141 11457 DEBUG neutron.agent.linux.utils [-] Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'vlan202'] Exit code: 0 Stdin: Stdout: {"data":[[["map",[]]]],"headings":["external_ids"]} Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 14:58:38.141 11457 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'vlan203'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100 2015-07-20 14:58:38.146 11457 DEBUG neutron.agent.linux.utils [-] Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'vlan203'] Exit code: 0 Stdin: Stdout: {"data":[[["map",[]]]],"headings":["external_ids"]} Stderr: execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134 2015-07-20 14:58:38.146 11457 INFO neutron.cmd.ovs_cleanup [-] OVS cleanup completed successfully 2015-07-20 14:58:38.148 11457 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=11615
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2015:1548