Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1241973

Summary:	neutron-ovs-cleanup fails if there is a bond on br-ex
Product:	Red Hat OpenStack	Reporter:	Brad P. Crochet <brad>
Component:	openstack-neutron	Assignee:	Terry Wilson <twilson>
Status:	CLOSED ERRATA	QA Contact:	Ofer Blaut <oblaut>
Severity:	urgent	Docs Contact:
Priority:	high
Version:	7.0 (Kilo)	CC:	amuller, chandra.s.rangavajjula, chrisw, hbrock, ihrachys, jschluet, mburns, nyechiel, oblaut, rrosa, tfreger, yeylon
Target Milestone:	ga
Target Release:	7.0 (Kilo)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-neutron-2015.1.0-12.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-08-05 13:29:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brad P. Crochet 2015-07-10 14:17:49 UTC

having any sort of bond on the br-ex bridge causes neutron-ovs-cleanup to fail

Comment 5 Dan Prince 2015-07-13 19:56:54 UTC

So I actually wrote my initial fix against kilo. What I did there (functionally seemed to work fine) was:

diff --git a/neutron/agent/common/ovs_lib.py b/neutron/agent/common/ovs_lib.py
index 7b1d400..0b53f5b 100644
--- a/neutron/agent/common/ovs_lib.py
+++ b/neutron/agent/common/ovs_lib.py
@@ -313,6 +313,10 @@ class OVSBridge(BaseOVS):
         edge_ports = []
         port_names = self.get_port_name_list()
         for name in port_names:
+            interfaces = self.db_get_val("Port", name, "interfaces",
+                                         check_error=True)
+            if not interfaces:
+                continue
             external_ids = self.db_get_val("Interface", name, "external_ids",
                                            check_error=True)
             ofport = self.db_get_val("Interface", name, "ofport",


----

The upstream Liberty fix here https://review.openstack.org/#/c/200223/ would also require upstream Neutron Git commit 4dc68ea88bf4f07b13253bf9eeedffe22b1f8013 (Read vif port information in bulk).

Comment 6 Dan Prince 2015-07-13 20:02:00 UTC

Sorry, The correct patch for kilo would be:

diff --git a/neutron/agent/common/ovs_lib.py b/neutron/agent/common/ovs_lib.py
index 7b1d400..1f428bd 100644
--- a/neutron/agent/common/ovs_lib.py
+++ b/neutron/agent/common/ovs_lib.py
@@ -313,6 +313,11 @@ class OVSBridge(BaseOVS):
         edge_ports = []
         port_names = self.get_port_name_list()
         for name in port_names:
+            interfaces = self.db_get_val("Port", name, "interfaces",
+                                         check_error=True)
+            # skip ports w/ interfaces (bonds)
+            if interfaces:
+                continue
             external_ids = self.db_get_val("Interface", name, "external_ids",
                                            check_error=True)
             ofport = self.db_get_val("Interface", name, "ofport",

Comment 7 Assaf Muller 2015-07-13 20:07:15 UTC

We'll definitely want to avoid backporting the 'vif port information in bulk' patch because that's a large patch with the potential for a lot of side effects. I'm already aware of one regression, so we'd need to backport the fix to that as well, and that's 3 patches now. I think we're better off with tailoring a simpler fix specially for Kilo and carrying that until the Liberty release.

Comment 9 Ofer Blaut 2015-07-19 09:31:05 UTC

tested openstack-neutron-2015.1.0-12.el7ost.noarch


I still have errors , but don't get can;t see  neutron-ovs-cleanup  failed on pcs status


   ovs_version: "2.3.1-git3282e51"
[root@overcloud-controller-0 heat-admin]# pcs status | grep -C 2 ovs
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]



cat /var/log/neutron/ovs-cleanup.log


2015-07-19 05:04:01.659 18360 ERROR neutron.agent.ovsdb.impl_vsctl [-] Unable to execute ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'bond1'].
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Traceback (most recent call last):
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl   File "/usr/lib/python2.7/site-packages/neutron/agent/ovsdb/impl_vsctl.py", line 63, in run_vsctl
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl     log_fail_as_error=False).rstrip()
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 137, in execute
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl     raise RuntimeError(m)
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl RuntimeError: 
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'bond1']
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Exit code: 1
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Stdin: 
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Stdout: 
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl Stderr: ovs-vsctl: no row "bond1" in table Interface
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl 
2015-07-19 05:04:01.659 18360 TRACE neutron.agent.ovsdb.impl_vsctl

Comment 10 Terry Wilson 2015-07-20 16:33:11 UTC

The error condition is still logged, but shouldn't cause a failure. I can amend the patch to not log the error either, but if it is just a logging issue I wouldn't think we should fail a blocker bug.

Ofer: is this the case? Just showing up in the log?

Comment 12 Ofer Blaut 2015-07-20 18:19:03 UTC

The ovs-cleanup seems to be running by pacemaker, i can't tell if it worked or not

Since i don't have old ports to be cleaned yet

Comment 13 Ofer Blaut 2015-07-21 11:03:01 UTC

seems to be working ( tested on computes ), do nothing on controlers



Stdout:
Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 15:00:35.017 20447 INFO neutron.cmd.ovs_cleanup [-] Deleting port: qvo4814d9c2-9e
2015-07-20 15:00:35.017 20447 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'qvoa9fb8e08-51'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2015-07-20 15:00:35.024 20447 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', '-o', 'link', 'show', 'qvoa9fb8e08-51']
Exit code: 0
Stdin:
Stdout: 23: qvoa9fb8e08-51: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\    link/ether 1a:af:a8:1e:49:ca brd ff:ff:ff:ff:ff:ff

Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 15:00:35.024 20447 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'link', 'delete', 'qvoa9fb8e08-51'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2015-07-20 15:00:35.043 20447 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', 'link', 'delete', u'qvoa9fb8e08-51']
Exit code: 0
Stdin:
Stdout:
Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 15:00:35.044 20447 INFO neutron.cmd.ovs_cleanup [-] Deleting port: qvoa9fb8e08-51
2015-07-20 15:00:35.044 20447 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'qvoccd9bf44-7a'] create_process /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2015-07-20 15:00:35.048 20447 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', '-o', 'link', 'show', 'qvoccd9bf44-7a']
Exit code: 0
Stdin:
Stdout: 20: qvoccd9bf44-7a: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000\    link/ether fe:3e:fe:ba:63:a6 brd ff:ff:ff:ff:ff:ff

Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 15:00:35.049 20447 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'link', 'delete', 'qvoccd9bf44-7a'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2015-07-20 15:00:35.075 20447 DEBUG neutron.agent.linux.utils [-]
Command: ['ip', 'link', 'delete', u'qvoccd9bf44-7a']
Exit code: 0
Stdin:
Stdout:
Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 15:00:35.076 20447 INFO neutron.cmd.ovs_cleanup [-] Deleting port: qvoccd9bf44-7a
2015-07-20 15:00:35.076 20447 INFO neutron.cmd.ovs_cleanup [-] OVS cleanup completed successfully
2015-07-20 15:00:35.078 20447 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=20454



Does nothing on controllers

Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 14:58:38.131 11457 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'vlan201'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2015-07-20 14:58:38.136 11457 DEBUG neutron.agent.linux.utils [-]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'vlan201']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[]]]],"headings":["external_ids"]}

Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 14:58:38.136 11457 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'vlan202'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2015-07-20 14:58:38.141 11457 DEBUG neutron.agent.linux.utils [-]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'vlan202']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[]]]],"headings":["external_ids"]}

Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 14:58:38.141 11457 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', 'vlan203'] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2015-07-20 14:58:38.146 11457 DEBUG neutron.agent.linux.utils [-]
Command: ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=external_ids', 'list', 'Interface', u'vlan203']
Exit code: 0
Stdin:
Stdout: {"data":[[["map",[]]]],"headings":["external_ids"]}

Stderr:  execute /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:134
2015-07-20 14:58:38.146 11457 INFO neutron.cmd.ovs_cleanup [-] OVS cleanup completed successfully
2015-07-20 14:58:38.148 11457 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=11615

Comment 15 errata-xmlrpc 2015-08-05 13:29:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1548