Bug 990646 - Execution of ovs-cleanup-service fails
Execution of ovs-cleanup-service fails
Status: CLOSED CURRENTRELEASE
Product: RDO
Classification: Community
Component: openstack-quantum (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: RHOS Maint
Ofer Blaut
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-31 11:50 EDT by Derek Higgins
Modified: 2013-12-19 06:40 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-12-19 06:39:58 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
debug log from a failed ovs cleanup (9.63 KB, text/x-log)
2013-07-31 11:54 EDT, Derek Higgins
no flags Details

  None (edit)
Description Derek Higgins 2013-07-31 11:50:46 EDT
While installing with packstack the ovs-cleanup-service occasionally fails (about 4 times out of 100 runs in my tests), this in turn causes the quantum puppet module to report an error which in turn causes packstack to fail.

On RHEL 6.4 RDO packages

I can reproduce this looping through the following script (puppet manifests came from packstack which was run before this test)

puppet apply --modulepath=/usr/lib/python2.6/site-packages/packstack/puppet/modules/ ./manifests/192.168.122.171_quantum.pp
puppet apply --modulepath=/usr/lib/python2.6/site-packages/packstack/puppet/modules/ ./manifests/192.168.122.171_provision.pp
sed -i -e 's/debug =.*/debug = True/g' /etc/quantum/quantum.conf
su -s /bin/bash quantum -c '/usr/bin/quantum-ovs-cleanup --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini'

and then it fails, the traceback in /var/log/quantum/quantum-ovs-cleanup.log is

2013-07-31 15:16:05 CRITICAL [quantum]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'delete', 'tap89d51b72-76']
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: Operation not supported\n'
Traceback (most recent call last):
  File "/usr/bin/quantum-ovs-cleanup", line 26, in <module>
    main()
  File "/usr/lib/python2.6/site-packages/quantum/agent/ovs_cleanup_util.py", line 108, in main
    delete_quantum_ports(ports, conf.AGENT.root_helper)
  File "/usr/lib/python2.6/site-packages/quantum/agent/ovs_cleanup_util.py", line 72, in delete_quantum_ports
    device.link.delete()
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 203, in delete
    self._as_root('delete', self.name)
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 167, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 47, in _as_root
    namespace)
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/ip_lib.py", line 58, in _execute
    root_helper=root_helper)
  File "/usr/lib/python2.6/site-packages/quantum/agent/linux/utils.py", line 61, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'link', 'delete', 'tap89d51b72-76']
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: Operation not supported\n'


which I can confirm here

[root@PS1 logs(keystone_admin)]# ip link show tap89d51b72-76
973: tap89d51b72-76: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether f6:a2:60:bc:84:1e brd ff:ff:ff:ff:ff:ff
[root@PS1 logs(keystone_admin)]# su -s /bin/bash quantum -c 'sudo quantum-rootwrap /etc/quantum/rootwrap.conf ip link delete tap89d51b72-76'
RTNETLINK answers: Operation not supported
[root@PS1 logs(keystone_admin)]# 

I'll also attach the full debug output from quantum-ovs-cleanup

# rpm -qa | grep -e quantum -e packstack -e kernel 
libreport-plugin-kerneloops-2.0.9-15.el6.x86_64
dracut-kernel-004-303.el6.noarch
python-quantum-2013.1.2-2.el6.noarch
openstack-packstack-2013.1.1-0.21.dev651.el6.noarch
openstack-quantum-openvswitch-2013.1.2-2.el6.noarch
abrt-addon-kerneloops-2.0.8-15.el6.x86_64
kernel-2.6.32-358.el6.x86_64
kernel-2.6.32-358.114.1.openstack.el6.x86_64
openstack-quantum-2013.1.2-2.el6.noarch
python-quantumclient-2.2.1-2.el6.noarch
kernel-firmware-2.6.32-358.114.1.openstack.el6.noarch
Comment 1 Derek Higgins 2013-07-31 11:54:14 EDT
Created attachment 781224 [details]
debug log from a failed ovs cleanup
Comment 2 Derek Higgins 2013-07-31 11:58:30 EDT
In packstack the error manifests itself like this

err: /Stage[main]/Quantum::Agents::Ovs/Service[ovs-cleanup-service]/ensure: change from stopped to running failed: Could not start Service[ovs-cleanup-service]: Execution of '/sbin/service quantum-ovs-cleanup start' returned 1:  at /var/tmp/packstack/7dca6d6ed69748b3a431b7578d94a179/modules/quantum/manifests/agents/ovs.pp:130
Comment 3 Kashyap Chamarthy 2013-12-11 13:01:17 EST
Trying to triage these bugs. It's been 5 months since this report, do you still see this with latest RDO packages? You haven't indicated any specific version details (Neutron/Open vSwithc, iproute, etc).


Side note - https://wiki.openstack.org/wiki/BugFilingRecommendations
Comment 4 Derek Higgins 2013-12-16 06:37:27 EST
The version info for packstack/quantum are included at the end of the description, I havn't tried this since it was reported and no longer have the environment setup, so if nobody else is reporting the problem it can probably be closed (it may have been fixed in newer versions). IIRC the error was intermittent and only happened about 1 in every 20 runs.
Comment 5 Kashyap Chamarthy 2013-12-19 06:39:58 EST
(In reply to Derek Higgins from comment #4)
> The version info for packstack/quantum are included at the end of the
> description,

Sorry, I was just blind for a moment.

> I havn't tried this since it was reported and no longer have
> the environment setup, so if nobody else is reporting the problem it can
> probably be closed (it may have been fixed in newer versions). IIRC the
> error was intermittent and only happened about 1 in every 20 runs.

I don't' see any further reports similar to this as of now.

Closing it for now as CLOSED CURRENTRELEASE. 

As usual, feel free to re-open this if this crops up again.

Note You need to log in before you can comment on or make changes to this bug.