Bug 1162108 - Unable to delete 'active' namespaces via cleanup utility with "force" attribute
Summary: Unable to delete 'active' namespaces via cleanup utility with "force" attribute
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-neutron
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: z3
: 5.0 (RHEL 7)
Assignee: Miguel Angel Ajo
QA Contact: Toni Freger
URL:
Whiteboard:
Depends On: 1163470
Blocks: 1161524
TreeView+ depends on / blocked
 
Reported: 2014-11-10 09:53 UTC by Toni Freger
Modified: 2016-04-26 19:38 UTC (History)
8 users (show)

Fixed In Version: openstack-neutron-2014.1.3-11.el6ost openstack-neutron-2014.1.3-10.el7ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-12-02 17:00:27 UTC


Attachments (Terms of Use)
Netns cleanup fix tracking upstream changes in a single patch for later releases. (16.85 KB, patch)
2014-11-12 14:26 UTC, Miguel Angel Ajo
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2014:1942 normal SHIPPED_LIVE Moderate: openstack-neutron security and bug fix update 2014-12-02 21:58:59 UTC

Description Toni Freger 2014-11-10 09:53:17 UTC
Description of problem:
Unable to delete 'active' namespaces via cleanup utility with "force" attribute

(active namespace - It has additional devices other than the loopback device) 

neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini --force

Version-Release number of selected component (if applicable):

openstack-neutron-2014.1.3-7.el7ost.noarch
openstack-neutron-openvswitch-2014.1.3-7.el7ost.noarch
python-neutronclient-2.3.4-3.el7ost.noarch
python-neutron-2014.1.3-7.el7ost.noarch
openstack-neutron-ml2-2014.1.3-7.el7ost.noarch
openstack-neutron-metering-agent-2014.1.3-7.el7ost.noarch

How reproducible:
100%



Actual results:
neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini --force
2014-11-09 16:03:04.108 107268 INFO neutron.common.config [-] Logging enabled!
2014-11-09 16:03:06.168 107268 ERROR neutron.agent.netns_cleanup_util [-] Error unable to destroy namespace: qrouter-a66d28c0-04ba-4b7d-9a56-ebdb0c4e0278
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util Traceback (most recent call last):
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util   File "/usr/lib/python2.7/site-packages/neutron/agent/netns_cleanup_util.py", line 131, in destroy_namespace
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util     kill_dhcp(conf, namespace)
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util   File "/usr/lib/python2.7/site-packages/neutron/agent/netns_cleanup_util.py", line 85, in kill_dhcp
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util     if dhcp_driver.active:
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 253, in active
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util     return self.process_monitor.is_active(self.network.id,
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util AttributeError: 'NoneType' object has no attribute 'is_active'
2014-11-09 16:03:06.168 107268 TRACE neutron.agent.netns_cleanup_util
2014-11-09 16:03:06.169 107268 ERROR neutron.agent.netns_cleanup_util [-] Error unable to destroy namespace: qdhcp-b876a9d3-9c65-4ca2-95e9-6eb35bdc70c0
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util Traceback (most recent call last):
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util   File "/usr/lib/python2.7/site-packages/neutron/agent/netns_cleanup_util.py", line 131, in destroy_namespace
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util     kill_dhcp(conf, namespace)
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util   File "/usr/lib/python2.7/site-packages/neutron/agent/netns_cleanup_util.py", line 85, in kill_dhcp
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util     if dhcp_driver.active:
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 253, in active
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util     return self.process_monitor.is_active(self.network.id,
2014-11-09 16:03:06.169 107268 TRACE neutron.agent.netns_cleanup_util AttributeError: 'NoneType' object has no attribute 'is_active'

Expected results:


Additional info:

Comment 2 Ihar Hrachyshka 2014-11-10 12:18:11 UTC
Copying from email:

Miguel has backported process monitor patches to RHOSP5, and it seems
that cleanup utils were broken. The failure is as follows:

DhcpLocalProcess.active() was modified to use process monitor to track
process state. For this to work, process monitor should be passed when
the class is instantiated. The problem is that cleanup tools were not
updated to do so.

The problematic part is: neutron/agent/netns_cleanup_util.py

    dhcp_driver = importutils.import_object(
        conf.dhcp_driver,
        conf=conf,
        network=dhcp.NetModel(conf.use_namespaces, {'id': network_id}),
        root_helper=root_helper,
        plugin=FakeDhcpPlugin())

As you can see, there is no process monitor argument appended, hence
the failure. I think we should:

1) pass the argument in cleanup tools;
2) make it required (atm it's =None, allowing such failures to sneak).

The fix itself is easy, though we may need to make patch reshuffling
(or just band aid existing backports).

Comment 3 Miguel Angel Ajo 2014-11-12 14:24:41 UTC
Fixing upstream kilo. Including the fix downstream as a last patch in the series.

Comment 4 Miguel Angel Ajo 2014-11-12 14:26:21 UTC
Created attachment 956753 [details]
Netns cleanup fix tracking upstream changes in a single patch for later releases.

Comment 5 Miguel Angel Ajo 2014-11-12 18:26:21 UTC
The fix is not complete,  dnsmasqs won't be deleted, I also found the backport broke something else (reload of dnsmasq config file when we restart the agent) filling a new blocker for that.

Comment 7 Toni Freger 2014-11-17 09:38:57 UTC
Tested in Rhel7 AIO + Compute node
OpenStack/5.0-RHEL-7/2014-11-14.1

openstack-neutron-2014.1.3-10.el7ost.noarch
openstack-neutron-ml2-2014.1.3-10.el7ost.noarch
openstack-neutron-metering-agent-2014.1.3-10.el7ost.noarch
openstack-neutron-openvswitch-2014.1.3-10.el7ost.noarch


These two cleanup options work as expected without errors messages:

1) neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini --force

2) neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/dhcp_agent.ini

Comment 9 errata-xmlrpc 2014-12-02 17:00:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2014-1942.html


Note You need to log in before you can comment on or make changes to this bug.