Bug 1292587

Summary: neutron-l3-agent cannot correctly create routers on RHEL 7.2
Product: Red Hat OpenStack Reporter: Assaf Muller <amuller>
Component: openstack-neutronAssignee: Arie Bregman <abregman>
Status: CLOSED ERRATA QA Contact: Ofer Blaut <oblaut>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.0 (Kilo)CC: abregman, amuller, chricker, chrisw, ganguly, jdonohue, lpeer, nagashima-sj, nyechiel, oblaut, tfreger, timotcla, yeylon
Target Milestone: asyncKeywords: Reopened, Triaged, ZStream
Target Release: 7.0 (Kilo)   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-neutron-2015.1.2-6.el7ost Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1292570 Environment:
Last Closed: 2016-01-21 13:52:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1292570, 1292589, 1292591    
Bug Blocks: 1191185, 1235795    

Comment 3 Chandra Ganguly 2016-01-07 17:45:06 UTC
Cisco is hitting this issue with Kilo (OSP7) and RHEL7.2

Description of problem: neutron-l3-agent cannot correctly create routers on RHEL 7.2 due to updated 'ip netns' output. New 'ip netns list' output includes an (id: ) field that the l3-agent can't parse:

  [root@mcp-rhel-7 vagrant]# ip netns list
  qrouter-ec16ed02-229c-47dc-a927-63b2d91a8721 (id: 0)

Appears to be fixed in upstream, and I've confirmed the patch addresses the issue: https://review.openstack.org/#/c/258493/1/neutron/agent/linux/ip_lib.py

Version-Release number of selected component (if applicable):  OSP8 Beta (2015-12-03.2)

How reproducible: Consistently

Steps to Reproduce:
1. Install and configure neutron with ml2 and linuxbridge. 
2. Install and configure l3-agent.
3. Attempt to create a router.

Actual results:
Router creation reports success, but log shows errors.

Expected results:
Error free creation of routers, as with above linked patch.

Additional info:

Stack trace from neutron-l3-agent:
2015-12-17 19:52:18.821 20224 ERROR neutron.agent.l3.router_info
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'ec16ed02-229c-47dc-a927-63b2d91a8721'
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 498, in _process_router_update
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 441, in _process_router_if_compatible
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self._process_added_router(router)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 449, in _process_added_router
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     ri.process(self)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self.logger(e)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 694, in process
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self.process_external(agent)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 660, in process_external
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port, agent.pd)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 569, in _process_external_gateway
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self.external_gateway_added(ex_gw_port, interface_name)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 534, in external_gateway_added
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     ex_gw_port, interface_name, self.ns_name, preserve_ips)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 496, in _external_gateway_added
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self._plug_external_gateway(ex_gw_port, interface_name, ns_name)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 477, in _plug_external_gateway
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     prefix=EXTERNAL_DEV_PREFIX)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     bridge, namespace, prefix)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     namespace2=namespace)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self.ensure_namespace(namespace2)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     ip = self.netns.add(name)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     self._as_root([], ('add', name), use_root_namespace=True)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     use_root_namespace=use_root_namespace)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent     raise RuntimeError(m)
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent RuntimeError:
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent Command: ['ip', 'netns', 'add', u'qrouter-ec16ed02-229c-47dc-a927-63b2d91a8721']
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent Exit code: 1
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent Stdin:
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent Stdout:
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec16ed02-229c-47dc-a927-63b2d91a8721": File exists
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent
2015-12-17 19:52:18.824 20224 ERROR neutron.agent.l3.agent

Comment 7 Ofer Blaut 2016-01-20 09:59:28 UTC
I have create router on L3_HA setup, 
listed the host holding the router and check the logs, no error is found

Tested with OVS

openstack-neutron-2015.1.2-6.el7ost.noarch

Comment 9 errata-xmlrpc 2016-01-21 13:52:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:0060