Bug 1549536

Summary:	[OPS13] Overcloud deployment fails over "ERROR Host is not connected to any segments on routed provider network. It should be connected to one".
Product:	Red Hat OpenStack	Reporter:	Omri Hochman <ohochman>
Component:	instack-undercloud	Assignee:	Harald Jensås <hjensas>
Status:	CLOSED ERRATA	QA Contact:	Omri Hochman <ohochman>
Severity:	high	Docs Contact:
Priority:	high
Version:	13.0 (Queens)	CC:	aschultz, bfournie, dhill, hjensas, jschluet, knylande, mburns, psedlak, racedoro, rhel-osp-director-maint, sasha, skatlapa
Target Milestone:	beta	Keywords:	Triaged
Target Release:	13.0 (Queens)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	instack-undercloud-8.2.1-0.20180220181802.529796b.el7osp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-06-27 13:45:51 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Omri Hochman 2018-02-27 11:57:28 UTC

[OPS13] Deployment on BM fails over "ERROR Host is not connected to any segments on routed provider network. It should be connected to one".

Environment:
------------
instack-undercloud-8.2.1-0.20180214022816.8a7609e.el7ost.noarch
instack-8.1.1-0.20180210031553.1fe782a.el7ost.noarch
openstack-tripleo-heat-templates-8.0.0-0.20180215092255.el7ost.noarch


Steps: 
------
(1) Attempt to deploy osp13 on BM environment.  

Results : 
----------
Deployment failed over ->  Error while preparing to deploy to node : Unable to set binding:host_id for neutron port . Error: Host is not connected to any segments on routed provider network . It should be connected to one.


Log from undercloud /var/log/messages: 

Feb 20 16:29:11 undercloud75 haproxy[6213]: 192.168.0.3:46738 [20/Feb/2018:16:29:06.384] neutron neutron/192.168.0.1 4279/0/0/571/4850 409 441 - - ---- 35/7/1/2/0 0/0 "PUT /v2.
0/ports/f5542a4c-a16a-4c8f-b04d-337e6c6008d2 HTTP/1.1"
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat [req-3c528dc7-b512-4e8f-8d17-0eed53eff8c7 9d94c4fb227c46a3
b39602580ae1bac4 df65940a9f3d4fe395a7acce3a021382 - default default] Unable to set binding:host_id for neutron port f5542a4c-a16a-4c8f-b04d-337e6c6008d2. Error: Host 4a63ea60-0
618-493c-bc67-2e1aa3a9793d is not connected to any segments on routed provider network 'af4db330-6dca-464c-ade0-bd055c9115e1'.  It should be connected to one.
Feb 20 16:29:11 undercloud75 ironic-conductor: Neutron server returns request_ids: ['req-b04ff161-9232-4fdb-a962-3b3ce3d0b262']: Conflict: Host 4a63ea60-0618-493c-bc67-2e1aa3a9
793d is not connected to any segments on routed provider network 'af4db330-6dca-464c-ade0-bd055c9115e1'.  It should be connected to one.
Feb 20 16:29:11 undercloud75 ironic-conductor: Neutron server returns request_ids: ['req-b04ff161-9232-4fdb-a962-3b3ce3d0b262']
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat Traceback (most recent call last):
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/ironic/drivers/mo
dules/network/flat.py", line 78, in add_provisioning_network
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     client.update_port(vif_port_id, body)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 799, in update_port
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     revision_number=revision_number)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 2375, in _update_resource
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     return self.put(path, **kwargs)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 363, in put
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     headers=headers, params=params)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     headers=headers, params=params)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     self._handle_fault_response(status_code, replybody, resp)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     exception_handler_v20(status_code, error_body)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat     request_ids=request_ids)
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat Conflict: Host 4a63ea60-0618-493c-bc67-2e1aa3a9793d is not connected to any segments on routed provider network 'af4db330-6dca-464c-ade0-bd055c9115e1'.  It should be connected to one.
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat Neutron server returns request_ids: ['req-b04ff161-9232-4fdb-a962-3b3ce3d0b262']
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.235 7891 ERROR ironic.drivers.modules.network.flat
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.237 7891 ERROR ironic.conductor.manager [req-3c528dc7-b512-4e8f-8d17-0eed53eff8c7 9d94c4fb227c46a3b39602580ae1bac4 df65940a9f3d4fe395a7acce3a021382 - default default] Error while preparing to deploy to node 4a63ea60-0618-493c-bc67-2e1aa3a9793d: Unable to set binding:host_id for neutron port f5542a4c-a16a-4c8f-b04d-337e6c6008d2. Error: Host 4a63ea60-0618-493c-bc67-2e1aa3a9793d is not connected to any segments on routed provider network 'af4db330-6dca-464c-ade0-bd055c9115e1'.  It should be connected to one.
Feb 20 16:29:11 undercloud75 ironic-conductor: Neutron server returns request_ids: ['req-b04ff161-9232-4fdb-a962-3b3ce3d0b262']: NetworkError: Unable to set binding:host_id for neutron port f5542a4c-a16a-4c8f-b04d-337e6c6008d2. Error: Host 4a63ea60-0618-493c-bc67-2e1aa3a9793d is not connected to any segments on routed provider network 'af4db330-6dca-464c-ade0-bd055c9115e1'.  It should be connected to one.
Feb 20 16:29:11 undercloud75 ironic-conductor: 2018-02-20 16:29:11.238 7891 DEBUG ironic.common.states [req-3c528dc7-b512-4e8f-8d17-0eed53eff8c7 9d94c4fb227c46a3b39602580ae1bac4 df65940a9f3d4fe395a7acce3a021382 - default default] Exiting old state 'deploying' in response to event 'fail' on_exit /usr/lib/python2.7/site-packages/ironic/common/states.py:253

Comment 1 Pavel Sedlák 2018-02-27 13:27:33 UTC

likely same as https://bugzilla.redhat.com/show_bug.cgi?id=1549057

Comment 2 Alex Schultz 2018-03-01 19:34:51 UTC

*** Bug 1549263 has been marked as a duplicate of this bug. ***

Comment 3 Bob Fournier 2018-03-01 21:53:12 UTC

*** Bug 1549170 has been marked as a duplicate of this bug. ***

Comment 8 Bob Fournier 2018-03-29 13:52:56 UTC

I think this bug can be marked VERIFIED.

I checked that the fix is in the latest puddle (3.29.1):

(undercloud) [stack@host07 /]$ cat ./usr/share/instack-undercloud/undercloud.conf.sample | grep routed
# Enable support for routed ctlplane networks. (boolean value)
#enable_routed_networks = false

This was a major issue that was causing problems in CIX and all QE deployments. It has not occurred since the fix landed.

Comment 9 Harald Jensås 2018-03-30 15:40:18 UTC

Adding reference to gerrit change https://review.openstack.org/550234 as well since that is what is required to make this work with option ``enable_routed_networks`` set to true. This change is also in the 2018-03-29.1 puddle.

Comment 10 Omri Hochman 2018-04-03 14:22:07 UTC

Unable to reproduce with: 

(undercloud) [root@undercloud75 ~]# rpm -qa | grep instack-undercloud
instack-undercloud-8.4.0-2.el7ost.noarch

Comment 13 errata-xmlrpc 2018-06-27 13:45:51 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086