Created attachment 1456165 [details] undercloud_install.log Description of problem: Undercloud installation fails with error: INFO: dib-run-parts Tue Jul 3 05:11:19 EDT 2018 99-refresh-completed 0.478 INFO: dib-run-parts Tue Jul 3 05:11:19 EDT 2018 INFO: dib-run-parts Tue Jul 3 05:11:19 EDT 2018 --------------------- END PROFILING --------------------- INFO: [2018-07-03 05:11:19,213] (os-refresh-config) [INFO] Completed phase post-configure INFO: os-refresh-config completed successfully INFO: Network create/update failed HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> 2018-07-03 05:11:20,467 DEBUG: An exception occurred Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2288, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1983, in _post_config network = _ensure_neutron_network(sdk) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2028, in _ensure_neutron_network network = list(sdk.network.networks(name=PHYSICAL_NETWORK)) File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 903, in list exceptions.raise_from_response(response) File "/usr/lib/python2.7/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> 2018-07-03 05:11:20,467 ERROR: ############################################################################# Undercloud install failed. Reason: HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> See the previous output for details about what went wrong. The full install log can be found at /home/stack/.instack/install-undercloud.log. ############################################################################# Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2288, in install _post_config(instack_env, upgrade) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1983, in _post_config network = _ensure_neutron_network(sdk) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2028, in _ensure_neutron_network network = list(sdk.network.networks(name=PHYSICAL_NETWORK)) File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 903, in list exceptions.raise_from_response(response) File "/usr/lib/python2.7/site-packages/openstack/exceptions.py", line 212, in raise_from_response http_status=http_status, request_id=request_id openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> How reproducible: Always Steps to Reproduce: 1. Deploy OSPd14 using InfraRed virthost topology 1:1:1:1, puddle 2018-07-03.1 2. Use undercloud.conf: [DEFAULT] local_interface = eth0 local_ip = 192.168.24.1/24 network_gateway = 192.168.24.1 undercloud_public_vip = 192.168.24.2 undercloud_admin_vip = 192.168.24.3 network_cidr = 192.168.24.0/24 masquerade_network = 192.168.24.0/24 dhcp_start = 192.168.24.5 dhcp_end = 192.168.24.24 inspection_iprange = 192.168.24.100,192.168.24.120 3. openstack undercloud install Additional info: instack-undercloud-9.1.1-0.20180623161301.aab1180.el7ost.noarch instack-9.0.1-0.20180530160825.1dca54c.el7ost.noarch
Kicking over to Networking. Neutron wsgi is unhappy 2018-07-03 05:08:50.369 26588 DEBUG neutron.wsgi [-] (26588) accepted ('192.168.24.1', 45922) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883 2018-07-03 05:08:50.373 26588 INFO neutron.wsgi [-] Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 490, in handle_one_response result = self.application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__ return app(environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__ resp = self.call_func(req, *args, **kw) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__ response = req.get_response(self.application) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send application, catch_exc_info=False) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__ resp = self.call_func(req, *args, **kw) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__ response = req.get_response(self.application) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send application, catch_exc_info=False) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__ return self.application(environ, start_response) File "/usr/lib/python2.7/site-packages/pecan/core.py", line 835, in __call__ return super(Pecan, self).__call__(environ, start_response) File "/usr/lib/python2.7/site-packages/pecan/core.py", line 677, in __call__ controller, args, kwargs = self.find_controller(state) File "/usr/lib/python2.7/site-packages/pecan/core.py", line 853, in find_controller controller, args, kw = super(Pecan, self).find_controller(_state) File "/usr/lib/python2.7/site-packages/pecan/core.py", line 480, in find_controller accept.startswith('text/html,') and AttributeError: 'NoneType' object has no attribute 'startswith'
Same issue there. On neutron side, we don't have the same result using haproxy or not. # with haproxy [stack@undercloud ~]$ curl https://192.168.24.2:13696/ <html><body><h1>503 Service Unavailable</h1> No server is available to handle this request. </body></html> # without haproxy stack@undercloud ~]$ curl http://192.168.24.1:9696/ {"versions": [{"status": "CURRENT", "id": "v2.0", "links": [{"href": "http://192.168.24.1:9696/v2.0/", "rel": "self"}]}]} The only difference from the previous puddle is that the python-webob package has been upgraded from 1.7.2 to 1.8.1 [stack@undercloud ~]$ sudo rpm -qa *webob python2-webob-1.8.1-1.el7ost.noarch
Not sure if this is relevant, but a recent upstream CI run (using devstack) showed that it's using webob 1.8.2.
pecan==1.3.2 as well.
Need information from apevec on how to bump the pecan version.
It looks like the pecan version that included the fix was imported to RDO a day after you reported this bug. Can you confirm you're no longer seeing this issue on more recent CI runs?
Yes, I can confirm we are not hitting this bug in CI with python2-pecan-1.3.2-1.el7ost present on UC since puddle 2018-07-04.3.
Looks like Mikey provided the needed info, and his text is correct.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045