Bug 1597622 - Network create/update failed HttpException: 503: Server Error
Summary: Network create/update failed HttpException: 503: Server Error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-pecan
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: Upstream M3
: 14.0 (Rocky)
Assignee: Nate Johnston
QA Contact: Filip Hubík
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-07-03 10:11 UTC by Filip Hubík
Modified: 2019-01-11 11:50 UTC (History)
10 users (show)

Fixed In Version: python-pecan-1.3.2-1.el7ost
Doc Type: Bug Fix
Doc Text:
Previously, API requests to the policies file for checking non-admin user access permissions caused the entire file to reload and reparse. This resulted in slower processing time and degraded performance. This bug fix adds caching of the policies file so that queries to the file do not reload the entire file. Now, only changes to the file result in reloading and reparsing the file.
Clone Of:
Environment:
Last Closed: 2019-01-11 11:50:23 UTC
Target Upstream Version:


Attachments (Terms of Use)
undercloud_install.log (286.73 KB, text/plain)
2018-07-03 10:11 UTC, Filip Hubík
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:0045 None None None 2019-01-11 11:50:37 UTC

Description Filip Hubík 2018-07-03 10:11:53 UTC
Created attachment 1456165 [details]
undercloud_install.log

Description of problem:

Undercloud installation fails with error:

INFO: dib-run-parts Tue Jul  3 05:11:19 EDT 2018 99-refresh-completed                          0.478
INFO: dib-run-parts Tue Jul  3 05:11:19 EDT 2018
INFO: dib-run-parts Tue Jul  3 05:11:19 EDT 2018 --------------------- END PROFILING ---------------------
INFO: [2018-07-03 05:11:19,213] (os-refresh-config) [INFO] Completed phase post-configure
INFO: os-refresh-config completed successfully
INFO: Network create/update failed HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2018-07-03 05:11:20,467 DEBUG: An exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2288, in install
    _post_config(instack_env, upgrade)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1983, in _post_config
    network = _ensure_neutron_network(sdk)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2028, in _ensure_neutron_network
    network = list(sdk.network.networks(name=PHYSICAL_NETWORK))
  File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 903, in list
    exceptions.raise_from_response(response)
  File "/usr/lib/python2.7/site-packages/openstack/exceptions.py", line 212, in raise_from_response
    http_status=http_status, request_id=request_id
HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

2018-07-03 05:11:20,467 ERROR: 
#############################################################################
Undercloud install failed.

Reason: HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>


See the previous output for details about what went wrong.  The full install
log can be found at /home/stack/.instack/install-undercloud.log.

#############################################################################

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2288, in install
    _post_config(instack_env, upgrade)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1983, in _post_config
    network = _ensure_neutron_network(sdk)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2028, in _ensure_neutron_network
    network = list(sdk.network.networks(name=PHYSICAL_NETWORK))
  File "/usr/lib/python2.7/site-packages/openstack/resource.py", line 903, in list
    exceptions.raise_from_response(response)
  File "/usr/lib/python2.7/site-packages/openstack/exceptions.py", line 212, in raise_from_response
    http_status=http_status, request_id=request_id
openstack.exceptions.HttpException: HttpException: 503: Server Error for url: https://192.168.24.2:13696/networks?name=ctlplane, <html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

How reproducible:
Always

Steps to Reproduce:
1. Deploy OSPd14 using InfraRed virthost topology 1:1:1:1, puddle 2018-07-03.1
2. Use undercloud.conf:

[DEFAULT]
local_interface = eth0
local_ip = 192.168.24.1/24
network_gateway = 192.168.24.1
undercloud_public_vip = 192.168.24.2
undercloud_admin_vip = 192.168.24.3
network_cidr = 192.168.24.0/24
masquerade_network = 192.168.24.0/24
dhcp_start = 192.168.24.5
dhcp_end = 192.168.24.24
inspection_iprange = 192.168.24.100,192.168.24.120

3. openstack undercloud install

Additional info:
instack-undercloud-9.1.1-0.20180623161301.aab1180.el7ost.noarch
instack-9.0.1-0.20180530160825.1dca54c.el7ost.noarch

Comment 2 Alex Schultz 2018-07-03 15:19:15 UTC
Kicking over to Networking. Neutron wsgi is unhappy

2018-07-03 05:08:50.369 26588 DEBUG neutron.wsgi [-] (26588) accepted ('192.168.24.1', 45922) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2018-07-03 05:08:50.373 26588 INFO neutron.wsgi [-] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 490, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 835, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 677, in __call__
    controller, args, kwargs = self.find_controller(state)
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 853, in find_controller
    controller, args, kw = super(Pecan, self).find_controller(_state)
  File "/usr/lib/python2.7/site-packages/pecan/core.py", line 480, in find_controller
    accept.startswith('text/html,') and
AttributeError: 'NoneType' object has no attribute 'startswith'

Comment 3 Dimitri Savineau 2018-07-04 19:06:04 UTC
Same issue there. On neutron side, we don't have the same result using haproxy or not.

# with haproxy
[stack@undercloud ~]$ curl https://192.168.24.2:13696/
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>

# without haproxy
stack@undercloud ~]$ curl http://192.168.24.1:9696/
{"versions": [{"status": "CURRENT", "id": "v2.0", "links": [{"href": "http://192.168.24.1:9696/v2.0/", "rel": "self"}]}]}

The only difference from the previous puddle is that the python-webob package has been upgraded from 1.7.2 to 1.8.1

[stack@undercloud ~]$ sudo rpm -qa *webob
python2-webob-1.8.1-1.el7ost.noarch

Comment 4 Assaf Muller 2018-07-05 17:30:41 UTC
Not sure if this is relevant, but a recent upstream CI run (using devstack) showed that it's using webob 1.8.2.

Comment 5 Assaf Muller 2018-07-05 17:33:00 UTC
pecan==1.3.2 as well.

Comment 7 Nate Johnston 2018-07-06 15:12:41 UTC
Need information from apevec on how to bump the pecan version.

Comment 10 Assaf Muller 2018-07-06 17:49:16 UTC
It looks like the pecan version that included the fix was imported to RDO a day after you reported this bug. Can you confirm you're no longer seeing this issue on more recent CI runs?

Comment 11 Filip Hubík 2018-07-09 09:49:42 UTC
Yes, I can confirm we are not hitting this bug in CI with python2-pecan-1.3.2-1.el7ost present on UC since puddle 2018-07-04.3.

Comment 19 Nate Johnston 2019-01-09 19:45:04 UTC
Looks like Mikey provided the needed info, and his text is correct.

Comment 21 errata-xmlrpc 2019-01-11 11:50:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.