Prior to this update, the secure_proxy_ssl_header option for Compute was not being set in nova.conf by Red Hat OpenStack Platform director (as discussed in the upstream bug https://bugs.launchpad.net/tripleo/+bug/1606863).
Consequently, when haproxy and SSL were enabled for the director deployment, nova-api could not handle service requests since it was not configured to handle the "X-Forwarded-Proto" header in HTTP requests. In particular, the tempest.api.compute.test_versions.TestVersions.test_get_version_details tests failed with the error:
'Connection aborted.', BadStatusLine("''",)
With this update, the secure_proxy_ssl_header is now set to the appropriate value (X-Forwarded-Proto) for director deployments, see https://review.openstack.org/#/c/347806/ for more details.
As a result, the nova-api service should now be able to handle service requests correctly when haproxy and SSL are enabled for the director deployment.
Description of problem:
tempest.api.compute.test_versions.TestVersions.test_get_version_details fails with BadStatusLine because it doesn't set the option secure_proxy_ssl_header when OSPD is installed with HAProxy and SSL enabled
Version-Release number of selected component (if applicable):
openstack-nova-scheduler-13.1.0-1.el7ost.noarch
python-novaclient-3.3.0-1.el7ost.noarch
openstack-nova-common-13.1.0-1.el7ost.noarch
openstack-nova-compute-13.1.0-1.el7ost.noarch
openstack-nova-api-13.1.0-1.el7ost.noarch
openstack-nova-cert-13.1.0-1.el7ost.noarch
openstack-nova-conductor-13.1.0-1.el7ost.noarch
openstack-nova-console-13.1.0-1.el7ost.noarch
python-nova-13.1.0-1.el7ost.noarch
openstack-nova-novncproxy-13.1.0-1.el7ost.noarch
How reproducible:
Whenever you deploy OSPD 9 with HAProxy and SSL enabled
Steps to Reproduce:
1. Install OSPD with HA and SSL enabled
2. Setup tempest
3. Run tempest test tempest.api.compute.test_versions.TestVersions.test_get_version_details
Actual results:
Fails with the following:
ythonlogging:'': {{{
2016-06-29 04:18:25,940 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,941 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=8, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,942 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=7, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,943 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=6, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,944 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=5, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,944 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=4, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,945 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=3, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,946 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=2, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,947 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=1, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
2016-06-29 04:18:25,948 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=0, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/
}}}
Traceback (most recent call last):
File "tempest/api/compute/test_versions.py", line 69, in test_get_version_details
result = self.versions_client.get_version_by_url(link['href'])
File "tempest/lib/services/compute/versions_client.py", line 58, in get_version_by_url
{'X-Auth-Token': self.token})
File "tempest/lib/common/rest_client.py", line 578, in raw_request
body=body, chunked=chunked)
File "tempest/lib/common/http.py", line 54, in request
*args, **new_kwargs)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/request.py", line 69, in request
**urlopen_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/request.py", line 90, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/poolmanager.py", line 248, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen
release_conn=release_conn, **response_kw)
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 640, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/util/retry.py", line 287, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=u'10.0.0.101', port=13774): Max retries exceeded with url: /v2/ (Caused by ProtocolError('Connection aborted.', BadStatusLine("''",)))
Expected results:
Test pass
Additional info:
Editing /etc/nova/nova.conf on the controller, and adds the option secure_proxy_ssl_header=HTTP_X_FORWARDED_PROTO and restarting openstack-nova-api fix the problem.
Looking at puppet-nova, I see this is being set as so:
nova/manifests/api.pp: $secure_proxy_ssl_header = $::os_service_default,
So my guess is tht just needs to configure this, in the same manner as is done elsewhere
moved to ON_DEV - I filed an upstream bug for this (required for stable/mitaka, which is where this is needed here) https://bugs.launchpad.net/tripleo/+bug/1606863 - reviews to master and mitaka linked.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHEA-2016-1762.html
Description of problem: tempest.api.compute.test_versions.TestVersions.test_get_version_details fails with BadStatusLine because it doesn't set the option secure_proxy_ssl_header when OSPD is installed with HAProxy and SSL enabled Version-Release number of selected component (if applicable): openstack-nova-scheduler-13.1.0-1.el7ost.noarch python-novaclient-3.3.0-1.el7ost.noarch openstack-nova-common-13.1.0-1.el7ost.noarch openstack-nova-compute-13.1.0-1.el7ost.noarch openstack-nova-api-13.1.0-1.el7ost.noarch openstack-nova-cert-13.1.0-1.el7ost.noarch openstack-nova-conductor-13.1.0-1.el7ost.noarch openstack-nova-console-13.1.0-1.el7ost.noarch python-nova-13.1.0-1.el7ost.noarch openstack-nova-novncproxy-13.1.0-1.el7ost.noarch How reproducible: Whenever you deploy OSPD 9 with HAProxy and SSL enabled Steps to Reproduce: 1. Install OSPD with HA and SSL enabled 2. Setup tempest 3. Run tempest test tempest.api.compute.test_versions.TestVersions.test_get_version_details Actual results: Fails with the following: ythonlogging:'': {{{ 2016-06-29 04:18:25,940 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=9, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,941 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=8, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,942 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=7, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,943 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=6, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,944 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=5, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,944 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=4, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,945 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=3, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,946 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=2, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,947 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=1, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ 2016-06-29 04:18:25,948 9368 WARNING [urllib3.connectionpool] Retrying (Retry(total=0, connect=None, read=None, redirect=5)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine("''",))': /v2/ }}} Traceback (most recent call last): File "tempest/api/compute/test_versions.py", line 69, in test_get_version_details result = self.versions_client.get_version_by_url(link['href']) File "tempest/lib/services/compute/versions_client.py", line 58, in get_version_by_url {'X-Auth-Token': self.token}) File "tempest/lib/common/rest_client.py", line 578, in raw_request body=body, chunked=chunked) File "tempest/lib/common/http.py", line 54, in request *args, **new_kwargs) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/request.py", line 69, in request **urlopen_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/request.py", line 90, in request_encode_url return self.urlopen(method, url, **extra_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/poolmanager.py", line 248, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 668, in urlopen release_conn=release_conn, **response_kw) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 640, in urlopen _stacktrace=sys.exc_info()[2]) File "/home/stack/tempest/.tox/py27/lib/python2.7/site-packages/urllib3/util/retry.py", line 287, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host=u'10.0.0.101', port=13774): Max retries exceeded with url: /v2/ (Caused by ProtocolError('Connection aborted.', BadStatusLine("''",))) Expected results: Test pass Additional info: Editing /etc/nova/nova.conf on the controller, and adds the option secure_proxy_ssl_header=HTTP_X_FORWARDED_PROTO and restarting openstack-nova-api fix the problem.