Bug 1451402

Here is how we fixed Glance manually:

Change to haproxy:
~~~
listen glance_api
  bind 10.0.0.4:13292 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 172.18.0.10:9292 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0 172.18.0.11:9292 check fall 5 inter 2000 rise 2
~~~

Note: mode http !!

Apply https://review.openstack.org/#/c/294681/2/glance/common/wsgi.py

[root@overcloud-controller-0 common]# pwd
/usr/lib/python2.7/site-packages/glance/common
[root@overcloud-controller-0 common]# diff -u wsgi.py.old  wsgi.py
--- wsgi.py.old	2017-05-16 17:26:05.754104430 +0000
+++ wsgi.py	2017-05-16 17:27:19.389453638 +0000
@@ -105,6 +105,14 @@
                       'wait forever.')),
 ]
 
+wsgi_opts = [
+    cfg.StrOpt('secure_proxy_ssl_header',
+               help=_('The HTTP header used to determine the scheme for the '
+                      'original request, even if it was removed by an SSL '
+                      'terminating proxy. Typical value is '
+                      '"HTTP_X_FORWARDED_PROTO".')),
+]
+
 profiler_opts = [
     cfg.BoolOpt("enabled", default=False,
                 help=_('If False fully disable profiling feature.')),
@@ -119,6 +127,7 @@
 CONF.register_opts(bind_opts)
 CONF.register_opts(socket_opts)
 CONF.register_opts(eventlet_opts)
+CONF.register_opts(wsgi_opts)
 CONF.register_opts(profiler_opts, group="profiler")
 
 ASYNC_EVENTLET_THREAD_POOL_LIST = []
@@ -720,6 +729,13 @@
 class Request(webob.Request):
     """Add some OpenStack API-specific logic to the base webob.Request."""
 
+    def __init__(self, environ, *args, **kwargs):
+        if CONF.secure_proxy_ssl_header:
+            scheme = environ.get(CONF.secure_proxy_ssl_header)
+            if scheme:
+                environ['wsgi.url_scheme'] = scheme
+        super(Request, self).__init__(environ, *args, **kwargs)
+
     def best_match_content_type(self):
         """Determine the requested response content-type."""
         supported = ('application/json',)
[root@overcloud-controller-0 common]#


Modify glance configuration files:
~~~
[root@overcloud-controller-0 common]# head /etc/glance/glance-api.conf 
[DEFAULT]
secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO

#
# From glance.api
#
~~~

https://access.redhat.com/solutions/3034681

Hi test failed since the following parameter is missing in glance-api.conf
"secure_proxy_ssl_header" that enable the feature  
you can take a look on upstream conf file 
https://github.com/openstack/glance/blob/master/etc/glance-api.conf

@Avi: I think we should not hardcode this value in our package. This seems like a configuration option that has to be set by the user. In #5, we see that they use:


[root@overcloud-controller-0 common]# head /etc/glance/glance-api.conf 
[DEFAULT]
secure_proxy_ssl_header = HTTP_X_FORWARDED_PROTO

@Andreas: Would you agree with that?

Hi,

Yes, I agree and I'm well aware that I need to set:
crudini --set /etc/glance/glance-api.conf DEFAULT secure_proxy_ssl_header HTTP_X_FORWARDED_PROTO


Exactly the same behavior as for nova :-)

- Andreas

@cyril If the parameter is not documented anywhere how should the system administrator that's need to enable the feature know about it if it is not documented in the configuration file ?  
It is not clear to the user that such parameter for activates the option for https. 
where do I find what should be the optional values of the parameter ( Is it  secure_proxy_ssl_header = true )?
Do I miss something major here ?

@Avi: in Ocata branch, I can see the following in etc/glance-api.conf:

# DEPRECATED: The HTTP header used to determine the scheme for the original
# request, even if it was removed by an SSL terminating proxy. Typical value is
# "HTTP_X_FORWARDED_PROTO". (string value)
# This option is deprecated for removal.
# Its value may be silently ignored in the future.
# Reason: Use the http_proxy_to_wsgi middleware instead.
#secure_proxy_ssl_header = <None>


Isn't this enough? Should we add more documentation?

@cyril this parameter not part of configuration file in python-glance-11.0.1-7.el7ost.noarch

So, this seems to work:

curl -H 'X_FORWARDED_PROTO: https' https://10.0.0.101:13292/

But I believe you do not set this header manually for every curl command. Would you care to give us details about your setup, so that Avi can reproduce it?

Also, I stand corrected, the documentation doesn't say anything about this new option, even though some documentation should be generated somehow...

Hi,


All steps to fix this manually (without haproxy / cinder / neutron /nova patches) are outlined here: https://access.redhat.com/solutions/3034681  --- the KCS also contains links to the other backport requests.

Also, see comment 11

Yes, I agree and I'm well aware that I need to set:
crudini --set /etc/glance/glance-api.conf DEFAULT secure_proxy_ssl_header HTTP_X_FORWARDED_PROTO

The settings are the same as in nova (you are porting a patch that was initially applied to nova to cinder, so the options are obviously the same as well).

FYI, I do _not_ know why it needs to be HTTP_X_FORWARDED_PROTO when haproxy is injecting the header `X-Forwarded-Proto` - IIRC, some method in the code is doing an upper() and converts the '-' to '_' ... this isn't clearly documented anywhere, neither in nova nor in cinder, btw. We found it out by trial and error
~~~
  mode http                                                # <----------------------- add this
  http-request set-header X-Forwarded-Proto https if { ssl_fc } # <----------------------- add this
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc } # <----------------------- add this
~~~

- Andreas

@Avi: this new package has the doc fix you requested. In order to test the fix, you should "curl -H 'X_FORWARDED_PROTO: https' https://10.0.0.101:13292/" to simulate the header injection performed by haproxy.

@cyril the haproxy configuration is missing I opened a new bug 
https://bugzilla.redhat.com/show_bug.cgi?id=1461358
without the update in /etc/haproxy/heproxy.conf the response is not in https protocol

Hi,

Closing https://bugzilla.redhat.com/show_bug.cgi?id=1461358 as duplicate of 

This was documented in https://access.redhat.com/solutions/3034681

Thanks,

Andreas

Hi,

https://bugzilla.redhat.com/show_bug.cgi?id=1461358 --(dup of)--> https://bugzilla.redhat.com/show_bug.cgi?id=1451495 which ships openstack-puppet-modules that change haproxy configuration. This one can be verified as well by making a manual change to haproxy:

~~~
  mode http                                                # <----------------------- add this
  http-request set-header X-Forwarded-Proto https if { ssl_fc } # <----------------------- add this
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc } # <----------------------- add this
~~~

More details are outlined in: https://access.redhat.com/solutions/3034681

This one can also be verified by:
curl -H 'X_FORWARDED_PROTO: https' https://10.0.0.101:13292/
or
curl -H 'X-Forwarded-Proto: https' https://10.0.0.101:13292/

Regards,

Andreas

Andreas, 

Can you confirm if we can ship this out without the accompanying fix:
https://bugzilla.redhat.com/show_bug.cgi?id=1451495

And customer that wants it can use the info here for manual step until that's available?

https://access.redhat.com/solutions/3034681

Hi Paul,

IMO the most important is to be able to have something that doesn't require a manual code change. In that sense, getting and shipping this without the accompanying haproxy fix is o.k. to me, it's easy to fix haproxy.cfg manually.

- Andreas

Following the last comments and after we verified this bug with the workaround mentioned on comment #22 - seems like it is working fine.

We need to make sure it is documented right.

A.

The the fix was verified with manual update of haproxy.conf and adding a parameter to glance-api.conf manually.
and an https replay was given to the client  

curl  https://10.0.0.101:13292/ 
{"versions": [{"status": "CURRENT", "id": "v2.3", "links": [{"href": "https://10.0.0.101:13292/v2/", "rel": "self"}]}, {"status": "SUPPORTED", "id": "v2.2", "links": [{"href": "https://10.0.0.101:13292/v2/", "rel": "self"}]}, {"status": "SUPPORTED", "id": "v2.1", "links": [{"href": "https://10.0.0.101:13292/v2/", "rel": "self"}]}, {"status": "SUPPORTED", "id": "v2.0", "links": [{"href": "https://10.0.0.101:13292/v2/", "rel": "self"}]}, {"status": "SUPPORTED", "id": "v1.1", "links": [{"href": "https://10.0.0.101:13292/v1/", "rel": "self"}]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1544