Description of problem: When querying OpenStack APIs, the links that are returned by APIs are with an http:// and not https:// Found some workaround for other services like nova, cinder and glance by specifically mentioning "public_endpoint" in configuration file in that is not present then by setting "secure_proxy_ssl_header" in configuration files. But none of these parameter is present in neutron.conf file. Version-Release number of selected component (if applicable): RHEL OSP 8 How reproducible: Everytime. Steps to Reproduce: 1. Try to do curl on endpoint in overcloud setup which is deployed using SSL, command is returning http instead of https. curl -s -H "X-Auth-Token: $OS_TOKEN" https://test.net:13696/v2.0/ | jq 2. 3. Actual results: It's returning http. Expected results: It should return https instead of http. Additional info: Found some old bugs regarding same query but can't the public_endpoint parameter in neutron.conf file. https://bugzilla.redhat.com/show_bug.cgi?id=1237207
Reassigning to Keystone. I'm not sure if Neutron is doing something wrong but I hope that Keystone developers will be able to shed more light on this issue.
Service catalog setup is done by Tripleo, and is not affected by, nor controlled by Keystone. Keystone is just a repository. Making this work requires TLS everywhere in Tripleo setup.
So, this is handled by the http_proxy_to_wsgi middleware which was introduced to several services as part of the TLS work for TripleO that we did. This is no longer a problem in nova, glance and cinder because they use that middleware (as verified by Marius Cornea). However, neutron is not using this middleware (yet). So I have to propose a patch for neutron, and then enable the middleware in TripleO for it to work. Fortunately, making calls to neutron seems to use the https endpoint (cause it's coming from the keystone catalog). Anyway, this is a neutron issue (since it doesn't use such middleware) and we need to enable this functionality in TripleO once that lands.
Marius Cornea took the time to check more services and these are the ones that need the middleware: * neutron * aodh * ceilometer * heat-cfn (heat-api uses it already) * gnocchi I'm submitting patches for them.
Separate bugs have been filed for addressing this issue (by using the HTTPProxyToWSGI middleware) in some of the components mentioned in comment#6: neutron - needs downstream BZ (https://review.openstack.org/384294) aodh - bug#1383180 (https://review.openstack.org/384305) ceilometer - bug#1383183 (https://review.openstack.org/384311) heat-cfn - bug#1383185 (https://review.openstack.org/384314) gnocchi - bug#1383186 (https://review.openstack.org/384301) Additionally, the puppet modules for these services need to be updated. puppet-neutron - needs downstream BZ (patch not submitted upstream yet?) puppet-aodh - needs downstream BZ (https://review.openstack.org/384371) puppet-ceilometer - needs downstream BZ (https://review.openstack.org/384366) puppet-heat - needs downstream BZ (patch not submitted upstream yet?) puppet-gnocchi - needs downstream BZ (https://review.openstack.org/384358) This will still need to be addressed in openstack-tripleo-heat-templates once all of the affected services and puppet modules are updated to use the HTTPProxyToWSGI middleware. This bug should be used for the updates to openstack-tripleo-heat-templates.
Nathan, the heat fix doesn't need a change in puppet-heat. The option is already available, since heat-api was using it already.
The haproxy configuration is still wrong, haproxy is configured to do TCP proxy and not mode http proxy: defaults ... mode tcp ... So all 'http-request ....' are ignored by haproxy, since it does parse the http layer. And application continue to not receive the X-Forwarded-Proto header. I would suggest to change the mode to 'http' at least for listeners that need to change headers.
Added the change to the CR. Heat worked cause we did have mode http from before.
verified for - First, check keystone's service catalog: [stack@undercloud-0 ~]$ source overcloudrc [stack@undercloud-0 ~]$ openstack catalog list +------------+----------------+---------------------------------------------------------------------------------+ | Name | Type | Endpoints | +------------+----------------+---------------------------------------------------------------------------------+ | nova | compute | regionOne | | | | publicURL: https://10.0.0.101:13774/v2.1 | | | | internalURL: http://172.17.1.12:8774/v2.1 | | | | adminURL: http://172.17.1.12:8774/v2.1 | | | | | | neutron | network | regionOne | | | | publicURL: https://10.0.0.101:13696 | | | | internalURL: http://172.17.1.12:9696 | | | | adminURL: http://172.17.1.12:9696 | | | | | | cinderv2 | volumev2 | regionOne | | | | publicURL: https://10.0.0.101:13776/v2/99a38f7af81c4ee5b3c92bb837674314 | | | | internalURL: http://172.17.1.12:8776/v2/99a38f7af81c4ee5b3c92bb837674314 | | | | adminURL: http://172.17.1.12:8776/v2/99a38f7af81c4ee5b3c92bb837674314 | | | | | | cinderv3 | volumev3 | regionOne | | | | publicURL: https://10.0.0.101:13776/v3/99a38f7af81c4ee5b3c92bb837674314 | | | | internalURL: http://172.17.1.12:8776/v3/99a38f7af81c4ee5b3c92bb837674314 | | | | adminURL: http://172.17.1.12:8776/v3/99a38f7af81c4ee5b3c92bb837674314 | | | | | | aodh | alarming | regionOne | | | | publicURL: https://10.0.0.101:13042 | | | | internalURL: http://172.17.1.12:8042 | | | | adminURL: http://172.17.1.12:8042 | | | | | | glance | image | regionOne | | | | publicURL: https://10.0.0.101:13292 | | | | internalURL: http://172.17.3.10:9292 | | | | adminURL: http://172.17.3.10:9292 | | | | | | ceilometer | metering | regionOne | | | | publicURL: https://10.0.0.101:13777 | | | | internalURL: http://172.17.1.12:8777 | | | | adminURL: http://172.17.1.12:8777 | | | | | | heat-cfn | cloudformation | regionOne | | | | publicURL: https://10.0.0.101:13005/v1 | | | | internalURL: http://172.17.1.12:8000/v1 | | | | adminURL: http://172.17.1.12:8000/v1 | | | | | | cinder | volume | regionOne | | | | publicURL: https://10.0.0.101:13776/v1/99a38f7af81c4ee5b3c92bb837674314 | | | | internalURL: http://172.17.1.12:8776/v1/99a38f7af81c4ee5b3c92bb837674314 | | | | adminURL: http://172.17.1.12:8776/v1/99a38f7af81c4ee5b3c92bb837674314 | | | | | | heat | orchestration | regionOne | | | | publicURL: https://10.0.0.101:13004/v1/99a38f7af81c4ee5b3c92bb837674314 | | | | internalURL: http://172.17.1.12:8004/v1/99a38f7af81c4ee5b3c92bb837674314 | | | | adminURL: http://172.17.1.12:8004/v1/99a38f7af81c4ee5b3c92bb837674314 | | | | | | swift | object-store | regionOne | | | | publicURL: https://10.0.0.101:13808/v1/AUTH_99a38f7af81c4ee5b3c92bb837674314 | | | | internalURL: http://172.17.3.10:8080/v1/AUTH_99a38f7af81c4ee5b3c92bb837674314 | | | | adminURL: http://172.17.3.10:8080 | | | | | | gnocchi | metric | regionOne | | | | publicURL: https://10.0.0.101:13041 | | | | internalURL: http://172.17.1.12:8041 | | | | adminURL: http://172.17.1.12:8041 | | | | | | keystone | identity | regionOne | | | | publicURL: https://10.0.0.101:13000/v2.0 | | | | internalURL: http://172.17.1.12:5000/v2.0 | | | | adminURL: http://192.0.2.8:35357/v2.0 | | | | | +------------+----------------+---------------------------------------------------------------------------------+ - Now, check individual APIs: * neutron: [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" https://10.0.0.101:13696 | python -m json.tool { "versions": [ { "id": "v2.0", "links": [ { "href": "https://10.0.0.101:13696/v2.0", "rel": "self" } ], "status": "CURRENT" } ] } * aodh: [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" https://10.0.0.101:13042 | python -m json.tool { "versions": { "values": [ { "id": "v2", "links": [ { "href": "https://10.0.0.101:13042/v2", "rel": "self" }, { "href": "http://docs.openstack.org/", "rel": "describedby", "type": "text/html" } ], "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.telemetry-v2+json" }, { "base": "application/xml", "type": "application/vnd.openstack.telemetry-v2+xml" } ], "status": "stable", "updated": "2013-02-13T00:00:00Z" } ] } } * ceilometer [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" https://10.0.0.101:13777 | python -m json.tool { "versions": { "values": [ { "id": "v2", "links": [ { "href": "https://10.0.0.101:13777/v2", "rel": "self" }, { "href": "http://docs.openstack.org/", "rel": "describedby", "type": "text/html" } ], "media-types": [ { "base": "application/json", "type": "application/vnd.openstack.telemetry-v2+json" }, { "base": "application/xml", "type": "application/vnd.openstack.telemetry-v2+xml" } ], "status": "stable", "updated": "2013-02-13T00:00:00Z" } ] } } * heat-cfn [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" https://10.0.0.101:13005 | python -m json.tool { "versions": [ { "id": "v1.0", "links": [ { "href": "https://10.0.0.101:13005/v1/", "rel": "self" } ], "status": "CURRENT" } ] } * gnocchi [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" https://10.0.0.101:13041 | python -m json.tool { "versions": [ { "id": "v1.0", "links": [ { "href": "https://10.0.0.101:13041/v1/", "rel": "self" } ], "status": "CURRENT", "updated": "2015-03-19" } ] }
(In reply to Rodrigo Duarte from comment #22) > verified for > > - First, check keystone's service catalog: > > [stack@undercloud-0 ~]$ source overcloudrc > [stack@undercloud-0 ~]$ openstack catalog list > +------------+----------------+---------------------------------------------- > -----------------------------------+ > | Name | Type | Endpoints > | > +------------+----------------+---------------------------------------------- > -----------------------------------+ > | nova | compute | regionOne > | > | | | publicURL: https://10.0.0.101:13774/v2.1 > | > | | | internalURL: http://172.17.1.12:8774/v2.1 > | > | | | adminURL: http://172.17.1.12:8774/v2.1 > | > | | | > | > | neutron | network | regionOne > | > | | | publicURL: https://10.0.0.101:13696 > | > | | | internalURL: http://172.17.1.12:9696 > | > | | | adminURL: http://172.17.1.12:9696 > | > | | | > | > | cinderv2 | volumev2 | regionOne > | > | | | publicURL: > https://10.0.0.101:13776/v2/99a38f7af81c4ee5b3c92bb837674314 | > | | | internalURL: > http://172.17.1.12:8776/v2/99a38f7af81c4ee5b3c92bb837674314 | > | | | adminURL: > http://172.17.1.12:8776/v2/99a38f7af81c4ee5b3c92bb837674314 | > | | | > | > | cinderv3 | volumev3 | regionOne > | > | | | publicURL: > https://10.0.0.101:13776/v3/99a38f7af81c4ee5b3c92bb837674314 | > | | | internalURL: > http://172.17.1.12:8776/v3/99a38f7af81c4ee5b3c92bb837674314 | > | | | adminURL: > http://172.17.1.12:8776/v3/99a38f7af81c4ee5b3c92bb837674314 | > | | | > | > | aodh | alarming | regionOne > | > | | | publicURL: https://10.0.0.101:13042 > | > | | | internalURL: http://172.17.1.12:8042 > | > | | | adminURL: http://172.17.1.12:8042 > | > | | | > | > | glance | image | regionOne > | > | | | publicURL: https://10.0.0.101:13292 > | > | | | internalURL: http://172.17.3.10:9292 > | > | | | adminURL: http://172.17.3.10:9292 > | > | | | > | > | ceilometer | metering | regionOne > | > | | | publicURL: https://10.0.0.101:13777 > | > | | | internalURL: http://172.17.1.12:8777 > | > | | | adminURL: http://172.17.1.12:8777 > | > | | | > | > | heat-cfn | cloudformation | regionOne > | > | | | publicURL: https://10.0.0.101:13005/v1 > | > | | | internalURL: http://172.17.1.12:8000/v1 > | > | | | adminURL: http://172.17.1.12:8000/v1 > | > | | | > | > | cinder | volume | regionOne > | > | | | publicURL: > https://10.0.0.101:13776/v1/99a38f7af81c4ee5b3c92bb837674314 | > | | | internalURL: > http://172.17.1.12:8776/v1/99a38f7af81c4ee5b3c92bb837674314 | > | | | adminURL: > http://172.17.1.12:8776/v1/99a38f7af81c4ee5b3c92bb837674314 | > | | | > | > | heat | orchestration | regionOne > | > | | | publicURL: > https://10.0.0.101:13004/v1/99a38f7af81c4ee5b3c92bb837674314 | > | | | internalURL: > http://172.17.1.12:8004/v1/99a38f7af81c4ee5b3c92bb837674314 | > | | | adminURL: > http://172.17.1.12:8004/v1/99a38f7af81c4ee5b3c92bb837674314 | > | | | > | > | swift | object-store | regionOne > | > | | | publicURL: > https://10.0.0.101:13808/v1/AUTH_99a38f7af81c4ee5b3c92bb837674314 | > | | | internalURL: > http://172.17.3.10:8080/v1/AUTH_99a38f7af81c4ee5b3c92bb837674314 | > | | | adminURL: http://172.17.3.10:8080 > | > | | | > | > | gnocchi | metric | regionOne > | > | | | publicURL: https://10.0.0.101:13041 > | > | | | internalURL: http://172.17.1.12:8041 > | > | | | adminURL: http://172.17.1.12:8041 > | > | | | > | > | keystone | identity | regionOne > | > | | | publicURL: https://10.0.0.101:13000/v2.0 > | > | | | internalURL: http://172.17.1.12:5000/v2.0 > | > | | | adminURL: http://192.0.2.8:35357/v2.0 > | > | | | > | > +------------+----------------+---------------------------------------------- > -----------------------------------+ > > - Now, check individual APIs: > > * neutron: > > [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" > https://10.0.0.101:13696 | python -m json.tool > { > "versions": [ > { > "id": "v2.0", > "links": [ > { > "href": "https://10.0.0.101:13696/v2.0", > "rel": "self" > } > ], > "status": "CURRENT" > } > ] > } > > * aodh: > > [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" > https://10.0.0.101:13042 | python -m json.tool > { > "versions": { > "values": [ > { > "id": "v2", > "links": [ > { > "href": "https://10.0.0.101:13042/v2", > "rel": "self" > }, > { > "href": "http://docs.openstack.org/", > "rel": "describedby", > "type": "text/html" > } > ], > "media-types": [ > { > "base": "application/json", > "type": "application/vnd.openstack.telemetry-v2+json" > }, > { > "base": "application/xml", > "type": "application/vnd.openstack.telemetry-v2+xml" > } > ], > "status": "stable", > "updated": "2013-02-13T00:00:00Z" > } > ] > } > } > > * ceilometer > > [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" > https://10.0.0.101:13777 | python -m json.tool > { > "versions": { > "values": [ > { > "id": "v2", > "links": [ > { > "href": "https://10.0.0.101:13777/v2", > "rel": "self" > }, > { > "href": "http://docs.openstack.org/", > "rel": "describedby", > "type": "text/html" > } > ], > "media-types": [ > { > "base": "application/json", > "type": "application/vnd.openstack.telemetry-v2+json" > }, > { > "base": "application/xml", > "type": "application/vnd.openstack.telemetry-v2+xml" > } > ], > "status": "stable", > "updated": "2013-02-13T00:00:00Z" > } > ] > } > } > > * heat-cfn > > [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" > https://10.0.0.101:13005 | python -m json.tool > { > "versions": [ > { > "id": "v1.0", > "links": [ > { > "href": "https://10.0.0.101:13005/v1/", > "rel": "self" > } > ], > "status": "CURRENT" > } > ] > } > > * gnocchi > > [stack@undercloud-0 ~]$ curl -s -H "X-Auth-Token: $OS_TOKEN" > https://10.0.0.101:13041 | python -m json.tool > { > "versions": [ > { > "id": "v1.0", > "links": [ > { > "href": "https://10.0.0.101:13041/v1/", > "rel": "self" > } > ], > "status": "CURRENT", > "updated": "2015-03-19" > } > ] > } verified for puppet-tripleo-5.4.0-3.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2948.html