+++ This bug was initially created as a clone of Bug #1465969 +++ In default SSL deployments, 35357 is exposed as non-SSL on the external network: $ curl http://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3 {"version": {"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3/", "rel": "self"}]}} $ curl https://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3 curl: (35) SSL received a record that exceeded the maximum permissible length. I can confirm the same behavior in a lab environment: [stack@undercloud-1 ~]$ curl http://osp.example.net:35357 {"versions": {"values": [{"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://osp.example.net:35357/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://osp.example.net:35357/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}[stack@undercloud-1 ~]$ [stack@undercloud-1 ~]$ [root@overcloud-controller-0 ~]# ss -lntp | grep 5000 LISTEN 0 128 172.16.2.12:5000 *:* users:(("haproxy",pid=156879,fd=30)) LISTEN 0 128 172.16.2.5:5000 *:* users:(("httpd",pid=401621,fd=4),("httpd",pid=401620,fd=4),("httpd",pid=401586,fd=4),("httpd",pid=128954,fd=4),("httpd",pid=128953,fd=4),("httpd",pid=128912,fd=4),("httpd",pid=116970,fd=4),("httpd",pid=116969,fd=4),("httpd",pid=116968,fd=4),("httpd",pid=116967,fd=4),("httpd",pid=116966,fd=4),("httpd",pid=116965,fd=4),("httpd",pid=116964,fd=4),("httpd",pid=116963,fd=4),("httpd",pid=116946,fd=4)) [root@overcloud-controller-0 ~]# ss -lntp | grep 35357 LISTEN 0 128 10.0.0.4:35357 *:* users:(("haproxy",pid=156879,fd=28)) LISTEN 0 128 10.0.0.14:35357 *:* users:(("httpd",pid=401621,fd=3),("httpd",pid=401620,fd=3),("httpd",pid=401586,fd=3),("httpd",pid=128954,fd=3),("httpd",pid=128953,fd=3),("httpd",pid=128912,fd=3),("httpd",pid=116970,fd=3),("httpd",pid=116969,fd=3),("httpd",pid=116968,fd=3),("httpd",pid=116967,fd=3),("httpd",pid=116966,fd=3),("httpd",pid=116965,fd=3),("httpd",pid=116964,fd=3),("httpd",pid=116963,fd=3),("httpd",pid=116946,fd=3)) Even in an SSL environment, we are indeed exposing both horizon and the admin URL via http. [root@overcloud-controller-0 ~]# grep 10.0.0.4 /etc/haproxy/haproxy.cfg bind 10.0.0.4:13042 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13777 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13776 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13292 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13041 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13004 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1 bind 10.0.0.4:13005 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1 bind 10.0.0.4:13003 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1 bind 10.0.0.4:443 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:80 transparent redirect scheme https code 301 if { hdr(host) -i 10.0.0.4 } !{ ssl_fc } bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:35357 transparent bind 10.0.0.4:13000 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem redirect scheme https code 301 if { hdr(host) -i 10.0.0.4 } !{ ssl_fc } bind 10.0.0.4:13696 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13080 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13774 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:13808 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem As a consequence, we can indeed query the non-http 35357 and obtain a token: [stack@undercloud-1 ~]$ cat overcloudrc export OS_NO_CACHE=True export OS_CLOUDNAME=overcloud export OS_AUTH_URL=https://osp.example.net:13000/v2.0 export NOVA_VERSION=1.1 export COMPUTE_API_VERSION=1.1 export OS_USERNAME=admin export no_proxy=,osp.example.net,osp.example.net export OS_PASSWORD=VncJqqqRuNmCpvbkzmcsf9veG export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available" export OS_TENANT_NAME=admin [stack@undercloud-1 ~]$ curl -i \ > -H "Content-Type: application/json" \ > -d ' > { "auth": { > "identity": { > "methods": ["password"], > "password": { > "user": { > "name": "admin", > "domain": { "id": "default" }, > "password": "adminpwd" > } > } > } > } > }' \ > http://localhost:5000/v3/auth/tokens ; echo curl: (7) Failed connect to localhost:5000; Connection refused [stack@undercloud-1 ~]$ curl -i -H "Content-Type: application/json" -d ' { "auth": { "identity": { "methods": ["password"], "password": { "user": { "name": "admin", "domain": { "id": "default" }, "password": "VncJqqqRuNmCpvbkzmcsf9veG" } } } } }' http://osp.example.net:35357/v3/auth/tokens ; echo HTTP/1.1 201 Created Date: Wed, 28 Jun 2017 14:30:27 GMT Server: Apache X-Subject-Token: 7d7dc3a39b36451b8a15dee2e46a0f45 Vary: X-Auth-Token x-openstack-request-id: req-25916036-7fa2-4ec6-9d40-377fd8836466 Content-Length: 283 Content-Type: application/json {"token": {"issued_at": "2017-06-28T14:30:28.000000Z", "audit_ids": ["RonOtcN2Q4usfrKvufkfZQ"], "methods": ["password"], "expires_at": "2017-06-28T15:30:28.000000Z", "user": {"domain": {"id": "default", "name": "Default"}, "id": "9ef896e2ec794306b0c11e48656fd419", "name": "admin"}}} --- Additional comment from Andreas Karis on 2017-07-05 15:25:35 EDT --- First deployment: ~~~ openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e ${template_base_dir}/network-environment.yaml \ -e ${template_base_dir}/enable-tls.yaml \ -e ${template_base_dir}/cloudname.yaml \ -e ${template_base_dir}/inject-trust-anchor.yaml \ -e ${template_base_dir}/tls-endpoints-public-dns.yaml \ -e ${template_base_dir}/service-net-map.yaml \ --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage \ --control-scale $control_scale --compute-scale $compute_scale --ceph-storage-scale $ceph_scale \ --ntp-server $ntpserver \ --neutron-network-type vxlan --neutron-tunnel-types vxlan ~~~ ~~~ stack@undercloud-4 templates]$ grep KeystoneAd enable-tls.yaml KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'} stack@undercloud-4 templates]$ grep KeystoneAd tls-endpoints-public-dns.yaml KeystoneAdmin: {protocol: 'https', port: '13357', host: 'CLOUDNAME'} ~~~ ~~~ [stack@undercloud-4 templates]$ cat cloudname.yaml parameter_defaults: CloudName: osp.example.net DnsServers: ["192.0.2.1"] PublicVirtualFixedIPs: [{'ip_address':'10.0.0.4'}] ~~~ ~~~ [stack@undercloud-4 templates]$ cat service-net-map.yaml parameter_defaults: ServiceNetMap: KeystoneAdminApiNetwork: external ~~~ Which leads to /etc/haproxy/haproxy.cfg: ~~~ (...) listen keystone_admin bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:35357 transparent mode http http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc } server overcloud-controller-0.external.localdomain 10.0.0.11:35357 check fall 5 inter 2000 rise 2 (...) ~~~ --- Additional comment from Andreas Karis on 2017-07-05 15:25:45 EDT --- Hi, Second deployment - this should cover the default SSL/TLS deployment as per our documentation: ~~~ openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e ${template_base_dir}/network-environment.yaml \ -e ${template_base_dir}/enable-tls.yaml \ -e ${template_base_dir}/cloudname.yaml \ -e ${template_base_dir}/inject-trust-anchor.yaml \ -e ${template_base_dir}/tls-endpoints-public-dns.yaml \ --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage \ --control-scale $control_scale --compute-scale $compute_scale --ceph-storage-scale $ceph_scale \ --ntp-server $ntpserver \ --neutron-network-type vxlan --neutron-tunnel-types vxlan ~~~ ~~~ [stack@undercloud-1 templates]$ grep KeystoneAd enable-tls.yaml KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'} [stack@undercloud-1 templates]$ grep KeystoneAd tls-endpoints-public-dns.yaml KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'} ~~~ Which leads to /etc/haproxy/haproxy.cfg: ~~~ [root@overcloud-controller-0 ~]# cat /etc/haproxy/haproxy.cfg | grep 35357 -B2 -A3 listen keystone_admin bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 192.0.2.14:35357 transparent mode http http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc } server overcloud-controller-0.ctlplane.localdomain 192.0.2.15:35357 check fall 5 inter 2000 rise 2 listen keystone_public bind 10.0.0.4:13000 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem ~~~ The port is now only exposed on the provisioning network: ~~~ [root@overcloud-controller-0 ~]# ss -lntp | grep 35357 LISTEN 0 128 192.0.2.14:35357 *:* users:(("haproxy",pid=158546,fd=28)) LISTEN 0 128 192.0.2.15:35357 *:* users:(("httpd",pid=129493,fd=8),("httpd",pid=129492,fd=8),("httpd",pid=129430,fd=8),("httpd",pid=117868,fd=8),("httpd",pid=117867,fd=8),("httpd",pid=117866,fd=8),("httpd",pid=117865,fd=8),("httpd",pid=117864,fd=8),("httpd",pid=117863,fd=8),("httpd",pid=117862,fd=8),("httpd",pid=117861,fd=8),("httpd",pid=117844,fd=8)) ~~~ So the default deployment is fine. - Andreas --- Additional comment from Andreas Karis on 2017-07-05 15:26:40 EDT --- Given that this does not happen for our default deployments as documented in the OSP 10 guide ... https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/advanced_overcloud_customization/sect-enabling_ssltls_on_the_overcloud ... but only when users follow the following knowledge base articles: https://access.redhat.com/solutions/2891731 https://access.redhat.com/solutions/2943481 I updated both of them with the following text: ++++++++++++++ ### A word of warning ### Moving the Admin Endpoint to the External API Network will expose the non-encrypted `35357` on the external network as well. While communication over `13357` will be encrypted, users might accidentally connect to `35357`. Administrators should make sure that `35357` on the external network is not accessible by users (e.g., create firewall rules). After a deployment with the method described in this article, `/etc/haproxy/haproxy.cfg` will look similar to the following: ~~~ (...) listen keystone_admin bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem bind 10.0.0.4:35357 transparent mode http http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc } server overcloud-controller-0.external.localdomain 10.0.0.11:35357 check fall 5 inter 2000 rise 2 (...) ~~~ ++++++++++++++ We may want to investigate this nevertheless: customers often want the admin endpoint on the External network, and in SSL, we need to be able to move the SSL endpoint only without exposing the http endpoint. --- Additional comment from Juan Antonio Osorio on 2017-08-15 11:40:44 EDT --- Seems to me that the fact that it even tries to serve TLS using the public certificate is a bug by itself. It should only serve on one IP. Also, is it really necessary for it to listen on the external IP? It's not meant for public use. If deployers REALLY need it, we could work it out. But I don't suggest we do this.
This has been submitted upstream for stable/ocata in the following review: https://review.openstack.org/#/c/494947
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0310