Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be available on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1483749 - Admin URL is exposed as non-SSL (35357) after SSL deployment
Summary: Admin URL is exposed as non-SSL (35357) after SSL deployment
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: puppet-tripleo
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z4
: 11.0 (Ocata)
Assignee: RHOS Maint
QA Contact: Prasanth Anbalagan
URL:
Whiteboard:
Depends On: 1465969 1483750
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-21 20:56 UTC by Nathan Kinder
Modified: 2020-08-13 09:43 UTC (History)
16 users (show)

Fixed In Version: puppet-tripleo-6.5.4-1.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1465969
Environment:
Last Closed: 2018-02-13 16:31:14 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 493937 0 None MERGED Remove extra keystone admin haproxy listen and allow TLS 2020-01-27 15:31:32 UTC
OpenStack gerrit 494947 0 None MERGED Remove extra keystone admin haproxy listen and allow TLS 2020-01-27 15:31:32 UTC
Red Hat Product Errata RHBA-2018:0310 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 11.0 director Bug Fix Advisory 2018-02-14 00:13:02 UTC

Description Nathan Kinder 2017-08-21 20:56:58 UTC
+++ This bug was initially created as a clone of Bug #1465969 +++

In default SSL deployments, 35357 is exposed as non-SSL on the external network:

$ curl http://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3
{"version": {"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3/", "rel": "self"}]}} 

$ curl https://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3
curl: (35) SSL received a record that exceeded the maximum permissible length.

I can confirm the same behavior in a lab environment:

[stack@undercloud-1 ~]$ curl http://osp.example.net:35357
{"versions": {"values": [{"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://osp.example.net:35357/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://osp.example.net:35357/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}[stack@undercloud-1 ~]$ 
[stack@undercloud-1 ~]$ 


[root@overcloud-controller-0 ~]# ss -lntp | grep 5000
LISTEN     0      128    172.16.2.12:5000                     *:*                   users:(("haproxy",pid=156879,fd=30))
LISTEN     0      128    172.16.2.5:5000                     *:*                   users:(("httpd",pid=401621,fd=4),("httpd",pid=401620,fd=4),("httpd",pid=401586,fd=4),("httpd",pid=128954,fd=4),("httpd",pid=128953,fd=4),("httpd",pid=128912,fd=4),("httpd",pid=116970,fd=4),("httpd",pid=116969,fd=4),("httpd",pid=116968,fd=4),("httpd",pid=116967,fd=4),("httpd",pid=116966,fd=4),("httpd",pid=116965,fd=4),("httpd",pid=116964,fd=4),("httpd",pid=116963,fd=4),("httpd",pid=116946,fd=4))
[root@overcloud-controller-0 ~]# ss -lntp | grep 35357
LISTEN     0      128    10.0.0.4:35357                    *:*                   users:(("haproxy",pid=156879,fd=28))
LISTEN     0      128    10.0.0.14:35357                    *:*                   users:(("httpd",pid=401621,fd=3),("httpd",pid=401620,fd=3),("httpd",pid=401586,fd=3),("httpd",pid=128954,fd=3),("httpd",pid=128953,fd=3),("httpd",pid=128912,fd=3),("httpd",pid=116970,fd=3),("httpd",pid=116969,fd=3),("httpd",pid=116968,fd=3),("httpd",pid=116967,fd=3),("httpd",pid=116966,fd=3),("httpd",pid=116965,fd=3),("httpd",pid=116964,fd=3),("httpd",pid=116963,fd=3),("httpd",pid=116946,fd=3))


Even in an SSL environment, we are indeed exposing both horizon and the admin URL via http.

[root@overcloud-controller-0 ~]# grep 10.0.0.4 /etc/haproxy/haproxy.cfg 
  bind 10.0.0.4:13042 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13777 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13776 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13292 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13041 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13004 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1
  bind 10.0.0.4:13005 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1
  bind 10.0.0.4:13003 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1
  bind 10.0.0.4:443 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:80 transparent
  redirect scheme https code 301 if { hdr(host) -i 10.0.0.4 } !{ ssl_fc }
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:35357 transparent
  bind 10.0.0.4:13000 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  redirect scheme https code 301 if { hdr(host) -i 10.0.0.4 } !{ ssl_fc }
  bind 10.0.0.4:13696 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13080 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13774 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13808 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem

As a consequence, we can indeed query the non-http 35357 and obtain a token:

[stack@undercloud-1 ~]$ cat overcloudrc 
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=https://osp.example.net:13000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,osp.example.net,osp.example.net
export OS_PASSWORD=VncJqqqRuNmCpvbkzmcsf9veG
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin
[stack@undercloud-1 ~]$ curl -i \
>   -H "Content-Type: application/json" \
>   -d '
> { "auth": {
>     "identity": {
>       "methods": ["password"],
>       "password": {
>         "user": {
>           "name": "admin",
>           "domain": { "id": "default" },
>           "password": "adminpwd"
>         }
>       }
>     }
>   }
> }' \
>   http://localhost:5000/v3/auth/tokens ; echo
curl: (7) Failed connect to localhost:5000; Connection refused

[stack@undercloud-1 ~]$ curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "id": "default" },
          "password": "VncJqqqRuNmCpvbkzmcsf9veG"
        }
      }
    }
  }
}'   http://osp.example.net:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Wed, 28 Jun 2017 14:30:27 GMT
Server: Apache
X-Subject-Token: 7d7dc3a39b36451b8a15dee2e46a0f45
Vary: X-Auth-Token
x-openstack-request-id: req-25916036-7fa2-4ec6-9d40-377fd8836466
Content-Length: 283
Content-Type: application/json

{"token": {"issued_at": "2017-06-28T14:30:28.000000Z", "audit_ids": ["RonOtcN2Q4usfrKvufkfZQ"], "methods": ["password"], "expires_at": "2017-06-28T15:30:28.000000Z", "user": {"domain": {"id": "default", "name": "Default"}, "id": "9ef896e2ec794306b0c11e48656fd419", "name": "admin"}}}

--- Additional comment from Andreas Karis on 2017-07-05 15:25:35 EDT ---

First deployment:
~~~
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e ${template_base_dir}/network-environment.yaml \
-e ${template_base_dir}/enable-tls.yaml \
-e ${template_base_dir}/cloudname.yaml \
-e ${template_base_dir}/inject-trust-anchor.yaml \
-e ${template_base_dir}/tls-endpoints-public-dns.yaml \
-e ${template_base_dir}/service-net-map.yaml \
--control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage \
--control-scale $control_scale --compute-scale $compute_scale --ceph-storage-scale $ceph_scale \
--ntp-server $ntpserver \
--neutron-network-type vxlan --neutron-tunnel-types vxlan
~~~

~~~
stack@undercloud-4 templates]$ grep KeystoneAd enable-tls.yaml 
    KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'}
stack@undercloud-4 templates]$ grep KeystoneAd tls-endpoints-public-dns.yaml 
    KeystoneAdmin: {protocol: 'https', port: '13357', host: 'CLOUDNAME'}
~~~

~~~
[stack@undercloud-4 templates]$ cat cloudname.yaml 
parameter_defaults:
  CloudName: osp.example.net
  DnsServers: ["192.0.2.1"]
  PublicVirtualFixedIPs: [{'ip_address':'10.0.0.4'}]
~~~

~~~
[stack@undercloud-4 templates]$ cat service-net-map.yaml 
parameter_defaults:
  ServiceNetMap:
    KeystoneAdminApiNetwork: external
~~~

Which leads to /etc/haproxy/haproxy.cfg:
~~~
(...)
listen keystone_admin
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0.external.localdomain 10.0.0.11:35357 check fall 5 inter 2000 rise 2
(...)
~~~

--- Additional comment from Andreas Karis on 2017-07-05 15:25:45 EDT ---

Hi,

Second deployment - this should cover the default SSL/TLS deployment as per our documentation:
~~~
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e ${template_base_dir}/network-environment.yaml \
-e ${template_base_dir}/enable-tls.yaml \
-e ${template_base_dir}/cloudname.yaml \
-e ${template_base_dir}/inject-trust-anchor.yaml \
-e ${template_base_dir}/tls-endpoints-public-dns.yaml \
--control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage \
--control-scale $control_scale --compute-scale $compute_scale --ceph-storage-scale $ceph_scale \
--ntp-server $ntpserver \
--neutron-network-type vxlan --neutron-tunnel-types vxlan
~~~

~~~
[stack@undercloud-1 templates]$  grep KeystoneAd enable-tls.yaml
    KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'}
[stack@undercloud-1 templates]$  grep KeystoneAd tls-endpoints-public-dns.yaml 
    KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'}
~~~

Which leads to /etc/haproxy/haproxy.cfg:
~~~
[root@overcloud-controller-0 ~]# cat /etc/haproxy/haproxy.cfg | grep 35357 -B2 -A3
listen keystone_admin
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 192.0.2.14:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0.ctlplane.localdomain 192.0.2.15:35357 check fall 5 inter 2000 rise 2

listen keystone_public
  bind 10.0.0.4:13000 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
~~~

The port is now only exposed on the provisioning network:
~~~
[root@overcloud-controller-0 ~]# ss -lntp | grep 35357
LISTEN     0      128    192.0.2.14:35357                    *:*                   users:(("haproxy",pid=158546,fd=28))
LISTEN     0      128    192.0.2.15:35357                    *:*                   users:(("httpd",pid=129493,fd=8),("httpd",pid=129492,fd=8),("httpd",pid=129430,fd=8),("httpd",pid=117868,fd=8),("httpd",pid=117867,fd=8),("httpd",pid=117866,fd=8),("httpd",pid=117865,fd=8),("httpd",pid=117864,fd=8),("httpd",pid=117863,fd=8),("httpd",pid=117862,fd=8),("httpd",pid=117861,fd=8),("httpd",pid=117844,fd=8))
~~~

So the default deployment is fine.

- Andreas

--- Additional comment from Andreas Karis on 2017-07-05 15:26:40 EDT ---

Given that this does not happen for our default deployments as documented in the OSP 10 guide ...
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/advanced_overcloud_customization/sect-enabling_ssltls_on_the_overcloud

... but only when users follow the following knowledge base articles:
https://access.redhat.com/solutions/2891731
https://access.redhat.com/solutions/2943481

I updated both of them with the following text:
++++++++++++++
### A word of warning ###
Moving the Admin Endpoint to the External API Network will expose the non-encrypted `35357` on the external network as well. While communication over `13357` will be encrypted, users might accidentally connect to `35357`. Administrators should make sure that `35357` on the external network is not accessible by users (e.g., create firewall rules).

After a deployment with the method described in this article, `/etc/haproxy/haproxy.cfg` will look similar to the following:
~~~
(...)
listen keystone_admin
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0.external.localdomain 10.0.0.11:35357 check fall 5 inter 2000 rise 2
(...)
~~~
++++++++++++++


We may want to investigate this nevertheless: customers often want the admin endpoint on the External network, and in SSL, we need to be able to move the SSL endpoint only without exposing the http endpoint.

--- Additional comment from Juan Antonio Osorio on 2017-08-15 11:40:44 EDT ---

Seems to me that the fact that it even tries to serve TLS using the public certificate is a bug by itself. It should only serve on one IP. Also, is it really necessary for it to listen on the external IP? It's not meant for public use.

If deployers REALLY need it, we could work it out. But I don't suggest we do this.

Comment 1 Nathan Kinder 2017-08-21 20:58:46 UTC
This has been submitted upstream for stable/ocata in the following review:

  https://review.openstack.org/#/c/494947

Comment 8 errata-xmlrpc 2018-02-13 16:31:14 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0310


Note You need to log in before you can comment on or make changes to this bug.