Bug 1483750

+++ This bug was initially created as a clone of Bug #1465969 +++

In default SSL deployments, 35357 is exposed as non-SSL on the external network:

$ curl http://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3
{"version": {"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3/", "rel": "self"}]}} 

$ curl https://ci-rhos.centralci.eng.rdu2.redhat.com:35357/v3
curl: (35) SSL received a record that exceeded the maximum permissible length.

I can confirm the same behavior in a lab environment:

[stack@undercloud-1 ~]$ curl http://osp.example.net:35357
{"versions": {"values": [{"status": "stable", "updated": "2016-10-06T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.7", "links": [{"href": "http://osp.example.net:35357/v3/", "rel": "self"}]}, {"status": "deprecated", "updated": "2016-08-04T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://osp.example.net:35357/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}]}}[stack@undercloud-1 ~]$ 
[stack@undercloud-1 ~]$ 


[root@overcloud-controller-0 ~]# ss -lntp | grep 5000
LISTEN     0      128    172.16.2.12:5000                     *:*                   users:(("haproxy",pid=156879,fd=30))
LISTEN     0      128    172.16.2.5:5000                     *:*                   users:(("httpd",pid=401621,fd=4),("httpd",pid=401620,fd=4),("httpd",pid=401586,fd=4),("httpd",pid=128954,fd=4),("httpd",pid=128953,fd=4),("httpd",pid=128912,fd=4),("httpd",pid=116970,fd=4),("httpd",pid=116969,fd=4),("httpd",pid=116968,fd=4),("httpd",pid=116967,fd=4),("httpd",pid=116966,fd=4),("httpd",pid=116965,fd=4),("httpd",pid=116964,fd=4),("httpd",pid=116963,fd=4),("httpd",pid=116946,fd=4))
[root@overcloud-controller-0 ~]# ss -lntp | grep 35357
LISTEN     0      128    10.0.0.4:35357                    *:*                   users:(("haproxy",pid=156879,fd=28))
LISTEN     0      128    10.0.0.14:35357                    *:*                   users:(("httpd",pid=401621,fd=3),("httpd",pid=401620,fd=3),("httpd",pid=401586,fd=3),("httpd",pid=128954,fd=3),("httpd",pid=128953,fd=3),("httpd",pid=128912,fd=3),("httpd",pid=116970,fd=3),("httpd",pid=116969,fd=3),("httpd",pid=116968,fd=3),("httpd",pid=116967,fd=3),("httpd",pid=116966,fd=3),("httpd",pid=116965,fd=3),("httpd",pid=116964,fd=3),("httpd",pid=116963,fd=3),("httpd",pid=116946,fd=3))


Even in an SSL environment, we are indeed exposing both horizon and the admin URL via http.

[root@overcloud-controller-0 ~]# grep 10.0.0.4 /etc/haproxy/haproxy.cfg 
  bind 10.0.0.4:13042 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13777 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13776 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13292 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13041 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13004 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1
  bind 10.0.0.4:13005 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1
  bind 10.0.0.4:13003 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  rsprep ^Location:\ http://10.0.0.4(.*) Location:\ https://10.0.0.4\1
  bind 10.0.0.4:443 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:80 transparent
  redirect scheme https code 301 if { hdr(host) -i 10.0.0.4 } !{ ssl_fc }
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:35357 transparent
  bind 10.0.0.4:13000 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  redirect scheme https code 301 if { hdr(host) -i 10.0.0.4 } !{ ssl_fc }
  bind 10.0.0.4:13696 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13080 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13774 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:13808 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem

As a consequence, we can indeed query the non-http 35357 and obtain a token:

[stack@undercloud-1 ~]$ cat overcloudrc 
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=https://osp.example.net:13000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,osp.example.net,osp.example.net
export OS_PASSWORD=VncJqqqRuNmCpvbkzmcsf9veG
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin
[stack@undercloud-1 ~]$ curl -i \
>   -H "Content-Type: application/json" \
>   -d '
> { "auth": {
>     "identity": {
>       "methods": ["password"],
>       "password": {
>         "user": {
>           "name": "admin",
>           "domain": { "id": "default" },
>           "password": "adminpwd"
>         }
>       }
>     }
>   }
> }' \
>   http://localhost:5000/v3/auth/tokens ; echo
curl: (7) Failed connect to localhost:5000; Connection refused

[stack@undercloud-1 ~]$ curl -i   -H "Content-Type: application/json"   -d '
{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "admin",
          "domain": { "id": "default" },
          "password": "VncJqqqRuNmCpvbkzmcsf9veG"
        }
      }
    }
  }
}'   http://osp.example.net:35357/v3/auth/tokens ; echo
HTTP/1.1 201 Created
Date: Wed, 28 Jun 2017 14:30:27 GMT
Server: Apache
X-Subject-Token: 7d7dc3a39b36451b8a15dee2e46a0f45
Vary: X-Auth-Token
x-openstack-request-id: req-25916036-7fa2-4ec6-9d40-377fd8836466
Content-Length: 283
Content-Type: application/json

{"token": {"issued_at": "2017-06-28T14:30:28.000000Z", "audit_ids": ["RonOtcN2Q4usfrKvufkfZQ"], "methods": ["password"], "expires_at": "2017-06-28T15:30:28.000000Z", "user": {"domain": {"id": "default", "name": "Default"}, "id": "9ef896e2ec794306b0c11e48656fd419", "name": "admin"}}}

--- Additional comment from Andreas Karis on 2017-07-05 15:25:35 EDT ---

First deployment:
~~~
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e ${template_base_dir}/network-environment.yaml \
-e ${template_base_dir}/enable-tls.yaml \
-e ${template_base_dir}/cloudname.yaml \
-e ${template_base_dir}/inject-trust-anchor.yaml \
-e ${template_base_dir}/tls-endpoints-public-dns.yaml \
-e ${template_base_dir}/service-net-map.yaml \
--control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage \
--control-scale $control_scale --compute-scale $compute_scale --ceph-storage-scale $ceph_scale \
--ntp-server $ntpserver \
--neutron-network-type vxlan --neutron-tunnel-types vxlan
~~~

~~~
stack@undercloud-4 templates]$ grep KeystoneAd enable-tls.yaml 
    KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'}
stack@undercloud-4 templates]$ grep KeystoneAd tls-endpoints-public-dns.yaml 
    KeystoneAdmin: {protocol: 'https', port: '13357', host: 'CLOUDNAME'}
~~~

~~~
[stack@undercloud-4 templates]$ cat cloudname.yaml 
parameter_defaults:
  CloudName: osp.example.net
  DnsServers: ["192.0.2.1"]
  PublicVirtualFixedIPs: [{'ip_address':'10.0.0.4'}]
~~~

~~~
[stack@undercloud-4 templates]$ cat service-net-map.yaml 
parameter_defaults:
  ServiceNetMap:
    KeystoneAdminApiNetwork: external
~~~

Which leads to /etc/haproxy/haproxy.cfg:
~~~
(...)
listen keystone_admin
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0.external.localdomain 10.0.0.11:35357 check fall 5 inter 2000 rise 2
(...)
~~~

--- Additional comment from Andreas Karis on 2017-07-05 15:25:45 EDT ---

Hi,

Second deployment - this should cover the default SSL/TLS deployment as per our documentation:
~~~
openstack overcloud deploy --templates \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e ${template_base_dir}/network-environment.yaml \
-e ${template_base_dir}/enable-tls.yaml \
-e ${template_base_dir}/cloudname.yaml \
-e ${template_base_dir}/inject-trust-anchor.yaml \
-e ${template_base_dir}/tls-endpoints-public-dns.yaml \
--control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage \
--control-scale $control_scale --compute-scale $compute_scale --ceph-storage-scale $ceph_scale \
--ntp-server $ntpserver \
--neutron-network-type vxlan --neutron-tunnel-types vxlan
~~~

~~~
[stack@undercloud-1 templates]$  grep KeystoneAd enable-tls.yaml
    KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'}
[stack@undercloud-1 templates]$  grep KeystoneAd tls-endpoints-public-dns.yaml 
    KeystoneAdmin: {protocol: 'http', port: '35357', host: 'IP_ADDRESS'}
~~~

Which leads to /etc/haproxy/haproxy.cfg:
~~~
[root@overcloud-controller-0 ~]# cat /etc/haproxy/haproxy.cfg | grep 35357 -B2 -A3
listen keystone_admin
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 192.0.2.14:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0.ctlplane.localdomain 192.0.2.15:35357 check fall 5 inter 2000 rise 2

listen keystone_public
  bind 10.0.0.4:13000 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
~~~

The port is now only exposed on the provisioning network:
~~~
[root@overcloud-controller-0 ~]# ss -lntp | grep 35357
LISTEN     0      128    192.0.2.14:35357                    *:*                   users:(("haproxy",pid=158546,fd=28))
LISTEN     0      128    192.0.2.15:35357                    *:*                   users:(("httpd",pid=129493,fd=8),("httpd",pid=129492,fd=8),("httpd",pid=129430,fd=8),("httpd",pid=117868,fd=8),("httpd",pid=117867,fd=8),("httpd",pid=117866,fd=8),("httpd",pid=117865,fd=8),("httpd",pid=117864,fd=8),("httpd",pid=117863,fd=8),("httpd",pid=117862,fd=8),("httpd",pid=117861,fd=8),("httpd",pid=117844,fd=8))
~~~

So the default deployment is fine.

- Andreas

--- Additional comment from Andreas Karis on 2017-07-05 15:26:40 EDT ---

Given that this does not happen for our default deployments as documented in the OSP 10 guide ...
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/advanced_overcloud_customization/sect-enabling_ssltls_on_the_overcloud

... but only when users follow the following knowledge base articles:
https://access.redhat.com/solutions/2891731
https://access.redhat.com/solutions/2943481

I updated both of them with the following text:
++++++++++++++
### A word of warning ###
Moving the Admin Endpoint to the External API Network will expose the non-encrypted `35357` on the external network as well. While communication over `13357` will be encrypted, users might accidentally connect to `35357`. Administrators should make sure that `35357` on the external network is not accessible by users (e.g., create firewall rules).

After a deployment with the method described in this article, `/etc/haproxy/haproxy.cfg` will look similar to the following:
~~~
(...)
listen keystone_admin
  bind 10.0.0.4:13357 transparent ssl crt /etc/pki/tls/private/overcloud_endpoint.pem
  bind 10.0.0.4:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server overcloud-controller-0.external.localdomain 10.0.0.11:35357 check fall 5 inter 2000 rise 2
(...)
~~~
++++++++++++++


We may want to investigate this nevertheless: customers often want the admin endpoint on the External network, and in SSL, we need to be able to move the SSL endpoint only without exposing the http endpoint.

--- Additional comment from Juan Antonio Osorio on 2017-08-15 11:40:44 EDT ---

Seems to me that the fact that it even tries to serve TLS using the public certificate is a bug by itself. It should only serve on one IP. Also, is it really necessary for it to listen on the external IP? It's not meant for public use.

If deployers REALLY need it, we could work it out. But I don't suggest we do this.