Description of problem:
A user will not get functional environment if he will decide to deploy overcloud with internal TLS [1] and will also decide to set KernelDisableIPv6 flag.
This issue will occur because we implement internal TLS for some services by putting httpd proxy between openstack endpoint and haproxy. For example, for neutron-server we will:
- start neutron_server_tls_proxy containers on every controller node
- start httpd inside this container with configuration [2]
- configure neutron to listen on localhost [3]
As a result, we will get the following set of servers listening on neutron-server port:
[root@controller-0 ~]# netstat -tupln | grep 9696
tcp 0 0 172.17.1.23:9696 0.0.0.0:* LISTEN 110860/httpd
tcp 0 0 172.17.1.101:9696 0.0.0.0:* LISTEN 71254/haproxy
tcp6 0 0 ::1:9696 :::* LISTEN 113384/python2
As you can see, neutron-server uses IPv6 localhost address to process requests that were proxied by httpd. This scheme works great unless user decides to tune kernel networking settings. In our case customer enabled KernelDisableIPv6 flag.
As a result, httpd failed to proxy requests and generated the following errors:
[Wed Apr 03 18:43:33.927156 2019] [proxy:error] [pid 16] (111)Connection refused: AH00957: HTTP: attempt to connect to 127.0.0.1:9696 (localhost) failed
[Wed Apr 03 18:43:33.927188 2019] [proxy:error] [pid 16] AH00959: ap_proxy_connect_backend disabling worker for (localhost) for 10s
[Wed Apr 03 18:43:33.927194 2019] [proxy_http:error] [pid 16] [client 10.164.227.136:39478] AH01114: HTTP: failed to make connection to backend: localhost
[Wed Apr 03 18:43:25.913045 2019] [proxy:error] [pid 16] AH00940: HTTP: disabled connection for (localhost)
Setting bug's severity to high as important customer is affected.
[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-enabling_internal_ssltls_on_the_overcloud
[2]
[root@controller-0 ~]# cat /var/lib/config-data/puppet-generated/neutron/etc/httpd/conf.d/25-neutron-api-proxy.conf
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
<VirtualHost 172.17.1.23:9696>
ServerName controller-0.internalapi.redhat.local
## Logging
ErrorLog "/var/log/httpd/neutron-api-proxy_error_ssl.log"
ServerSignature Off
CustomLog "/var/log/httpd/neutron-api-proxy_access_ssl.log" combined
## Request header rules
## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
RequestHeader set X-Forwarded-Proto "https"
## Proxy rules
ProxyRequests Off
ProxyPreserveHost Off
ProxyPass / http://localhost:9696/ retry=10
ProxyPassReverse / http://localhost:9696/
## SSL directives
SSLEngine on
SSLCertificateFile "/etc/pki/tls/certs/httpd/httpd-internal_api.crt"
SSLCertificateKeyFile "/etc/pki/tls/private/httpd/httpd-internal_api.key"
</VirtualHost>
[3]
[root@controller-0 ~]# grep bind_host /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf
#bind_host = 0.0.0.0
bind_host=localhost