Bug 1361100

Summary: SSL enabled undercloud doesn't configure the AODH public VIP in haproxy
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: instack-undercloudAssignee: Emilien Macchi <emilien>
Status: CLOSED ERRATA QA Contact: Omri Hochman <ohochman>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 9.0 (Mitaka)CC: bperkins, dbecker, emacchi, jason.dobies, jjoyce, mburns, morazi, pkilambi, rhel-osp-director-maint, sasha
Target Milestone: gaKeywords: Triaged
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: instack-undercloud-4.0.0-11.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-11 11:37:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
install-undercloud.log none

Description Marius Cornea 2016-07-28 10:56:07 UTC
Description of problem:
SSL enabled undercloud doesn't configure the AODH public VIP in haproxy. The result is that AODH connections fail.

Version-Release number of selected component (if applicable):
instack-undercloud-4.0.0-8.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy SSL enabled undercloud
2. openstack catalog show aodh | grep public
|           |   publicURL: https://192.168.0.2:13042 |
3. aodh alarm list

Actual results:
Unable to establish connection to https://192.168.0.2:13042/v2/alarms

Expected results:
aodh connections work as expected.

Additional info:

There is no aodh listen section set up in haproxy.cfg:

[root@undercloud stack]# cat /etc/haproxy/haproxy.cfg 
# This file managed by Puppet
global
  daemon  
  group  haproxy
  log  /dev/log local0
  maxconn  20480
  pidfile  /var/run/haproxy.pid
  ssl-default-bind-ciphers  !SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES
  ssl-default-bind-options  no-sslv3
  user  haproxy

defaults
  log  global
  maxconn  4096
  mode  tcp
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen ceilometer
  bind 192.168.0.2:13777 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8777 transparent
  server 192.168.0.1 192.168.0.1:8777 check fall 5 inter 2000 rise 2

listen glance_api
  bind 192.168.0.2:13292 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:9292 transparent
  server 192.168.0.1 192.168.0.1:9292 check fall 5 inter 2000 rise 2

listen glance_registry
  bind 192.168.0.3:9191 transparent
  server 192.168.0.1 192.168.0.1:9191 check fall 5 inter 2000 rise 2

listen haproxy.stats
  bind 192.168.0.3:1993 transparent
  mode http
  stats enable
  stats uri /
  stats auth admin:e67b6b2d07a8c36b52d8531f00f2634688aeeb6e

listen heat_api
  bind 192.168.0.2:13004 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8004 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  rsprep ^Location:\ http://192.168.0.2(.*) Location:\ https://192.168.0.2\1
  server 192.168.0.1 192.168.0.1:8004 check fall 5 inter 2000 rise 2

listen ironic
  bind 192.168.0.2:13385 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:6385 transparent
  server 192.168.0.1 192.168.0.1:6385 check fall 5 inter 2000 rise 2

listen keystone_admin
  bind 192.168.0.2:13357 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server 192.168.0.1 192.168.0.1:35357 check fall 5 inter 2000 rise 2

listen keystone_public
  bind 192.168.0.2:13000 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:5000 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server 192.168.0.1 192.168.0.1:5000 check fall 5 inter 2000 rise 2

listen neutron
  bind 192.168.0.2:13696 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:9696 transparent
  server 192.168.0.1 192.168.0.1:9696 check fall 5 inter 2000 rise 2

listen nova_metadata
  bind 192.168.0.3:8775 transparent
  server 192.168.0.1 192.168.0.1:8775 check fall 5 inter 2000 rise 2

listen nova_osapi
  bind 192.168.0.2:13774 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8774 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server 192.168.0.1 192.168.0.1:8774 check fall 5 inter 2000 rise 2

listen rabbitmq
  bind 192.168.0.3:5672 transparent
  option tcpka
  timeout client 0
  timeout server 0
  server 192.168.0.1 192.168.0.1:5672 check fall 5 inter 2000 rise 2

listen swift_proxy_server
  bind 192.168.0.2:13808 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8080 transparent
  server 192.168.0.1 192.168.0.1:8080 check fall 5 inter 2000 rise 2

Comment 4 Omri Hochman 2016-07-29 14:06:42 UTC
undercloud upgrade 8.0 -> 9.0 with SSL failed.

We probably going to have an urgent fix fo it : https://review.openstack.org/348893  


the logs :
----------
haproxy failed to start  --> and caused httpd to fail to start . 

[root@undercloud72 ~]# haproxy -f /etc/haproxy/haproxy.cfg
[WARNING] 041/010347 (2674) : config : missing timeouts for proxy 'rabbitmq'.
   | While not properly invalid, you will certainly encounter various problems
   | with such a configuration. To fix this, please ensure that all following
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] 041/010347 (2674) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
[ALERT] 041/010347 (2674) : Starting proxy aodh: cannot bind socket [192.168.0.3:8042]
[ALERT] 041/010347 (2674) : sendto logger #1 failed: Resource temporarily unavailable (errno=11)

--------------------------------------------------------------------------

Undercloud upgrade view: 

04:53:40 Error: /Stage[main]/Swift::Keystone::Auth/Keystone::Resource::Service_identity[swift]/Keystone_user[swift]: Could not evaluate: Execution of '/bin/openstack domain list --quiet --format csv' returned 1: Unable to establish connection to http://192.168.0.1:35357/v3/domains (tried 37, for a total of 170 seconds)
04:56:19 Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: Unable to establish connection to http://192.168.0.1:35357/v3/projects (tried 37, for a total of 170 seconds)
04:56:19 Error: Not managing Keystone_tenant[service] due to earlier Keystone API failures.
04:56:19 Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[service]/ensure: change from absent to present failed: Not managing Keystone_tenant[service] due to earlier Keystone API failures.
04:56:19 Error: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
04:56:19 Error: /Stage[main]/Keystone::Roles::Admin/Keystone_tenant[admin]/ensure: change from absent to present failed: Not managing Keystone_tenant[admin] due to earlier Keystone API failures.
04:56:19 Error: Not managing Keystone_role[admin] due to earlier Keystone API failures.
04:56:19 Error: /Stage[main]/Keystone::Roles::Admin/Keystone_role[admin]/ensure: change from absent to present failed: Not managing Keystone_role[admin] due to earlier Keystone API failures.

Comment 5 Alexander Chuzhoy 2016-07-30 03:31:56 UTC
I'm unable to deploy undercloud with ssl, attaching the install-undercloud.log

Comment 6 Alexander Chuzhoy 2016-07-30 03:33:26 UTC
Created attachment 1185720 [details]
install-undercloud.log

Comment 8 Omri Hochman 2016-07-30 15:49:01 UTC
Unable to reproduce with newest poodle. waiting for the fix to be merged in puddle to in order to switch the bug to Verified.

Comment 9 Omri Hochman 2016-08-04 22:10:25 UTC
Verified with : instack-undercloud-4.0.0-11.el7ost.noarch 

[stack@undercloud72 ~]$ aodh alarm list

ohochman : output of the 'aodh alarm list' is empty < >  ,   but the connection seems to works successfully. 


[stack@undercloud72 ~]$ cat /etc/haproxy/haproxy.cfg 
# This file managed by Puppet
global
  daemon  
  group  haproxy
  log  /dev/log local0
  maxconn  20480
  pidfile  /var/run/haproxy.pid
  ssl-default-bind-ciphers  !SSLv2:kEECDH:kRSA:kEDH:kPSK:+3DES:!aNULL:!eNULL:!MD5:!EXP:!RC4:!SEED:!IDEA:!DES
  ssl-default-bind-options  no-sslv3
  user  haproxy

defaults
  log  global
  maxconn  4096
  mode  tcp
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

listen aodh
  bind 192.168.0.2:13042 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8042 transparent
  server 192.168.0.1 192.168.0.1:8042 check fall 5 inter 2000 rise 2

listen ceilometer
  bind 192.168.0.2:13777 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8777 transparent
  server 192.168.0.1 192.168.0.1:8777 check fall 5 inter 2000 rise 2

listen glance_api
  bind 192.168.0.2:13292 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:9292 transparent
  server 192.168.0.1 192.168.0.1:9292 check fall 5 inter 2000 rise 2

listen glance_registry
  bind 192.168.0.3:9191 transparent
  server 192.168.0.1 192.168.0.1:9191 check fall 5 inter 2000 rise 2

listen haproxy.stats
  bind 192.168.0.3:1993 transparent
  mode http
  stats enable
  stats uri /
  stats auth admin:9520b081400d225c5463eefbe051cfc168f528d4

listen heat_api
  bind 192.168.0.2:13004 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8004 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  rsprep ^Location:\ http://192.168.0.2(.*) Location:\ https://192.168.0.2\1
  server 192.168.0.1 192.168.0.1:8004 check fall 5 inter 2000 rise 2

listen ironic
  bind 192.168.0.2:13385 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:6385 transparent
  server 192.168.0.1 192.168.0.1:6385 check fall 5 inter 2000 rise 2

listen keystone_admin
  bind 192.168.0.2:13357 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:35357 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server 192.168.0.1 192.168.0.1:35357 check fall 5 inter 2000 rise 2

listen keystone_public
  bind 192.168.0.2:13000 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:5000 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server 192.168.0.1 192.168.0.1:5000 check fall 5 inter 2000 rise 2

listen neutron
  bind 192.168.0.2:13696 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:9696 transparent
  server 192.168.0.1 192.168.0.1:9696 check fall 5 inter 2000 rise 2

listen nova_metadata
  bind 192.168.0.3:8775 transparent
  server 192.168.0.1 192.168.0.1:8775 check fall 5 inter 2000 rise 2

listen nova_osapi
  bind 192.168.0.2:13774 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8774 transparent
  mode http
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
  server 192.168.0.1 192.168.0.1:8774 check fall 5 inter 2000 rise 2

listen rabbitmq
  bind 192.168.0.3:5672 transparent
  option tcpka
  timeout client 0
  timeout server 0
  server 192.168.0.1 192.168.0.1:5672 check fall 5 inter 2000 rise 2

listen swift_proxy_server
  bind 192.168.0.2:13808 transparent ssl crt /etc/pki/instack-certs/undercloud.pem
  bind 192.168.0.3:8080 transparent
  server 192.168.0.1 192.168.0.1:8080 check fall 5 inter 2000 rise 2

Comment 11 errata-xmlrpc 2016-08-11 11:37:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html