1486363 – OSP11 -> OSP12 upgrade and during OSP12+SSL deployment: haproxy_init_bundle container fails to start on SSL enabled overcloud because it cannot access certificates

Bug 1486363 - OSP11 -> OSP12 upgrade and during OSP12+SSL deployment: haproxy_init_bundle container fails to start on SSL enabled overcloud because it cannot access certificates

Summary: OSP11 -> OSP12 upgrade and during OSP12+SSL deployment: haproxy_init_bundle ...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-tripleo-heat-templates
Sub Component:
Version:	12.0 (Pike)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	urgent
Target Milestone:	beta
Target Release:	12.0 (Pike)
Assignee:	Angus Thomas
QA Contact:	Prasanth Anbalagan
Docs Contact:
URL:
Whiteboard:
Duplicates (3):	1488352 1488355 1488601 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-08-29 14:58 UTC by Marius Cornea
Modified:	2018-02-05 19:12 UTC (History)
CC List:	20 users (show)
Fixed In Version:	openstack-tripleo-heat-templates-7.0.0-0.20170913050524.0rc2.el7ost
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-12-13 21:58:11 UTC
Target Upstream Version:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1715132	None	None	None	2017-09-05 11:43:08 UTC
OpenStack gerrit	500779	None	MERGED	Mount public certificate in haproxy init container	2021-01-15 19:08:43 UTC
OpenStack gerrit	501127	None	MERGED	Mount public certificate in haproxy init container	2021-01-15 19:09:23 UTC
Red Hat Product Errata	RHEA-2017:3462	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 12.0 Enhancement Advisory	2018-02-16 01:43:25 UTC

Description Marius Cornea 2017-08-29 14:58:34 UTC

Description of problem:
OSP11 -> OSP12 upgrade: haproxy container fails to start on SSL enabled overcloud because it cannot access the SSL certificates:

[root@controller-0 heat-admin]# docker logs -f haproxy_init_bundle
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Info: Loading facts
Failed to get D-Bus connection: Operation not permitted
Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Warning: Undefined variable 'deploy_config_name'; 
   (file & line not available)
Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
Warning: ModuleLoader: module 'haproxy' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
   (file & line not available)
Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 131]:["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/haproxy_bundle.pp", 84]
   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 132]:["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/haproxy_bundle.pp", 84]
   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 135]:["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/haproxy_bundle.pp", 84]
   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Warning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Absolute_Path. There is further documentation for validate_legacy function in the README. at ["/etc/puppet/modules/haproxy/manifests/init.pp", 136]:["/etc/puppet/modules/tripleo/manifests/profile/pacemaker/haproxy_bundle.pp", 84]
   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
Warning: This method is deprecated, please use match expressions with Stdlib::Compat::Ipv6 instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. at ["/etc/puppet/modules/tripleo/manifests/pacemaker/haproxy_with_vip.pp", 62]:
   (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Notice: Compiled catalog for controller-0.localdomain in environment production in 2.75 seconds
Info: Applying configuration version '1504016750'
Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker]/ensure: created
Notice: /Stage[main]/Pacemaker::Corosync/File[etc-pacemaker-authkey]/ensure: defined content as '{md5}a422ca81cddf7f1e7a4fc6c1e1ed1a12'
Info: Class[Pacemaker::Corosync]: Unscheduling all events on Class[Pacemaker::Corosync]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Pacemaker::Resource::Bundle[haproxy-bundle]/Pcmk_bundle[haproxy-bundle]/ensure: created
Info: Pacemaker::Resource::Bundle[haproxy-bundle]: Unscheduling all events on Pacemaker::Resource::Bundle[haproxy-bundle]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_control_vip]/Pacemaker::Constraint::Order[control_vip-then-haproxy]/Pcmk_constraint[order-ip-192.168.24.14-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Order[control_vip-then-haproxy]: Unscheduling all events on Pacemaker::Constraint::Order[control_vip-then-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_control_vip]/Pacemaker::Constraint::Colocation[control_vip-with-haproxy]/Pcmk_constraint[colo-ip-192.168.24.14-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Colocation[control_vip-with-haproxy]: Unscheduling all events on Pacemaker::Constraint::Colocation[control_vip-with-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_public_vip]/Pacemaker::Constraint::Order[public_vip-then-haproxy]/Pcmk_constraint[order-ip-10.0.0.101-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Order[public_vip-then-haproxy]: Unscheduling all events on Pacemaker::Constraint::Order[public_vip-then-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_public_vip]/Pacemaker::Constraint::Colocation[public_vip-with-haproxy]/Pcmk_constraint[colo-ip-10.0.0.101-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Colocation[public_vip-with-haproxy]: Unscheduling all events on Pacemaker::Constraint::Colocation[public_vip-with-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_redis_vip]/Pacemaker::Constraint::Order[redis_vip-then-haproxy]/Pcmk_constraint[order-ip-172.17.1.13-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Order[redis_vip-then-haproxy]: Unscheduling all events on Pacemaker::Constraint::Order[redis_vip-then-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_redis_vip]/Pacemaker::Constraint::Colocation[redis_vip-with-haproxy]/Pcmk_constraint[colo-ip-172.17.1.13-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Colocation[redis_vip-with-haproxy]: Unscheduling all events on Pacemaker::Constraint::Colocation[redis_vip-with-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_internal_api_vip]/Pacemaker::Constraint::Order[internal_api_vip-then-haproxy]/Pcmk_constraint[order-ip-172.17.1.16-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Order[internal_api_vip-then-haproxy]: Unscheduling all events on Pacemaker::Constraint::Order[internal_api_vip-then-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_internal_api_vip]/Pacemaker::Constraint::Colocation[internal_api_vip-with-haproxy]/Pcmk_constraint[colo-ip-172.17.1.16-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Colocation[internal_api_vip-with-haproxy]: Unscheduling all events on Pacemaker::Constraint::Colocation[internal_api_vip-with-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_vip]/Pacemaker::Constraint::Order[storage_vip-then-haproxy]/Pcmk_constraint[order-ip-172.17.3.13-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Order[storage_vip-then-haproxy]: Unscheduling all events on Pacemaker::Constraint::Order[storage_vip-then-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_vip]/Pacemaker::Constraint::Colocation[storage_vip-with-haproxy]/Pcmk_constraint[colo-ip-172.17.3.13-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Colocation[storage_vip-with-haproxy]: Unscheduling all events on Pacemaker::Constraint::Colocation[storage_vip-with-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_mgmt_vip]/Pacemaker::Constraint::Order[storage_mgmt_vip-then-haproxy]/Pcmk_constraint[order-ip-172.17.4.10-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Order[storage_mgmt_vip-then-haproxy]: Unscheduling all events on Pacemaker::Constraint::Order[storage_mgmt_vip-then-haproxy]
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_storage_mgmt_vip]/Pacemaker::Constraint::Colocation[storage_mgmt_vip-with-haproxy]/Pcmk_constraint[colo-ip-172.17.4.10-haproxy-bundle]/ensure: created
Info: Pacemaker::Constraint::Colocation[storage_mgmt_vip-with-haproxy]: Unscheduling all events on Pacemaker::Constraint::Colocation[storage_mgmt_vip-with-haproxy]
Info: Computing checksum on file /etc/haproxy/haproxy.cfg
Info: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]: Filebucketed /etc/haproxy/haproxy.cfg to puppet with sum 1f337186b0e1ba5ee82760cb437fb810
Error: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20170829-8-16mt00u -c' returned 1: [ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:26] : 'bind 10.0.0.101:13776' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:37] : 'bind 10.0.0.101:13292' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:55] : 'bind 10.0.0.101:13004' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:68] : 'bind 10.0.0.101:13005' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:81] : 'bind 10.0.0.101:13003' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:94] : 'bind 10.0.0.101:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:96] : 'bind 172.17.1.16:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:117] : 'bind 10.0.0.101:13000' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:138] : 'bind 10.0.0.101:13696' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:154] : 'bind 10.0.0.101:13080' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:164] : 'bind 10.0.0.101:13774' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:175] : 'bind 10.0.0.101:13778' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:199] : 'bind 10.0.0.101:13386' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:207] : 'bind 10.0.0.101:13808' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg20170829-8-16mt00u
[ALERT] 240/142635 (753) : Proxy 'cinder': no SSL certificate specified for bind '10.0.0.101:13776' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:26] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'glance_api': no SSL certificate specified for bind '10.0.0.101:13292' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:37] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'heat_api': no SSL certificate specified for bind '10.0.0.101:13004' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:55] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'heat_cfn': no SSL certificate specified for bind '10.0.0.101:13005' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:68] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'heat_cloudwatch': no SSL certificate specified for bind '10.0.0.101:13003' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:81] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'horizon': no SSL certificate specified for bind '10.0.0.101:443' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:94] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'horizon': no SSL certificate specified for bind '172.17.1.16:443' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:96] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'keystone_public': no SSL certificate specified for bind '10.0.0.101:13000' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:117] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'neutron': no SSL certificate specified for bind '10.0.0.101:13696' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:138] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'nova_novncproxy': no SSL certificate specified for bind '10.0.0.101:13080' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:154] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'nova_osapi': no SSL certificate specified for bind '10.0.0.101:13774' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:164] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'nova_placement': no SSL certificate specified for bind '10.0.0.101:13778' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:175] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'sahara': no SSL certificate specified for bind '10.0.0.101:13386' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:199] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'swift_proxy_server': no SSL certificate specified for bind '10.0.0.101:13808' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:207] (use 'crt').
[ALERT] 240/142635 (753) : Fatal errors found in configuration.
Error: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: change from {md5}1f337186b0e1ba5ee82760cb437fb810 to {md5}b4de4b751b91639ecaaaf64f317b69d7 failed: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20170829-8-16mt00u -c' returned 1: [ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:26] : 'bind 10.0.0.101:13776' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:37] : 'bind 10.0.0.101:13292' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:55] : 'bind 10.0.0.101:13004' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:68] : 'bind 10.0.0.101:13005' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:81] : 'bind 10.0.0.101:13003' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:94] : 'bind 10.0.0.101:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:96] : 'bind 172.17.1.16:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:117] : 'bind 10.0.0.101:13000' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:138] : 'bind 10.0.0.101:13696' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:154] : 'bind 10.0.0.101:13080' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:164] : 'bind 10.0.0.101:13774' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:175] : 'bind 10.0.0.101:13778' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:199] : 'bind 10.0.0.101:13386' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : parsing [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:207] : 'bind 10.0.0.101:13808' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 240/142635 (753) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg20170829-8-16mt00u
[ALERT] 240/142635 (753) : Proxy 'cinder': no SSL certificate specified for bind '10.0.0.101:13776' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:26] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'glance_api': no SSL certificate specified for bind '10.0.0.101:13292' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:37] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'heat_api': no SSL certificate specified for bind '10.0.0.101:13004' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:55] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'heat_cfn': no SSL certificate specified for bind '10.0.0.101:13005' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:68] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'heat_cloudwatch': no SSL certificate specified for bind '10.0.0.101:13003' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:81] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'horizon': no SSL certificate specified for bind '10.0.0.101:443' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:94] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'horizon': no SSL certificate specified for bind '172.17.1.16:443' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:96] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'keystone_public': no SSL certificate specified for bind '10.0.0.101:13000' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:117] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'neutron': no SSL certificate specified for bind '10.0.0.101:13696' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:138] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'nova_novncproxy': no SSL certificate specified for bind '10.0.0.101:13080' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:154] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'nova_osapi': no SSL certificate specified for bind '10.0.0.101:13774' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:164] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'nova_placement': no SSL certificate specified for bind '10.0.0.101:13778' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:175] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'sahara': no SSL certificate specified for bind '10.0.0.101:13386' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:199] (use 'crt').
[ALERT] 240/142635 (753) : Proxy 'swift_proxy_server': no SSL certificate specified for bind '10.0.0.101:13808' at [/etc/haproxy/haproxy.cfg20170829-8-16mt00u:207] (use 'crt').
[ALERT] 240/142635 (753) : Fatal errors found in configuration.
Notice: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/mode: mode changed '0644' to '0640'
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Resource_restart_flag[haproxy-clone]/File[/var/lib/tripleo]: Dependency File[/etc/haproxy/haproxy.cfg] has failures: true
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Resource_restart_flag[haproxy-clone]/File[/var/lib/tripleo]: Skipping because of failed dependencies
Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Resource_restart_flag[haproxy-clone]/File[/var/lib/tripleo/pacemaker-restarts]: Dependency File[/etc/haproxy/haproxy.cfg] has failures: true
Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Resource_restart_flag[haproxy-clone]/File[/var/lib/tripleo/pacemaker-restarts]: Skipping because of failed dependencies
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Applied catalog in 42.40 seconds

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.0-0.20170821194253.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy OSP11 SSL enabled overcloud:

#!/bin/bash

timeout 180m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sahara.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
--log-file overcloud_deployment_59.log

2. Upgrade to containerized OSP12:

#!/bin/bash

timeout 180m openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sahara.yaml \
--environment-file /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/hostnames.yml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
-e /home/stack/docker-osp12.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
 
Actual results:
Upgrade gets stuck.

Expected results:
Upgrade succeeds.

Additional info:

Comment 2 Jose Luis Franco 2017-09-01 14:45:46 UTC

For some reason, it seems the volume /etc/pki/tls/private/overcloud_endpoint.pem is not being binded to the container. 
However, the binding is implemented in https://github.com/openstack/tripleo-heat-templates/blob/53db241cfbfc1b6a237b7f33486a051aa6934579/docker/services/haproxy.yaml#L118-L120.
I will try to reproduce the bug in my local environment to be able to get more information about the failure.

Comment 3 Alexander Chuzhoy 2017-09-01 16:17:42 UTC

Reproduce the issue during clean deployment of osp12

Comment 4 Martin André 2017-09-01 17:10:56 UTC

I suspect a misconfiguration. In the broken deployment:

[heat-admin@overcloud-controller-0 ~]$ sudo hiera -c /etc/puppet/hiera.yaml "tripleo::haproxy::service_certificate"
/etc/pki/tls/private/overcloud_endpoint.pem

While in my local successful deployment:

[heat-admin@overcloud-controller-0 ~]$ sudo hiera -c /etc/puppet/hiera.yaml "tripleo::haproxy::service_certificate"
/etc/pki/tls/certs/haproxy/overcloud-haproxy-external.pem

Comment 5 Alexander Chuzhoy 2017-09-01 21:31:17 UTC

(In reply to Martin André from comment #4)
> I suspect a misconfiguration. In the broken deployment:
> 
> [heat-admin@overcloud-controller-0 ~]$ sudo hiera -c /etc/puppet/hiera.yaml
> "tripleo::haproxy::service_certificate"
> /etc/pki/tls/private/overcloud_endpoint.pem
> 
> While in my local successful deployment:
> 
> [heat-admin@overcloud-controller-0 ~]$ sudo hiera -c /etc/puppet/hiera.yaml
> "tripleo::haproxy::service_certificate"
> /etc/pki/tls/certs/haproxy/overcloud-haproxy-external.pem



Here's my deployment command:
openstack overcloud deploy --templates \
--libvirt-type kvm \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \
-e /home/stack/templates/nodes_data.yaml \
-e  /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/inject-trust-anchor-hiera.yaml \
-e /home/stack/rhos12.yaml



In the included /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml 

By default we have:
DeployedSSLCertificatePath: /etc/pki/tls/private/overcloud_endpoint.pem

Yet that file doesn't exist on OC nodes.

Comment 6 Marius Cornea 2017-09-05 08:19:09 UTC

Removing the Triaged keywork(from upgrades) since it has been reassigned to containers DFG.

Comment 7 Attila Fazekas 2017-09-05 08:24:59 UTC

*** Bug 1488355 has been marked as a duplicate of this bug. ***

Comment 8 Martin André 2017-09-05 08:42:29 UTC

*** Bug 1488352 has been marked as a duplicate of this bug. ***

Comment 9 Alexander Chuzhoy 2017-09-05 20:01:39 UTC

Confirmed that the gerrit patch works.
For successful deployment also note this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1488552

Comment 10 Martin André 2017-09-06 15:16:39 UTC

The backport is on its way to stable/pike.

https://review.openstack.org/#/c/501127/

Comment 11 Nathan Kinder 2017-09-07 16:44:57 UTC

*** Bug 1488601 has been marked as a duplicate of this bug. ***

Comment 12 Prasanth Anbalagan 2017-09-20 13:02:24 UTC

Is there a build available with the fix?

Comment 13 Nathan Kinder 2017-09-21 19:53:05 UTC

(In reply to Prasanth Anbalagan from comment #12)
> Is there a build available with the fix?

I checked the srpm of the latest build, and it contains the fix for this issue.  The package fixed package is:

    openstack-tripleo-heat-templates-7.0.0-0.20170913050524.0rc2.el7ost

Comment 19 errata-xmlrpc 2017-12-13 21:58:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:3462

Note You need to log in before you can comment on or make changes to this bug.

afazekas
agurenko
ahrechan
bperkins
chjones
dbecker
dyasny
jcoufal
jfrancoa
josorior
jschluet
mandreou
m.andre
mburns
morazi
nkinder
ohochman
panbalag
rhel-osp-director-maint
sasha