I suspect this is a duplicate of the other SSL bug.  Closing this as a duplicate.  If it's not, please reopen

*** This bug has been marked as a duplicate of bug 1242675 ***

BZ 1242675 seems to be concerned with the existence of SSL errors/warnings in the output, but I don't see where what is described there is causing an failure of the undercloud install (but perhaps I've missed something).

I'm running into exactly the situation described here after first having an 'openstack undercloud install' run fail due to a bad SSL certificate (I didn't heed the documentation warning to "Set the Common Name to the undercloud_public_vip address of the provisioning interface"). After correcting the certificate, I've attempted to re-run 'openstack undercloud install' and I consistently get a failure with the above errors and traceback. My understanding is that there is a goal for the installer to be re-runnable without error, so this seems like a valid bug.

FWIW, the errors and traceback are also a perfect match for BZ 1243039, but I believe I've confirmed that I am _not_ hitting the described SELinux problem in my case.

The problem is that the error in this bug is just a generic "something went wrong".  All this means is that for some reason Keystone can't be contacted through HAProxy, and there are any number of potential reasons for that.  Without more details (output of 'journalctl -u haproxy' is probably the place to start) I can't say what is wrong.

Manually deleting the SSL cert and key files from the error does _not_ correct the problem. The same error and backtrace are reported

~~~
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
                                                                                                       /etc/keystone/ssl/certs/ca.pem already exists
                                                                                                                                                    /etc/keystone/ssl/private/signing_key.pem already exists
                                    /etc/keystone/ssl/certs/signing_cert.pem already exists
       Connection to 192.0.2.1 closed.
PKI initialization in init-keystone is deprecated and will be removed.
+ openstack role show ResellerAdmin
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
WARNING: keystoneclient.auth.identity.generic.base Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
ERROR: openstack SSL exception connecting to https://192.0.2.2:13000/v2.0/tokens: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
+ openstack role create ResellerAdmin
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
WARNING: keystoneclient.auth.identity.generic.base Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
ERROR: openstack SSL exception connecting to https://192.0.2.2:13000/v2.0/tokens: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[2015-10-21 17:43:07,424] (os-refresh-config) [ERROR] during post-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/post-configure.d']' returned non-zero exit status 1]

[2015-10-21 17:43:07,425] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 526, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 461, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 297, in _run_live_command
    raise RuntimeError('%s failed. See log for details.', name)
RuntimeError: ('%s failed. See log for details.', 'os-refresh-config')
ERROR: openstack Command 'instack-install-undercloud' returned non-zero exit status 1
~~~

$ sudo journalctl -u haproxy
-- Logs begin at Wed 2015-10-21 13:40:20 EDT, end at Wed 2015-10-21 17:51:07 EDT. --
Oct 21 14:48:28 director1.example.com systemd[1]: Starting HAProxy Load Balancer...
Oct 21 14:48:28 director1.example.com systemd[1]: Started HAProxy Load Balancer.
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: [WARNING] 293/144828 (24739) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: [WARNING] 293/144828 (24739) : config : missing timeouts for proxy 'rabbitmq'.
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: | While not properly invalid, you will certainly encounter various problems
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: | with such a configuration. To fix this, please ensure that all following
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: | timeouts are set to a non-zero value: 'client', 'connect', 'server'.
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: [ALERT] 293/144828 (24739) : sendto logger #1 failed: Resource temporarily unavailable (errno=11)
Oct 21 14:48:28 director1.example.com haproxy-systemd-wrapper[24723]: [ALERT] 293/144828 (24739) : sendto logger #1 failed: Resource temporarily unavailable (errno=11)
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy ceilometer started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy glance_api started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy glance_registry started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy haproxy.stats started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy heat_api started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy ironic started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy keystone_admin started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy keystone_public started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy neutron started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy nova_metadata started.
Oct 21 14:48:28 director1.example.com haproxy[24739]: Proxy nova_osapi started.
Oct 21 14:48:28 director1.example.com haproxy[24740]: Server glance_api/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:28 director1.example.com haproxy[24740]: proxy glance_api has no server available!
Oct 21 14:48:29 director1.example.com haproxy[24740]: Server glance_registry/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:29 director1.example.com haproxy[24740]: proxy glance_registry has no server available!
Oct 21 14:48:29 director1.example.com haproxy[24740]: Server heat_api/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:29 director1.example.com haproxy[24740]: proxy heat_api has no server available!
Oct 21 14:48:29 director1.example.com haproxy[24740]: Server keystone_admin/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:29 director1.example.com haproxy[24740]: proxy keystone_admin has no server available!
Oct 21 14:48:29 director1.example.com haproxy[24740]: Server keystone_public/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:29 director1.example.com haproxy[24740]: proxy keystone_public has no server available!
Oct 21 14:48:29 director1.example.com haproxy[24740]: Server neutron/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:29 director1.example.com haproxy[24740]: proxy neutron has no server available!
Oct 21 14:48:30 director1.example.com haproxy[24740]: Server nova_metadata/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:30 director1.example.com haproxy[24740]: proxy nova_metadata has no server available!
Oct 21 14:48:30 director1.example.com haproxy[24740]: Server nova_osapi/192.0.2.1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Oct 21 14:48:30 director1.example.com haproxy[24740]: proxy nova_osapi has no server available!
Oct 21 14:48:57 director1.example.com haproxy[24740]: Server neutron/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:49:03 director1.example.com haproxy[24740]: Server keystone_admin/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:49:03 director1.example.com haproxy[24740]: Server keystone_public/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:49:23 director1.example.com haproxy[24740]: Server glance_registry/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:49:25 director1.example.com haproxy[24740]: Server glance_api/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:49:48 director1.example.com haproxy[24740]: Server nova_metadata/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:49:48 director1.example.com haproxy[24740]: Server nova_osapi/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:50:33 director1.example.com haproxy[24740]: Server heat_api/192.0.2.1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Oct 21 14:51:27 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40328 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:28 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40333 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:28 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40338 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:29 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40344 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:29 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40349 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:30 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40354 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:31 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40360 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:31 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40365 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:32 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40369 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:55 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40531 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:55 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40533 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:55 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40535 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:55 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40537 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:56 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40543 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:56 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40545 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 14:51:56 director1.example.com haproxy[24740]: Connect from 192.0.2.2:40547 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:05 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49194 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:05 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49197 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:06 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49200 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:07 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49209 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:07 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49214 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:08 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49218 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:08 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49225 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:09 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49232 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:10 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49236 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49395 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49397 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49400 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49402 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49405 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49407 to 192.0.2.2:13000 (keystone_public/TCP)
Oct 21 16:33:33 director1.example.com haproxy[24740]: Connect from 192.0.2.2:49409 to 192.0.2.2:13000 (keystone_public/TCP)

This is readily reproducible for me:

Description of problem:
'openstack undercloud install' fails on re-run after correcting a bad SSL self-signed certificate

Version-Release number of selected component (if applicable):
instack-undercloud-2.1.2-29.el7ost.noarch
instack-0.0.7-1.el7ost.noarch


How reproducible:
Consistently when following the steps in section 3 of the Director Installation and Usage Guide and initially using a bad SSL certificate

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html-single/Director_Installation_and_Usage/index.html#chap-Installing_the_Undercloud


Steps to Reproduce:
1. Follow instructions in section 3 of the guide. When generating the SSL self-signed certificate, use the machine's hostname for the certificate Common Name instead of the undercloud_public_vip as noted in the IMPORTANT note in the doc.
2. Run 'openstack undercloud install' -- This will fail in an expected way because of the bad certificate.
3. Re-create the SSL certificate correctly, with the Common Name set to the undercloud_public_vip
4. Re-run 'openstack undercloud install'

Actual results:
Installation fails

Expected results:
Installation succeeds 

Additional info:

Second failure looks identical to the first. Rebooting the machine and running 'openstack undercloud install' a third time works as expected with a successful install (albeit with a lot of SSL warnings a la BZ 1242675).

Okay, that sounds like a problem with puppet not restarting haproxy after the certificate is changed.  I'm guessing what is happening (and correct me if I'm making any bad assumptions here) is that the bad certificate and the good certificate were both in the same place, so when puppet looks at the configuration it thinks nothing has changed and leaves haproxy running with the old cert.

If that's the case, a workaround would be to change the name of the certificate file and point the configuration at that instead.  I'll probably need to talk to the puppet folks about how to deal with this sort of situation.

Hi, 
I solved this bug restarting haproxy manually: "systemctl restart haproxy.service". Then, if you try to install again it works.

Dan, would you mind updating the product documentation to reflect the needed service restart?

OSP 7 has reached its retirement, please see https://access.redhat.com/errata/RHBA-2018:2362