Bug 1320293

Summary: rhel-osp-director: 7.3->8.0 undercloud+ssl upgrade fails: /bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED]
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED NOTABUG QA Contact: Arik Chernetsky <achernet>
Severity: high Docs Contact:
Priority: high    
Version: 8.0 (Liberty)CC: bnemec, dbecker, jcoufal, mburns, mcornea, morazi, rhel-osp-director-maint
Target Milestone: ga   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-23 14:10:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
install-undercloud.log none

Description Alexander Chuzhoy 2016-03-22 18:56:30 UTC 
rhel-osp-director: 7.3->8.0 undercloud+ssl upgrade fails: /bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED]


Environment:
instack-undercloud-2.2.6-1.el7ost.noarch

Steps to reproduce:
1. Deploy 7.3 with undercloud+ssl 
2. Attempt to upgrade the undercloud to 8.0 and run "openstack undercloud install".

Result:
Notice: /Stage[main]/Apache::Service/Service[httpd]: Triggered 'refresh' from 3 events
Error: /Stage[main]/Neutron::Keystone::Auth/Keystone::Resource::Service_identity[neutron]/Keystone_user[neutron]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)                                                                                    
Error: /Stage[main]/Heat::Keystone::Auth/Keystone::Resource::Service_identity[heat]/Keystone_user[heat]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)                                                                                             
Error: /Stage[main]/Aodh::Keystone::Auth/Keystone::Resource::Service_identity[aodh]/Keystone_user[aodh]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)                                                                                             
Error: /Stage[main]/Nova::Keystone::Auth/Keystone::Resource::Service_identity[nova service, user nova]/Keystone_user[nova]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)                                                                          
Error: /Stage[main]/Glance::Keystone::Auth/Keystone::Resource::Service_identity[glance]/Keystone_user[glance]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)                                                                                       
Error: /Stage[main]/Ironic::Keystone::Auth/Keystone::Resource::Service_identity[ironic]/Keystone_user[ironic]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)                                                                                       
Error: /Stage[main]/Ceilometer::Keystone::Auth/Keystone::Resource::Service_identity[ceilometer]/Keystone_user[ceilometer]: Could not evaluate: Execution of '/bin/openstack token issue --format value' returned 1: SSL exception connecting to https://192.0.2.2:13000/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)   


Expected result:
No errors.
Comment 2 Alexander Chuzhoy 2016-03-22 18:57:50 UTC 
Created attachment 1139195 [details]
install-undercloud.log
Comment 3 Ben Nemec 2016-03-22 19:46:30 UTC 
I would guess that this is because we don't set OS_CACERT in stackrc anymore.  The cacert file needs to be installed in the undercloud trust store before running an upgrade.  The same thing has to happen for new installations of 8.0 using SSL.

What needs to be run is:

sudo cp cacert.pem /etc/pki/ca-trust/source/anchors/
sudo update-ca-trust extract

This is documented upstream in http://docs.openstack.org/developer/tripleo-docs/installation/installation.html#installing-the-undercloud
Comment 4 Jaromir Coufal 2016-03-23 14:08:07 UTC 
No blocker, documentation needed. Can we address the automation of that? How difficult it would be?
Comment 5 Mike Burns 2016-03-23 14:10:07 UTC 
I'll roll this into our documentation.  This is a one time change for upgrades from 7 to 8, so no need for future automation.