Bug 1702918

Summary: zaqar websocket SSL cert name is hardcoded
Product: Red Hat OpenStack Reporter: Kamil Sambor <ksambor>
Component: python-tripleoclientAssignee: Kamil Sambor <ksambor>
Status: CLOSED ERRATA QA Contact: Filip Hubík <fhubik>
Severity: medium Docs Contact:
Priority: medium    
Version: 14.0 (Rocky)CC: cgoncalves, emacchi, fhubik, hbrock, jslagle, mburns, pkomarov
Target Milestone: z2Keywords: Reopened, Triaged, ZStream
Target Release: 14.0 (Rocky)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-tripleoclient-10.6.1-0.20190303235861.ab72340.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-07 10:53:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1697794, 1699612    

Description Kamil Sambor 2019-04-25 08:03:58 UTC 
in bug 1791970 / https://review.openstack.org/#/c/601677/ the path to the CA cert used by zaqar is hardcoded to /etc/pki/ca-trust/source/anchors/cm-local-ca.pem. this conflicts with the use case when one is using the "undercloud_service_certificate" parameter in undercloud.conf which seemingly should set up a different path for this. I'm observing the use of this parameter when using infrared to generate an RDO undercloud.

Per https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html, "However, it is possible to not use certmonger’s local CA.". When I use infrared to make an undercloud, it creates a new self-signed CA defaulting to /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem; this code is at https://github.com/redhat-openstack/infrared/blob/master/plugins/tripleo-undercloud/tasks/ssl.yml#L32 . Then it sets "undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem" in undercloud.conf and deploys the undercloud, following the instructions in https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html.

However, when I go to introspect nodes, I get the stack trace described in bug 1791970:

Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read.
# ...
 File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

to repair this, I just copy the undercloud-cacert.pem on top of the cm-local-ca.pem file:

[root@undercloud-0 ~]# cd /etc/pki/ca-trust/source/anchors/
[root@undercloud-0 anchors]# ls
cm-local-ca.pem undercloud-cacert.pem
[root@undercloud-0 anchors]# cp cm-local-ca.pem cm-local-ca.pem.saved
[root@undercloud-0 anchors]# cp undercloud-cacert.pem cm-local-ca.pem
cp: overwrite ‘cm-local-ca.pem’? y

I'm not sure how this should work, if tripleo is doing the wrong thing or if infrared is but it seems one side or the other needs to be changed.
Comment 1 Emilien Macchi 2019-04-25 12:36:58 UTC 
*** Bug 1703031 has been marked as a duplicate of this bug. ***
Comment 3 Filip Hubík 2019-04-29 07:43:35 UTC 
Verified by CI, phase1/2 jobs are passing with fixed package included.
Comment 6 errata-xmlrpc 2019-04-30 17:51:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0878
Comment 7 pkomarov 2019-06-06 12:43:05 UTC 
Still seeing at : 
https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/updates/job/DFG-pidone-updates-14_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-sanity/24/console


TASK [register hosts to instack and configure boot] ****************************
task path: /home/rhos-ci/jenkins/workspace/DFG-pidone-updates-14_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-sanity/infrared/plugins/tripleo-overcloud/introspect.yml:79
Thursday 06 June 2019  06:35:46 +0000 (0:00:00.142)       0:02:36.776 ********* 
fatal: [undercloud-0]: FAILED! => {
    "changed": true, 
    "cmd": "set -eo pipefail\n source ~/stackrc\n\n  openstack overcloud node import --instance-boot-option=local /home/stack/instackenv.json\n ", 
    "delta": "0:00:03.073003", 
    "end": "2019-06-06 02:35:50.415534", 
    "rc": 1, 
    "start": "2019-06-06 02:35:47.342531"
}

STDERR:

Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read.
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
Comment 8 pkomarov 2019-06-06 12:43:54 UTC 
logs at : https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/pidone/view/updates/job/DFG-pidone-updates-14_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-sanity/24/artifact/