Bug 1719354

Running down the issue currently it's looking like the rebase pulled in cert bundle checking, which dropped the default certs path/bundle that used to work.

This is the usage in OSP that broke usage of python-websocket-client.  There is a workaround if you export WEBSOCKET_CLIENT_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt  or the right path for your particular cert should resolve the verification issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1702715#c12

notes:

 As per mailing thread, this package fails QE since it requires the workaround: export WEBSOCKET_CLIENT_CA_BUNDLE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem otherwise deployment fails at the beginning with [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) error message. Once worked around though the deployment passes further with web sockets endpoint set to IPv6

Created attachment 1579429 [details]
potential fix

I can confirm basic run of OSP14 with this package (without IR workaround in place) passed. Package was installed after UC installation was done and images were prepared:

2019-06-11 18:56:32.820 | --> Running transaction check
2019-06-11 18:56:32.823 | ---> Package python-websocket-client.noarch 0:0.56.0-1.git3c25814.el7 will be updated
2019-06-11 18:56:32.826 | ---> Package python-websocket-client.noarch 0:0.56.0-2.git3c25814.el7 will be an update
2019-06-11 18:56:32.829 | --> Finished Dependency Resolution

The earlier 0.32 version of websocket-client handled things quite differently:

@@ -143,12 +242,19 @@
 
 def _ssl_socket(sock, user_sslopt, hostname):
     sslopt = dict(cert_reqs=ssl.CERT_REQUIRED)
-    certPath = os.path.join(
-        os.path.dirname(__file__), "cacert.pem")
-    if os.path.isfile(certPath):
-        sslopt['ca_certs'] = certPath
     sslopt.update(user_sslopt)
-    check_hostname = sslopt["cert_reqs"] != ssl.CERT_NONE and sslopt.pop('check_hostname', True)
+
+    certPath = os.environ.get('WEBSOCKET_CLIENT_CA_BUNDLE')
+    if certPath and os.path.isfile(certPath) \
+            and user_sslopt.get('ca_certs', None) is None \
+            and user_sslopt.get('ca_cert', None) is None:
+        sslopt['ca_certs'] = certPath
+    elif certPath and os.path.isdir(certPath) \
+            and user_sslopt.get('ca_cert_path', None) is None:
+        sslopt['ca_cert_path'] = certPath
+
+    check_hostname = sslopt["cert_reqs"] != ssl.CERT_NONE and sslopt.pop(
+        'check_hostname', True)
 
     if _can_use_sni():
         sock = _wrap_sni_socket(sock, sslopt, hostname, check_hostname)

The prior version appeared to ship its own cacert.pem, which 0.56 fixes, however, we need it to use the certificates shipped as part of RHEL - and/or the ones installed by system administrators.

Hello,
As per comment#16 I tested with package version python-websocket-client-0.56.0-3.git3c25814.el7.noarch installed manually after undercloud deployment.

root@uc $ sudo curl -O http://download.eng.bos.redhat.com/brewroot/vol/rhel-7/packages/python-websocket-client/0.56.0/3.git3c25814.el7/noarch/python-websocket-client-0.56.0-3.git3c25814.el7.noarch.rpm && sudo yum install -y python-websocket-client-0.56.0-3.git3c25814.el7.noarch.rpm
...
2019-06-13 11:01:54.960 |   Updating   : python-websocket-client-0.56.0-3.git3c25814.el7.noarch       1/2 
2019-06-13 11:01:54.966 |   Cleanup    : python-websocket-client-0.56.0-2.git3c25814.el7.noarch       2/2 
2019-06-13 11:01:54.969 |   Verifying  : python-websocket-client-0.56.0-3.git3c25814.el7.noarch       1/2 
2019-06-13 11:01:54.986 |   Verifying  : python-websocket-client-0.56.0-2.git3c25814.el7.noarch       2/2

Node import for introspection works now as expected:

root@uc $ source ~/stackrc
root@uc $ openstack overcloud node import --instance-boot-option=local /home/stack/instackenv.json

3 node(s) successfully moved to the "manageable" state.
2019-06-13 11:05:52.243 | Successfully registered node UUID 82dff966-e116-4c19-98cf-e22d2ae9c11f
2019-06-13 11:05:52.246 | Successfully registered node UUID 5fc92524-ae5b-417c-807c-e024fd542dd1
2019-06-13 11:05:52.248 | Successfully registered node UUID 69316184-f8dc-4de7-9a94-6292e6f4ce4e

# 0

#STDERR: 
#Waiting for messages on queue 'tripleo' with no timeout.

On this short notice, I was able to test OSP14+vxlan+ipv4/ipv6+ssl(as it is by default), OSP13+vxlan+ipv6+ssl minimal topologies (+ceph). All of them are able to pass OC deployment. Moreover, since
we were able to pass also with the previous version (0:0.56.0-2.git3c25814.el7, https://bugzilla.redhat.com/show_bug.cgi?id=1719354#c9) I consider this fix verified - though I will
also test OSP10 and run full Tempest testing to be sure, we also need to check the status of upgrade jobs (note these all remaining steps will take significantly more time), but these simple
OSP deployments seem to be passing.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1472