Description of problem: tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_expect_continue[id-84dafe57-9666-4f6d-84c8-0814d37923b8] fails only on cert, the other object tests accepted the cert. Probably it uses different cert verification method. The issue only seen on el9, not on el8. Both rgw and swift. The cert subject is an IP in the CN. Version-Release number of selected component (if applicable): python3-tempest-30.1.0-0.20220418200411.44dac69.el9ost.noarch python3-tempestconf-3.2.2-0.20220404161250.6bb29fd.el9ost.noarch openstack-tempest-30.1.0-0.20220418200411.44dac69.el9ost.noarch python3-3.9.10-2.el9.x86_64 python3-libs-3.9.10-2.el9.x86_64 How reproducible: always Actual results: # stestr run test_create_object_with_expect_continue /usr/lib/python3.9/site-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for 10.0.0.139 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.) warnings.warn( {0} tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_expect_continue [0.078599s] ... FAILED Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_services.py", line 174, in test_create_object_with_expect_continue status, _ = self.object_client.create_object_continue( File "/usr/lib/python3.9/site-packages/tempest/lib/services/object_storage/object_client.py", line 137, in create_object_continue conn.endheaders() File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 980, in send self.connect() File "/usr/lib64/python3.9/http/client.py", line 1454, in connect self.sock = self._context.wrap_socket(self.sock, File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/lib64/python3.9/ssl.py", line 1040, in _create self.do_handshake() File "/usr/lib64/python3.9/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '10.0.0.139'. (_ssl.c:1129) ============================== Failed 1 tests - output below: ============================== tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_expect_continue[id-84dafe57-9666-4f6d-84c8-0814d37923b8] ------------------------------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_services.py", line 174, in test_create_object_with_expect_continue status, _ = self.object_client.create_object_continue( File "/usr/lib/python3.9/site-packages/tempest/lib/services/object_storage/object_client.py", line 137, in create_object_continue conn.endheaders() File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/lib64/python3.9/http/client.py", line 980, in send self.connect() File "/usr/lib64/python3.9/http/client.py", line 1454, in connect self.sock = self._context.wrap_socket(self.sock, File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket return self.sslsocket_class._create( File "/usr/lib64/python3.9/ssl.py", line 1040, in _create self.do_handshake() File "/usr/lib64/python3.9/ssl.py", line 1309, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '10.0.0.139'. (_ssl.c:1129) ====== Totals ====== Ran: 1 tests in 0.0786 sec. - Passed: 0 - Skipped: 0 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 1 Sum of execute time for each test: 0.0786 sec. ============== Worker Balance ============== - Worker 0 (1 tests) => 0:00:00.078599 Additional info: default ssl setup with infrared. curl openssl accepts the cert.
if you edit your tempest.conf so that identity.disable_ssl_certificate_validation is set to True (not False as it is right now), the test will pass. Otherwise this is not executed: https://opendev.org/openstack/tempest/src/commit/9f21c90131d9621914963bfaa7743c442b26aa14/tempest/lib/services/object_storage/object_client.py#L173 and as no certificate is set in the tempest.conf, the test will fail with the above traceback. E.g. in upstream, tempest.conf contains a path to a certificate like identity.ca_certificates_file = /opt/stack/data/ca-bundle.pem <- in that case ssl validation may be enabled (disable_ssl_certificate_validation can be set to False).
curl uses /etc/pki/tls/certs/ca-bundle.crt , which should be used by default by all software on this machine. Setting ca_certificates_file = /etc/pki/tls/certs/ca-bundle.crt did not solved the issue. All other test case has no issue with the cert.
I'm trying to figure out the culprit behind that. I've got the same error when using tempest from master with python3.8 and 3.9 which suggests it's not caused by any change in tempest and it's not python version specific. In upstream, I see that centos-9-stream job passed the test - https://zuul.opendev.org/t/openstack/build/94eec7127012416c8f5dd1692fe9c675/logs
I still think this is a ssl or swift issue, e.g. we might have a wrong cert file, I tried e.g. this curl command and that failed too: $ curl --cacert /etc/ssl/cert.pem https://10.0.0.139:13808/v1/AUTH_2b2196d895004ddd92eaa92ed3fae62c/tempest-TestContainer-1379337311/tempest-TestObject-386186008 <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>(overcloud) [stack@undercloud-0 tempest]$
I did some investigation, and this is what I have found: - Wget uses gnutls library. - Using wget https://10.0.0.131 leads to this error: "The certificate's owner does not match hostname ‘10.0.0.131’". This seems to be suspiciously similar to: "IP address mismatch, certificate is not valid for '10.0.0.139'." - Verification of the certificate from https://10.0.0.131 using gnutls-cli fails with: "The certificate is NOT trusted. The name in the certificate does not match the expected." (gnutls-cli 10.0.0.131 -p 443). - Curl uses openssl library. - Curl https://10.0.0.131 runs successfully (meaning the certificate verification was successful). - Verification of the certificate from https://10.0.0.131 using openssl command succeeds. (openssl s_client -host 10.0.0.131 -port 443) I first thought that python uses gnutls however it turned out not to be true as python uses openssl. It does not make really sense to me. Because either openssl is broken or gnutls is.
Hmm, wget also has this. We might try to compare the tls/ssl handshake of both wget/curl. Most SSL library has special environment variable for asking it to save the keys into file, which can be used by wireshark. It might worth to compare those. (Server Name Indication (SNI) usage?) https://everything.curl.dev/usingcurl/tls/sslkeylogfile The clients might also have options for preferred tls/ssl version, that might also change the behavior. We might also switch source and try to bisect the issue if it is really an el8/el9 client side python behavior thing..
It seems this was a change in recent versions of python [1], which explains why it is only seen on rhel9; more specifically a comment in the python tracker goes as follows: "Python 3.6 is a little more forgiving than Python 3.7. Python 3.7 uses OpenSSL's hostname verification algorithms, which interpret the RFCs more strictly. You have to include a SAN field of type IP address. Matching against CN has been deprecated for more than 15 years, see https://bugs.chromium.org/p/chromium/issues/detail?id=308330" I think a valid fix might be to update the tripleo code which generates the certs to include the IP in the SAN field 1. https://bugs.python.org/issue34440
wget/gnutls probably simply refuses to use commonName. This function called in the name compare: https://www.gnutls.org/manual/html_node/X_002e509-certificate-names.html
This seems to be indeed issue with how the certificate provided to the deployment was generated. It is by mistake missing the SAN fields.
Based on all the findings, this is not a bug. The initial environment based on which Attila reported this, was using incorrect SSL certificates. While it was believed provided certificates should have the `subjectAltName`, it was indeed missing them - unexpected setup step was being used on our side in case of OSP 17, which was resulting in certificates without SAN fields. We corrected this mistake and it works as expected now.