Bug 2078898
| Summary: | test_create_object_with_expect_continue using different cert validator | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Attila Fazekas <afazekas> |
| Component: | openstack-tempest | Assignee: | Chandan Kumar <chkumar> |
| Status: | CLOSED NOTABUG | QA Contact: | Martin Kopec <mkopec> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 17.0 (Wallaby) | CC: | alee, apevec, fpantano, gfidente, lhh, lpiwowar, psedlak, slinaber, udesale |
| Target Milestone: | --- | Keywords: | Reopened, Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-09 09:45:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Attila Fazekas
2022-04-26 12:58:01 UTC
if you edit your tempest.conf so that identity.disable_ssl_certificate_validation is set to True (not False as it is right now), the test will pass. Otherwise this is not executed: https://opendev.org/openstack/tempest/src/commit/9f21c90131d9621914963bfaa7743c442b26aa14/tempest/lib/services/object_storage/object_client.py#L173 and as no certificate is set in the tempest.conf, the test will fail with the above traceback. E.g. in upstream, tempest.conf contains a path to a certificate like identity.ca_certificates_file = /opt/stack/data/ca-bundle.pem <- in that case ssl validation may be enabled (disable_ssl_certificate_validation can be set to False). curl uses /etc/pki/tls/certs/ca-bundle.crt , which should be used by default by all software on this machine. Setting ca_certificates_file = /etc/pki/tls/certs/ca-bundle.crt did not solved the issue. All other test case has no issue with the cert. I'm trying to figure out the culprit behind that. I've got the same error when using tempest from master with python3.8 and 3.9 which suggests it's not caused by any change in tempest and it's not python version specific. In upstream, I see that centos-9-stream job passed the test - https://zuul.opendev.org/t/openstack/build/94eec7127012416c8f5dd1692fe9c675/logs I still think this is a ssl or swift issue, e.g. we might have a wrong cert file, I tried e.g. this curl command and that failed too: $ curl --cacert /etc/ssl/cert.pem https://10.0.0.139:13808/v1/AUTH_2b2196d895004ddd92eaa92ed3fae62c/tempest-TestContainer-1379337311/tempest-TestObject-386186008 <html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>(overcloud) [stack@undercloud-0 tempest]$ I did some investigation, and this is what I have found: - Wget uses gnutls library. - Using wget https://10.0.0.131 leads to this error: "The certificate's owner does not match hostname ‘10.0.0.131’". This seems to be suspiciously similar to: "IP address mismatch, certificate is not valid for '10.0.0.139'." - Verification of the certificate from https://10.0.0.131 using gnutls-cli fails with: "The certificate is NOT trusted. The name in the certificate does not match the expected." (gnutls-cli 10.0.0.131 -p 443). - Curl uses openssl library. - Curl https://10.0.0.131 runs successfully (meaning the certificate verification was successful). - Verification of the certificate from https://10.0.0.131 using openssl command succeeds. (openssl s_client -host 10.0.0.131 -port 443) I first thought that python uses gnutls however it turned out not to be true as python uses openssl. It does not make really sense to me. Because either openssl is broken or gnutls is. Hmm, wget also has this. We might try to compare the tls/ssl handshake of both wget/curl. Most SSL library has special environment variable for asking it to save the keys into file, which can be used by wireshark. It might worth to compare those. (Server Name Indication (SNI) usage?) https://everything.curl.dev/usingcurl/tls/sslkeylogfile The clients might also have options for preferred tls/ssl version, that might also change the behavior. We might also switch source and try to bisect the issue if it is really an el8/el9 client side python behavior thing.. It seems this was a change in recent versions of python [1], which explains why it is only seen on rhel9; more specifically a comment in the python tracker goes as follows: "Python 3.6 is a little more forgiving than Python 3.7. Python 3.7 uses OpenSSL's hostname verification algorithms, which interpret the RFCs more strictly. You have to include a SAN field of type IP address. Matching against CN has been deprecated for more than 15 years, see https://bugs.chromium.org/p/chromium/issues/detail?id=308330" I think a valid fix might be to update the tripleo code which generates the certs to include the IP in the SAN field 1. https://bugs.python.org/issue34440 wget/gnutls probably simply refuses to use commonName. This function called in the name compare: https://www.gnutls.org/manual/html_node/X_002e509-certificate-names.html This seems to be indeed issue with how the certificate provided to the deployment was generated. It is by mistake missing the SAN fields. Based on all the findings, this is not a bug. The initial environment based on which Attila reported this, was using incorrect SSL certificates. While it was believed provided certificates should have the `subjectAltName`, it was indeed missing them - unexpected setup step was being used on our side in case of OSP 17, which was resulting in certificates without SAN fields. We corrected this mistake and it works as expected now. |