Bug 2078898 - test_create_object_with_expect_continue using different cert validator
Summary: test_create_object_with_expect_continue using different cert validator
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tempest
Version: 17.0 (Wallaby)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Chandan Kumar
QA Contact: Martin Kopec
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-26 12:58 UTC by Attila Fazekas
Modified: 2022-06-09 09:45 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-06-09 09:45:29 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OSP-14877 0 None None None 2022-04-26 13:14:45 UTC

Description Attila Fazekas 2022-04-26 12:58:01 UTC
Description of problem:
tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_expect_continue[id-84dafe57-9666-4f6d-84c8-0814d37923b8] fails only on cert, the other object tests accepted the cert. 
Probably it uses different cert verification method.
The issue only seen on el9, not on el8.
Both rgw and swift.

The cert subject is an IP in the CN.

Version-Release number of selected component (if applicable):
python3-tempest-30.1.0-0.20220418200411.44dac69.el9ost.noarch
python3-tempestconf-3.2.2-0.20220404161250.6bb29fd.el9ost.noarch
openstack-tempest-30.1.0-0.20220418200411.44dac69.el9ost.noarch
python3-3.9.10-2.el9.x86_64
python3-libs-3.9.10-2.el9.x86_64


How reproducible:
always

Actual results:
# stestr run test_create_object_with_expect_continue
/usr/lib/python3.9/site-packages/urllib3/connection.py:455: SubjectAltNameWarning: Certificate for 10.0.0.139 has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
{0} tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_expect_continue [0.078599s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_services.py", line 174, in test_create_object_with_expect_continue
        status, _ = self.object_client.create_object_continue(
      File "/usr/lib/python3.9/site-packages/tempest/lib/services/object_storage/object_client.py", line 137, in create_object_continue
        conn.endheaders()
      File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
        self.send(msg)
      File "/usr/lib64/python3.9/http/client.py", line 980, in send
        self.connect()
      File "/usr/lib64/python3.9/http/client.py", line 1454, in connect
        self.sock = self._context.wrap_socket(self.sock,
      File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket
        return self.sslsocket_class._create(
      File "/usr/lib64/python3.9/ssl.py", line 1040, in _create
        self.do_handshake()
      File "/usr/lib64/python3.9/ssl.py", line 1309, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '10.0.0.139'. (_ssl.c:1129)
    

==============================
Failed 1 tests - output below:
==============================

tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_expect_continue[id-84dafe57-9666-4f6d-84c8-0814d37923b8]
-------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/tempest/api/object_storage/test_object_services.py", line 174, in test_create_object_with_expect_continue
        status, _ = self.object_client.create_object_continue(
      File "/usr/lib/python3.9/site-packages/tempest/lib/services/object_storage/object_client.py", line 137, in create_object_continue
        conn.endheaders()
      File "/usr/lib64/python3.9/http/client.py", line 1280, in endheaders
        self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/lib64/python3.9/http/client.py", line 1040, in _send_output
        self.send(msg)
      File "/usr/lib64/python3.9/http/client.py", line 980, in send
        self.connect()
      File "/usr/lib64/python3.9/http/client.py", line 1454, in connect
        self.sock = self._context.wrap_socket(self.sock,
      File "/usr/lib64/python3.9/ssl.py", line 500, in wrap_socket
        return self.sslsocket_class._create(
      File "/usr/lib64/python3.9/ssl.py", line 1040, in _create
        self.do_handshake()
      File "/usr/lib64/python3.9/ssl.py", line 1309, in do_handshake
        self._sslobj.do_handshake()
    ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: IP address mismatch, certificate is not valid for '10.0.0.139'. (_ssl.c:1129)
    


======
Totals
======
Ran: 1 tests in 0.0786 sec.
 - Passed: 0
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 1
Sum of execute time for each test: 0.0786 sec.

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:00.078599


Additional info:
default ssl setup with infrared.
curl openssl accepts the cert.

Comment 1 Martin Kopec 2022-04-26 22:26:54 UTC
if you edit your tempest.conf so that identity.disable_ssl_certificate_validation is set to True (not False as it is right now), the test will pass.
Otherwise this is not executed:
https://opendev.org/openstack/tempest/src/commit/9f21c90131d9621914963bfaa7743c442b26aa14/tempest/lib/services/object_storage/object_client.py#L173
and as no certificate is set in the tempest.conf, the test will fail with the above traceback.

E.g. in upstream, tempest.conf contains a path to a certificate like identity.ca_certificates_file = /opt/stack/data/ca-bundle.pem <- in that case ssl validation may be enabled (disable_ssl_certificate_validation can be set to False).

Comment 3 Attila Fazekas 2022-04-27 15:30:50 UTC
curl uses /etc/pki/tls/certs/ca-bundle.crt , which should be used by default by all software on this machine.

Setting ca_certificates_file = /etc/pki/tls/certs/ca-bundle.crt  did not solved the issue.
All other test case has no issue with the cert.

Comment 4 Martin Kopec 2022-04-28 12:51:26 UTC
I'm trying to figure out the culprit behind that. I've got the same error when using tempest from master with python3.8 and 3.9 which suggests it's not caused by any change in tempest and it's not python version specific.

In upstream, I see that centos-9-stream job passed the test - https://zuul.opendev.org/t/openstack/build/94eec7127012416c8f5dd1692fe9c675/logs

Comment 5 Martin Kopec 2022-05-04 07:36:02 UTC
I still think this is a ssl or swift issue, e.g. we might have a wrong cert file, I tried e.g. this curl command and that failed too:

$ curl --cacert /etc/ssl/cert.pem https://10.0.0.139:13808/v1/AUTH_2b2196d895004ddd92eaa92ed3fae62c/tempest-TestContainer-1379337311/tempest-TestObject-386186008
<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>(overcloud) [stack@undercloud-0 tempest]$

Comment 6 Lukas Piwowarski 2022-05-26 16:18:50 UTC
I did some investigation, and this is what I have found: 

- Wget uses gnutls library.
- Using wget https://10.0.0.131 leads to this error:  "The certificate's owner does not match hostname ‘10.0.0.131’". This seems to be suspiciously similar to: "IP address mismatch, certificate is not valid for '10.0.0.139'."
- Verification of the certificate from https://10.0.0.131 using gnutls-cli fails with: "The certificate is NOT trusted. The name in the certificate does not match the expected." (gnutls-cli 10.0.0.131 -p 443).


- Curl uses openssl library.
- Curl https://10.0.0.131 runs successfully (meaning the certificate verification was successful).
- Verification of the certificate from https://10.0.0.131 using openssl command succeeds. (openssl s_client -host 10.0.0.131 -port 443)

I first thought that python uses gnutls however it turned out not to be true as python uses openssl. It does not make really sense to me. Because either openssl is broken or gnutls is.

Comment 10 Attila Fazekas 2022-05-31 13:48:55 UTC
Hmm, wget also has this.

We might try to compare the tls/ssl handshake of both wget/curl.
Most SSL library has special environment variable for asking it to save the keys into file,
which can be used by wireshark.
It might worth to compare those. (Server Name Indication (SNI) usage?)
https://everything.curl.dev/usingcurl/tls/sslkeylogfile


The clients might also have options for preferred tls/ssl version, that might also change the behavior.

We might also switch source and try to bisect the issue if it is really an el8/el9 client side python behavior thing..

Comment 13 Giulio Fidente 2022-06-01 13:29:52 UTC
It seems this was a change in recent versions of python [1], which explains why it is only seen on rhel9; more specifically a comment in the python tracker goes as follows:

"Python 3.6 is a little more forgiving than Python 3.7. Python 3.7 uses OpenSSL's hostname verification algorithms, which interpret the RFCs more strictly. You have to include a SAN field of type IP address. Matching against CN has been deprecated for more than 15 years, see https://bugs.chromium.org/p/chromium/issues/detail?id=308330"

I think a valid fix might be to update the tripleo code which generates the certs to include the IP in the SAN field

1. https://bugs.python.org/issue34440

Comment 15 Attila Fazekas 2022-06-01 15:27:26 UTC
wget/gnutls probably simply refuses to use commonName.
This function called in the name compare:
https://www.gnutls.org/manual/html_node/X_002e509-certificate-names.html

Comment 17 Pavel Sedlák 2022-06-02 10:15:05 UTC
This seems to be indeed issue with how the certificate provided to the deployment was generated.
It is by mistake missing the SAN fields.

Comment 19 Pavel Sedlák 2022-06-09 09:45:29 UTC
Based on all the findings, this is not a bug.

The initial environment based on which Attila reported this, was using incorrect SSL certificates.

While it was believed provided certificates should have the `subjectAltName`,
it was indeed missing them - unexpected setup step was being used on our side in case of OSP 17,
which was resulting in certificates without SAN fields. We corrected this mistake and it works as expected now.


Note You need to log in before you can comment on or make changes to this bug.