RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1975201 - unable to connect to httpd+mod_ssl: SSL routines::sslv3 alert unexpected message
Summary: unable to connect to httpd+mod_ssl: SSL routines::sslv3 alert unexpected message
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: httpd
Version: 9.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: beta
: ---
Assignee: Luboš Uhliarik
QA Contact: Branislav Náter
URL:
Whiteboard:
: 1974715 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-23 09:02 UTC by Karel Srot
Modified: 2021-12-07 22:00 UTC (History)
8 users (show)

Fixed In Version: httpd-2.4.48-4.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-07 21:57:54 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Karel Srot 2021-06-23 09:02:44 UTC 
Description of problem:

I am seeing an error where connecting via curl or wget to httpd with mod_ssl configured. The problem is very likely related to openssl v3 since the same test was passing before the openssl rebuild.

# for i in `seq 20`; do curl -k https://localhost/bz1585235hello.txt; done
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
hello
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
hello

As you can see above, majority of requests fail with an error but there were two requests which succeeded and returned page content "hello".

# curl -vvv -k https://localhost
*   Trying ::1:443...
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=localhost
*  start date: Jun 23 08:41:19 2021 GMT
*  expire date: Jun 23 08:41:19 2022 GMT
*  issuer: O=Example CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.76.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS alert, unexpected_message (522):
* OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0

There is nothing in /var/log/httpd/ssl_error.log, except the warning

[Wed Jun 23 04:41:19.837318 2021] [ssl:warn] [pid 9024:tid 9024] AH01909: myserver:443:0 server certificate does NOT include an ID which matches the server name


The problem does not appear on x86_64, I have noticed it on s390x.

Version-Release number of selected component (if applicable):
openssl-3.0.0-0.alpha16.4.el9.s390x
mod_ssl-2.4.48-3.el9.s390x
httpd-2.4.48-3.el9.s390x
curl-7.76.1-6.el9.s390x

How reproducible:
always

Steps to Reproduce:
1. configure httpd+mod_ssl
2. do a https query

Actual results:
error

Expected results:
page content
Comment 3 Karel Srot 2021-06-23 09:17:36 UTC 
adding openssl s_client output:

# openssl s_client -connect localhost:443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = localhost
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = localhost
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = localhost
verify return:1
---
Certificate chain
 0 s:CN = localhost
   i:O = Example CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 23 09:11:58 2021 GMT; NotAfter: Jun 23 09:11:58 2022 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDCzCCAfOgAwIBAgIBAjANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQKDApFeGFt
cGxlIENBMB4XDTIxMDYyMzA5MTE1OFoXDTIyMDYyMzA5MTE1OFowFDESMBAGA1UE
AwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAweYZ
W2UEcljWLBoqMVf1Dff+kKkwo/dsi43q8EKVvMgbq/gJMHoffLQI79yQy9oRsZUn
pFpZlYWdrXtZAkaq97QRrs8IV7mhAaaTmrfTYzRq3T2bMTrF9KSYnlHF3drtCjoV
Nf6W7L44zTYHiYJ573sqLkN+wnWNiJd+IJT21lKHdcq+SPv+WCRSxkZ9IYfcmH5v
k0ZfRU/PFarHKzs1x7h9jyEKmX1dH3MGQgmakuiBeq2Vk2QN0wesT4niX/jNHHoT
mAUgvtRumHbqKrvwk6rZCALzsdZtELIqRMTgOuhrGBiU8QfG54XWJOrHN7AgSYtt
R1rXQGJl1LQFxLvszQIDAQABo2cwZTAOBgNVHQ8BAf8EBAMCA6gwEwYDVR0lBAww
CgYIKwYBBQUHAwEwHQYDVR0OBBYEFPXedwIA3qiI9YWoubY7wvYu9BQ0MB8GA1Ud
IwQYMBaAFJ0u+PGyU/hqqB1BSdFBrlpVEFK+MA0GCSqGSIb3DQEBCwUAA4IBAQAI
QDsvU0ZtydlFgflXG6XYzxaxDsPTPKd+KDQCeNXHGCRwuaM0VvibzMvenOjRFfa3
w0VkIisuMf+gAQJx2UmeQry8HSdjMtWN3JO1t1unbwmOmWwVFAevD0cR7pl1CJmY
rAflgsx+hDDrlbHaDBT+FSTlwXZhwsKNaqwjBwHttPBlj4SMj5cudE/jxpkMHij8
o+hgfjqZiuavBUawNfVBz+iNnFp1wS01xzszL/dDPM8XB5NFiOC98jjZW7MRR7xH
VZ6zl6fL1bWfPHemUqBJWr5U+62pT0cYXTr/bQ63OzRL4/tJRX9HIFwrfWLboMaK
TZKQSxgDyajbs/r/oyHZ
-----END CERTIFICATE-----
subject=CN = localhost

issuer=O = Example CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1339 bytes and written 375 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 998FDAFF36A67BAFC3E63C38AF1132C19AB44EC81F1896590F10FB60BD87E90A
    Session-ID-ctx: 
    Resumption PSK: A615151C5178ABBC5CBC2DCAA13C4389D8254694B547B1D8423AFC2B718493D28F883EBF04233FDA6FE79D231ED041F0
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 1c 4f 26 5a cd c9 53 0f-23 2c 55 a0 f8 0d 0f de   .O&Z..S.#,U.....
    0010 - bf cc 80 10 7e f3 29 22-aa 67 18 c9 8e 3b d0 6f   ....~.)".g...;.o
    0020 - 25 d9 70 96 ed 57 6e 21-cc 48 3a f5 84 da ef 4a   %.p..Wn!.H:....J
    0030 - d4 1d 06 4d a5 50 7b 92-e0 bc 97 2b 82 95 59 5d   ...M.P{....+..Y]
    0040 - 76 8c 3c f8 9a 53 24 f5-b2 e0 2f b9 ed 38 66 f8   v.<..S$.../..8f.
    0050 - 75 1a dc bd a0 23 d3 48-4f 12 20 5a eb 42 3e 94   u....#.HO. Z.B>.
    0060 - 87 a9 ef 17 c8 a5 4a 9e-ce cc 3e 67 71 22 a7 1f   ......J...>gq"..
    0070 - 36 1f 44 b9 b3 a5 7e 1c-b4 c5 c1 47 a0 4e 86 e9   6.D...~....G.N..
    0080 - bc cf f4 3b b2 8e ec 6d-02 01 ac 07 b2 fa 18 c1   ...;...m........
    0090 - a7 31 19 93 a4 26 6b 72-e1 1d 41 db 23 57 f0 87   .1...&kr..A.#W..
    00a0 - dc 18 2c fb 30 d8 e5 34-41 09 1b 3d ec c4 ce d6   ..,.0..4A..=....
    00b0 - ac 5a 0d 8d bd a4 49 d9-dc 82 4a 2e 58 93 d7 7e   .Z....I...J.X..~
    00c0 - f7 2f d5 56 03 c5 31 e3-da 64 12 f9 27 08 92 11   ./.V..1..d..'...
    00d0 - 73 ef 46 5e bb fa 04 3a-3c bd 62 14 a0 8e 14 fe   s.F^...:<.b.....

    Start Time: 1624439727
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 185E1398CEA46B6946348A69CDBCB095ED0BF048281BD03E0E1C7A2771C666B9
    Session-ID-ctx: 
    Resumption PSK: E9C3C4EF4D61E73BF1B3DBDCE5472DAF70DDCCBA2ECDA6ADBF2831D8BFE3FF2A9793C60C91F73B750D740B36A7EDB48D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 1c 4f 26 5a cd c9 53 0f-23 2c 55 a0 f8 0d 0f de   .O&Z..S.#,U.....
    0010 - 22 3b ee 4b 2c e1 62 50-03 03 51 29 c4 a8 ae d7   ";.K,.bP..Q)....
    0020 - 44 5c d4 97 dd 80 32 38-66 da 32 09 c9 8f 88 f2   D\....28f.2.....
    0030 - b2 96 79 8d d6 b4 2e f6-bf f1 2b ff 49 6c 0d b1   ..y.......+.Il..
    0040 - 60 f1 5b c3 05 fc e8 8d-da d0 fc 55 24 1d 0a d0   `.[........U$...
    0050 - ab d2 e8 42 18 da cd f7-ce 19 a0 d5 75 1c 9c f2   ...B........u...
    0060 - 48 fe 7e 8f 82 ec 26 aa-18 5d 10 82 6d 1f 88 28   H.~...&..]..m..(
    0070 - 6e c4 7c 38 db e9 f2 c1-70 1f 3f 59 66 05 22 88   n.|8....p.?Yf.".
    0080 - c6 12 61 71 91 84 81 d4-64 7a ec bd ce a9 4c 65   ..aq....dz....Le
    0090 - b2 73 46 11 ff 20 cb 24-10 17 41 31 83 24 91 b9   .sF.. .$..A1.$..
    00a0 - e6 5f 16 1a 73 12 fc 71-75 db b3 fd 87 1e b5 d8   ._..s..qu.......
    00b0 - 6e b9 d3 95 2a 1f ed 5a-e7 09 a9 4e a7 fa cf e9   n...*..Z...N....
    00c0 - fb 40 72 46 7f 8b 93 21-df ec fd e3 03 42 4e 1a   .@rF...!.....BN.
    00d0 - 35 11 85 2f 02 66 10 21-d9 e8 77 fa 49 df 79 e1   5../.f.!..w.I.y.

    Start Time: 1624439727
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
<===================== CONNECTION HAS BEEN ESTABLISHED. AT THIS MOMENT I HAVE PRESSED ENTER
000003FF944F34A0:error:0A0003F2:SSL routines:ssl3_read_bytes:sslv3 alert unexpected message:ssl/record/rec_layer_s3.c:1588:SSL alert number 10
Comment 11 Hubert Kario 2021-06-24 17:11:34 UTC 
*** Bug 1974715 has been marked as a duplicate of this bug. ***
Comment 12 Karel Srot 2021-06-25 07:04:11 UTC 
The issue is not limited to s390x, I am updating the Hardware field accordingly.
Comment 13 Joe Orton 2021-06-25 07:43:16 UTC 
Can someone try https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=339791

This has the OpenSSL v3 compatibility fixes from upstream.
Comment 14 Karel Srot 2021-06-25 08:11:33 UTC 
(In reply to Joe Orton from comment #13)
> Can someone try
> https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=339791
> 
> This has the OpenSSL v3 compatibility fixes from upstream.

Hi Joe, tested on x86_64 and it fixes the problem for me. Can't test s390x ATM as we are low on resources.
Comment 16 Karel Srot 2021-06-25 08:33:10 UTC 
And it works also for s390x. I guess we should move this bug to httpd then.
Joe, did you see Hubert's comment #10?
Comment 18 Alfredo Moralejo 2021-06-25 09:52:06 UTC 
FYI, I've tested that httpd-2.4.48-4.el9.x86_64 fixes the issue reported with RDO deployment in https://bugzilla.redhat.com/show_bug.cgi?id=1974715 which was close as duplicated of this one.
Comment 19 Joe Orton 2021-06-25 11:19:30 UTC 
Regarding error-checking, thanks Hubert, I will address upstream.  FWIW this is a copy&paste of the upstream example code, which also doesn't have error checking.

https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_tlsext_ticket_key_cb.html
Comment 20 Joe Orton 2021-06-25 11:23:19 UTC 
Note that we have quite a lot remaining test failures for httpd, please have some patience as we work through these. Branislav has waived the -4 build through to unblock testing of other components.
Comment 21 Hubert Kario 2021-06-25 11:28:50 UTC 
(In reply to Joe Orton from comment #19)
> Regarding error-checking, thanks Hubert, I will address upstream.  FWIW this
> is a copy&paste of the upstream example code, which also doesn't have error
> checking.
> 
> https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_tlsext_ticket_key_cb.
> html

yes, unfortunately upstream examples commonly omit error checking, I'll try
to fix this particular instance

BTW: the 3.0.0 version supports -DUNUSEDRESULT_DEBUG macro to make compiler
emit warnings when methods that should have they return code checked don't
(it's not defined for all: https://github.com/openssl/openssl/issues/15902
but hopefully will be soon)
Comment 26 Branislav Náter 2021-06-25 12:03:40 UTC 
FYI httpd-2.4.48-4.el9 has been added to erratum and will be available in next development and nightly compose.
Note You need to log in before you can comment on or make changes to this bug.