Bug 1975201

Summary:	unable to connect to httpd+mod_ssl: SSL routines::sslv3 alert unexpected message
Product:	Red Hat Enterprise Linux 9	Reporter:	Karel Srot <ksrot>
Component:	httpd	Assignee:	Luboš Uhliarik <luhliari>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Branislav Náter <bnater>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	9.0	CC:	amoralej, bnater, hkario, jorton, luhliari, mspacek, sahana, vashirov
Target Milestone:	beta	Keywords:	Regression, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	httpd-2.4.48-4.el9	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-12-07 21:57:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Karel Srot 2021-06-23 09:02:44 UTC

Description of problem:

I am seeing an error where connecting via curl or wget to httpd with mod_ssl configured. The problem is very likely related to openssl v3 since the same test was passing before the openssl rebuild.

# for i in `seq 20`; do curl -k https://localhost/bz1585235hello.txt; done
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
hello
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
hello

As you can see above, majority of requests fail with an error but there were two requests which succeeded and returned page content "hello".

# curl -vvv -k https://localhost
*   Trying ::1:443...
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=localhost
*  start date: Jun 23 08:41:19 2021 GMT
*  expire date: Jun 23 08:41:19 2022 GMT
*  issuer: O=Example CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.76.1
> Accept: */*
> 
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS alert, unexpected_message (522):
* OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0
* Closing connection 0
curl: (56) OpenSSL SSL_read: error:0A0003F2:SSL routines::sslv3 alert unexpected message, errno 0

There is nothing in /var/log/httpd/ssl_error.log, except the warning

[Wed Jun 23 04:41:19.837318 2021] [ssl:warn] [pid 9024:tid 9024] AH01909: myserver:443:0 server certificate does NOT include an ID which matches the server name


The problem does not appear on x86_64, I have noticed it on s390x.

Version-Release number of selected component (if applicable):
openssl-3.0.0-0.alpha16.4.el9.s390x
mod_ssl-2.4.48-3.el9.s390x
httpd-2.4.48-3.el9.s390x
curl-7.76.1-6.el9.s390x

How reproducible:
always

Steps to Reproduce:
1. configure httpd+mod_ssl
2. do a https query

Actual results:
error

Expected results:
page content

Comment 3 Karel Srot 2021-06-23 09:17:36 UTC

adding openssl s_client output:

# openssl s_client -connect localhost:443
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = localhost
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = localhost
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = localhost
verify return:1
---
Certificate chain
 0 s:CN = localhost
   i:O = Example CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 23 09:11:58 2021 GMT; NotAfter: Jun 23 09:11:58 2022 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDCzCCAfOgAwIBAgIBAjANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQKDApFeGFt
cGxlIENBMB4XDTIxMDYyMzA5MTE1OFoXDTIyMDYyMzA5MTE1OFowFDESMBAGA1UE
AwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAweYZ
W2UEcljWLBoqMVf1Dff+kKkwo/dsi43q8EKVvMgbq/gJMHoffLQI79yQy9oRsZUn
pFpZlYWdrXtZAkaq97QRrs8IV7mhAaaTmrfTYzRq3T2bMTrF9KSYnlHF3drtCjoV
Nf6W7L44zTYHiYJ573sqLkN+wnWNiJd+IJT21lKHdcq+SPv+WCRSxkZ9IYfcmH5v
k0ZfRU/PFarHKzs1x7h9jyEKmX1dH3MGQgmakuiBeq2Vk2QN0wesT4niX/jNHHoT
mAUgvtRumHbqKrvwk6rZCALzsdZtELIqRMTgOuhrGBiU8QfG54XWJOrHN7AgSYtt
R1rXQGJl1LQFxLvszQIDAQABo2cwZTAOBgNVHQ8BAf8EBAMCA6gwEwYDVR0lBAww
CgYIKwYBBQUHAwEwHQYDVR0OBBYEFPXedwIA3qiI9YWoubY7wvYu9BQ0MB8GA1Ud
IwQYMBaAFJ0u+PGyU/hqqB1BSdFBrlpVEFK+MA0GCSqGSIb3DQEBCwUAA4IBAQAI
QDsvU0ZtydlFgflXG6XYzxaxDsPTPKd+KDQCeNXHGCRwuaM0VvibzMvenOjRFfa3
w0VkIisuMf+gAQJx2UmeQry8HSdjMtWN3JO1t1unbwmOmWwVFAevD0cR7pl1CJmY
rAflgsx+hDDrlbHaDBT+FSTlwXZhwsKNaqwjBwHttPBlj4SMj5cudE/jxpkMHij8
o+hgfjqZiuavBUawNfVBz+iNnFp1wS01xzszL/dDPM8XB5NFiOC98jjZW7MRR7xH
VZ6zl6fL1bWfPHemUqBJWr5U+62pT0cYXTr/bQ63OzRL4/tJRX9HIFwrfWLboMaK
TZKQSxgDyajbs/r/oyHZ
-----END CERTIFICATE-----
subject=CN = localhost

issuer=O = Example CA

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1339 bytes and written 375 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 998FDAFF36A67BAFC3E63C38AF1132C19AB44EC81F1896590F10FB60BD87E90A
    Session-ID-ctx: 
    Resumption PSK: A615151C5178ABBC5CBC2DCAA13C4389D8254694B547B1D8423AFC2B718493D28F883EBF04233FDA6FE79D231ED041F0
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 1c 4f 26 5a cd c9 53 0f-23 2c 55 a0 f8 0d 0f de   .O&Z..S.#,U.....
    0010 - bf cc 80 10 7e f3 29 22-aa 67 18 c9 8e 3b d0 6f   ....~.)".g...;.o
    0020 - 25 d9 70 96 ed 57 6e 21-cc 48 3a f5 84 da ef 4a   %.p..Wn!.H:....J
    0030 - d4 1d 06 4d a5 50 7b 92-e0 bc 97 2b 82 95 59 5d   ...M.P{....+..Y]
    0040 - 76 8c 3c f8 9a 53 24 f5-b2 e0 2f b9 ed 38 66 f8   v.<..S$.../..8f.
    0050 - 75 1a dc bd a0 23 d3 48-4f 12 20 5a eb 42 3e 94   u....#.HO. Z.B>.
    0060 - 87 a9 ef 17 c8 a5 4a 9e-ce cc 3e 67 71 22 a7 1f   ......J...>gq"..
    0070 - 36 1f 44 b9 b3 a5 7e 1c-b4 c5 c1 47 a0 4e 86 e9   6.D...~....G.N..
    0080 - bc cf f4 3b b2 8e ec 6d-02 01 ac 07 b2 fa 18 c1   ...;...m........
    0090 - a7 31 19 93 a4 26 6b 72-e1 1d 41 db 23 57 f0 87   .1...&kr..A.#W..
    00a0 - dc 18 2c fb 30 d8 e5 34-41 09 1b 3d ec c4 ce d6   ..,.0..4A..=....
    00b0 - ac 5a 0d 8d bd a4 49 d9-dc 82 4a 2e 58 93 d7 7e   .Z....I...J.X..~
    00c0 - f7 2f d5 56 03 c5 31 e3-da 64 12 f9 27 08 92 11   ./.V..1..d..'...
    00d0 - 73 ef 46 5e bb fa 04 3a-3c bd 62 14 a0 8e 14 fe   s.F^...:<.b.....

    Start Time: 1624439727
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 185E1398CEA46B6946348A69CDBCB095ED0BF048281BD03E0E1C7A2771C666B9
    Session-ID-ctx: 
    Resumption PSK: E9C3C4EF4D61E73BF1B3DBDCE5472DAF70DDCCBA2ECDA6ADBF2831D8BFE3FF2A9793C60C91F73B750D740B36A7EDB48D
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 1c 4f 26 5a cd c9 53 0f-23 2c 55 a0 f8 0d 0f de   .O&Z..S.#,U.....
    0010 - 22 3b ee 4b 2c e1 62 50-03 03 51 29 c4 a8 ae d7   ";.K,.bP..Q)....
    0020 - 44 5c d4 97 dd 80 32 38-66 da 32 09 c9 8f 88 f2   D\....28f.2.....
    0030 - b2 96 79 8d d6 b4 2e f6-bf f1 2b ff 49 6c 0d b1   ..y.......+.Il..
    0040 - 60 f1 5b c3 05 fc e8 8d-da d0 fc 55 24 1d 0a d0   `.[........U$...
    0050 - ab d2 e8 42 18 da cd f7-ce 19 a0 d5 75 1c 9c f2   ...B........u...
    0060 - 48 fe 7e 8f 82 ec 26 aa-18 5d 10 82 6d 1f 88 28   H.~...&..]..m..(
    0070 - 6e c4 7c 38 db e9 f2 c1-70 1f 3f 59 66 05 22 88   n.|8....p.?Yf.".
    0080 - c6 12 61 71 91 84 81 d4-64 7a ec bd ce a9 4c 65   ..aq....dz....Le
    0090 - b2 73 46 11 ff 20 cb 24-10 17 41 31 83 24 91 b9   .sF.. .$..A1.$..
    00a0 - e6 5f 16 1a 73 12 fc 71-75 db b3 fd 87 1e b5 d8   ._..s..qu.......
    00b0 - 6e b9 d3 95 2a 1f ed 5a-e7 09 a9 4e a7 fa cf e9   n...*..Z...N....
    00c0 - fb 40 72 46 7f 8b 93 21-df ec fd e3 03 42 4e 1a   .@rF...!.....BN.
    00d0 - 35 11 85 2f 02 66 10 21-d9 e8 77 fa 49 df 79 e1   5../.f.!..w.I.y.

    Start Time: 1624439727
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
<===================== CONNECTION HAS BEEN ESTABLISHED. AT THIS MOMENT I HAVE PRESSED ENTER
000003FF944F34A0:error:0A0003F2:SSL routines:ssl3_read_bytes:sslv3 alert unexpected message:ssl/record/rec_layer_s3.c:1588:SSL alert number 10

Comment 11 Hubert Kario 2021-06-24 17:11:34 UTC

*** Bug 1974715 has been marked as a duplicate of this bug. ***

Comment 12 Karel Srot 2021-06-25 07:04:11 UTC

The issue is not limited to s390x, I am updating the Hardware field accordingly.

Comment 13 Joe Orton 2021-06-25 07:43:16 UTC

Can someone try https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=339791

This has the OpenSSL v3 compatibility fixes from upstream.

Comment 14 Karel Srot 2021-06-25 08:11:33 UTC

(In reply to Joe Orton from comment #13)
> Can someone try
> https://kojihub.stream.rdu2.redhat.com/koji/taskinfo?taskID=339791
> 
> This has the OpenSSL v3 compatibility fixes from upstream.

Hi Joe, tested on x86_64 and it fixes the problem for me. Can't test s390x ATM as we are low on resources.

Comment 16 Karel Srot 2021-06-25 08:33:10 UTC

And it works also for s390x. I guess we should move this bug to httpd then.
Joe, did you see Hubert's comment #10?

Comment 18 Alfredo Moralejo 2021-06-25 09:52:06 UTC

FYI, I've tested that httpd-2.4.48-4.el9.x86_64 fixes the issue reported with RDO deployment in https://bugzilla.redhat.com/show_bug.cgi?id=1974715 which was close as duplicated of this one.

Comment 19 Joe Orton 2021-06-25 11:19:30 UTC

Regarding error-checking, thanks Hubert, I will address upstream.  FWIW this is a copy&paste of the upstream example code, which also doesn't have error checking.

https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_tlsext_ticket_key_cb.html

Comment 20 Joe Orton 2021-06-25 11:23:19 UTC

Note that we have quite a lot remaining test failures for httpd, please have some patience as we work through these. Branislav has waived the -4 build through to unblock testing of other components.

Comment 21 Hubert Kario 2021-06-25 11:28:50 UTC

(In reply to Joe Orton from comment #19)
> Regarding error-checking, thanks Hubert, I will address upstream.  FWIW this
> is a copy&paste of the upstream example code, which also doesn't have error
> checking.
> 
> https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_tlsext_ticket_key_cb.
> html

yes, unfortunately upstream examples commonly omit error checking, I'll try
to fix this particular instance

BTW: the 3.0.0 version supports -DUNUSEDRESULT_DEBUG macro to make compiler
emit warnings when methods that should have they return code checked don't
(it's not defined for all: https://github.com/openssl/openssl/issues/15902
but hopefully will be soon)

Comment 26 Branislav Náter 2021-06-25 12:03:40 UTC

FYI httpd-2.4.48-4.el9 has been added to erratum and will be available in next development and nightly compose.