Bug 1775158

Summary: POST request with TLS 1.3 PHA client auth fails: Re-negotiation handshake failed: Client certificate missing
Product: Red Hat Enterprise Linux 8 Reporter: Christian Heimes <cheimes>
Component: httpdAssignee: Luboš Uhliarik <luhliari>
Status: CLOSED ERRATA QA Contact: Maryna Nalbandian <mnalband>
Severity: high Docs Contact:
Priority: unspecified    
Version: 8.2CC: anon.amish, bnater, extras-qa, jkaluza, jorton, luhliari, pahan, pasik
Target Milestone: rc   
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1775146 Environment:
Last Closed: 2020-04-28 15:32:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1775146    
Bug Blocks: 1760850, 1777809    

Description Christian Heimes 2019-11-21 13:55:25 UTC 
+++ This bug was initially created as a clone of Bug #1775146 +++

Description of problem:
FreeIPA uses client cert authentication to communicate with Dogtag over a Apache HTTTpd proxy. Only some routes require TLS client cert authentication. These locations have SSLVerifyClient require.

Client cert auth works fine with TLS 1.2 and for GET requests with TLS 1.3 . However TLS 1.3 client cert authentication breaks for POST requests. All POST requests over TLS 1.3 are failing with "Re-negotiation handshake failed: Client certificate missing". The client has PHA enabled.


Version-Release number of selected component (if applicable):
openssl-1.1.1d-2.fc31.x86_64
httpd-2.4.41-5.fc31.x86_64
mod_ssl-2.4.41-5.fc31.x86_64


How reproducible:
always

Steps to Reproduce:
See attached reproducer script for an IPA installation
IPA has TLS 1.3 disabled by default in /etc/httpd/conf.d/ssl.conf. You need to modify SSLProtocol and restart httpd.service.

Actual results:
POST request is failing with 403 Forbidden
Apache error log contains:
[Thu Nov 21 07:50:21.288373 2019] [ssl:error] [pid 26416:tid 140101799589632] [client 10.0.138.108:36796] AH02263: Re-negotiation handshake failed: Client certificate missing

Expected results:
The POST request works with 200 OK

Additional info:

I have analyzed a GET and POST request with wireshark. For GET requests the post handshake authentication workflow works as expected. The client sends a GET request, server response with TLS message "Certificate Request", client sends the client cert in TLS messages "Certificate, Certificate Verify, Finished", server responds with HTTP 200 OK. For a POST request, the server sends "HTTP/1.1 403 Forbidden" before it reads the TLS messages "Certificate, Certificate Verify, Finished" from the connection.

# tshark -o "tls.desegment_ssl_records: TRUE" -o "tls.keylog_file:/tmp/keylog"  -i any port 443
Running as user "root" and group "root". This could be dangerous.
Capturing on 'any'
 
    1 0.000000000 10.0.138.108 → 10.0.138.108 TCP 76 36794 → 443 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605228 TSecr=0 WS=128
    2 0.000030627 10.0.138.108 → 10.0.138.108 TCP 76 443 → 36794 [SYN, ACK] Seq=0 Ack=1 Win=65483 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605228 TSecr=1228605228 WS=128
    3 0.000040559 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=1228605228 TSecr=1228605228
    4 0.000704884 10.0.138.108 → 10.0.138.108 TLSv1 585 Client Hello
    5 0.000721951 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=1 Ack=518 Win=65024 Len=0 TSval=1228605229 TSecr=1228605229
    6 0.002084036 10.0.138.108 → 10.0.138.108 TLSv1.3 3171 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished
    7 0.002100614 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=518 Ack=3104 Win=63360 Len=0 TSval=1228605230 TSecr=1228605230
    8 0.002553815 10.0.138.108 → 10.0.138.108 TLSv1.3 148 Change Cipher Spec, Finished
    9 0.002567863 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=3104 Ack=598 Win=65536 Len=0 TSval=1228605231 TSecr=1228605231
   10 0.002756021 10.0.138.108 → 10.0.138.108 HTTP 277 GET /ca/agent/ca/displayBySerial HTTP/1.1
   11 0.002759835 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=3104 Ack=807 Win=65408 Len=0 TSval=1228605231 TSecr=1228605231
   12 0.002776709 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket
   13 0.002784604 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=807 Ack=3407 Win=65280 Len=0 TSval=1228605231 TSecr=1228605231
   14 0.002870580 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket
   15 0.002875193 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=807 Ack=3710 Win=65024 Len=0 TSval=1228605231 TSecr=1228605231
   16 0.003138218 10.0.138.108 → 10.0.138.108 TLSv1.3 235 Certificate Request
   17 0.003151800 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=807 Ack=3877 Win=64896 Len=0 TSval=1228605231 TSecr=1228605231
   18 0.004778969 10.0.138.108 → 10.0.138.108 TLSv1.3 2664 Certificate, Certificate Verify, Finished
   19 0.004791816 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=3877 Ack=3403 Win=63616 Len=0 TSval=1228605233 TSecr=1228605233
   20 0.005226573 10.0.138.108 → 10.0.138.108 TLSv1.3 1379 New Session Ticket
   21 0.005239616 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=3403 Ack=5188 Win=64256 Len=0 TSval=1228605233 TSecr=1228605233
   22 0.005363821 10.0.138.108 → 10.0.138.108 TLSv1.3 1379 New Session Ticket
   23 0.005367288 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=3403 Ack=6499 Win=64256 Len=0 TSval=1228605233 TSecr=1228605233
   24 0.020897783 10.0.138.108 → 10.0.138.108 HTTP 1922 HTTP/1.1 200 200  (text/html)
   25 0.020904818 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=3403 Ack=8353 Win=64000 Len=0 TSval=1228605249 TSecr=1228605249
   26 0.021939091 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [FIN, ACK] Seq=3403 Ack=8353 Win=65536 Len=0 TSval=1228605250 TSecr=1228605249
   27 0.022087729 10.0.138.108 → 10.0.138.108 TLSv1.3 92 Alert (Level: Warning, Description: Close Notify)
   28 0.022112779 10.0.138.108 → 10.0.138.108 TCP 56 36794 → 443 [RST] Seq=3404 Win=0 Len=0


   29 0.030330047 10.0.138.108 → 10.0.138.108 TCP 76 36796 → 443 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605258 TSecr=0 WS=128
   30 0.030348996 10.0.138.108 → 10.0.138.108 TCP 76 443 → 36796 [SYN, ACK] Seq=0 Ack=1 Win=65483 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605258 TSecr=1228605258 WS=128
   31 0.030357883 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=1228605258 TSecr=1228605258
   32 0.030889786 10.0.138.108 → 10.0.138.108 TLSv1 585 Client Hello
   33 0.030906029 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=1 Ack=518 Win=65024 Len=0 TSval=1228605259 TSecr=1228605259
   34 0.032502581 10.0.138.108 → 10.0.138.108 TLSv1.3 3171 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished
   35 0.032521963 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=518 Ack=3104 Win=63360 Len=0 TSval=1228605261 TSecr=1228605261
   36 0.032968162 10.0.138.108 → 10.0.138.108 TLSv1.3 148 Change Cipher Spec, Finished
   37 0.032981319 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=3104 Ack=598 Win=65536 Len=0 TSval=1228605261 TSecr=1228605261
   38 0.033130757 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket
   39 0.033135265 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=598 Ack=3407 Win=65280 Len=0 TSval=1228605261 TSecr=1228605261
   40 0.033148744 10.0.138.108 → 10.0.138.108 TLSv1.3 298 [TLS segment of a reassembled PDU]
   41 0.033151866 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=3407 Ack=828 Win=65408 Len=0 TSval=1228605261 TSecr=1228605261
   42 0.033165106 10.0.138.108 → 10.0.138.108 HTTP 114 POST /ca/agent/ca/displayBySerial HTTP/1.1
   43 0.033167675 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=3407 Ack=874 Win=65408 Len=0 TSval=1228605261 TSecr=1228605261
   44 0.033180766 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket
   45 0.033183758 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=874 Ack=3710 Win=65024 Len=0 TSval=1228605261 TSecr=1228605261
   46 0.033299383 10.0.138.108 → 10.0.138.108 TLSv1.3 235 Certificate Request
   47 0.033316804 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=874 Ack=3877 Win=64896 Len=0 TSval=1228605261 TSecr=1228605261
   48 0.033399513 10.0.138.108 → 10.0.138.108 HTTP 570 HTTP/1.1 403 Forbidden  (text/html)
   49 0.033402002 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=874 Ack=4379 Win=64768 Len=0 TSval=1228605261 TSecr=1228605261
   50 0.034499630 10.0.138.108 → 10.0.138.108 TLSv1.3 2664 Certificate, Certificate Verify, Finished
   51 0.034504331 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=4379 Ack=3470 Win=63616 Len=0 TSval=1228605263 TSecr=1228605263
   52 0.035032328 10.0.138.108 → 10.0.138.108 TLSv1.3 1379 New Session Ticket
   53 0.035038891 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=3470 Ack=5690 Win=64256 Len=0 TSval=1228605263 TSecr=1228605263
   54 0.035091948 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [RST, ACK] Seq=3470 Ack=5690 Win=65536 Len=0 TSval=1228605263 TSecr=1228605263
Comment 1 Christian Heimes 2019-11-21 13:57:23 UTC 
I can reproduce the issue on RHEL 8.2 with idm:DL1 module:

Version-Release number of selected component (if applicable):
openssl-1.1.1c-6.el8.x86_64
mod_ssl-2.4.37-18.module+el8.2.0+4739+89aa1872.x86_64
httpd-2.4.37-18.module+el8.2.0+4739+89aa1872.x86_64

Actual results:
[Thu Nov 21 08:54:02.891746 2019] [ssl:error] [pid 5242:tid 140395275077376] [client 10.0.137.160:36094] AH: verify client post handshake
[Thu Nov 21 08:54:02.936393 2019] [ssl:error] [pid 5242:tid 140395241506560] [client 10.0.137.160:36096] AH: verify client post handshake
[Thu Nov 21 08:54:02.936472 2019] [ssl:error] [pid 5242:tid 140395241506560] [client 10.0.137.160:36096] AH02263: Re-negotiation handshake failed: Client certificate missing
Comment 2 Christian Heimes 2019-11-25 10:38:11 UTC 
Joe has pushed patched builds to F30 and F31. The patches fix the issue that I'm seeing with TLS 1.3 PHA and HTTP POST requests. FreeIPA's test suite is passing with patched HTTPd and TLS 1.3 enabled.
Comment 12 errata-xmlrpc 2020-04-28 15:32:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1585