Created attachment 1638447 [details] reproducer script Description of problem: FreeIPA uses client cert authentication to communicate with Dogtag over a Apache HTTTpd proxy. Only some routes require TLS client cert authentication. These locations have SSLVerifyClient require. Client cert auth works fine with TLS 1.2 and for GET requests with TLS 1.3 . However TLS 1.3 client cert authentication breaks for POST requests. All POST requests over TLS 1.3 are failing with "Re-negotiation handshake failed: Client certificate missing". The client has PHA enabled. Version-Release number of selected component (if applicable): openssl-1.1.1d-2.fc31.x86_64 httpd-2.4.41-5.fc31.x86_64 mod_ssl-2.4.41-5.fc31.x86_64 How reproducible: always Steps to Reproduce: See attached reproducer script for an IPA installation IPA has TLS 1.3 disabled by default in /etc/httpd/conf.d/ssl.conf. You need to modify SSLProtocol and restart httpd.service. Actual results: POST request is failing with 403 Forbidden Apache error log contains: [Thu Nov 21 07:50:21.288373 2019] [ssl:error] [pid 26416:tid 140101799589632] [client 10.0.138.108:36796] AH02263: Re-negotiation handshake failed: Client certificate missing Expected results: The POST request works with 200 OK Additional info: I have analyzed a GET and POST request with wireshark. For GET requests the post handshake authentication workflow works as expected. The client sends a GET request, server response with TLS message "Certificate Request", client sends the client cert in TLS messages "Certificate, Certificate Verify, Finished", server responds with HTTP 200 OK. For a POST request, the server sends "HTTP/1.1 403 Forbidden" before it reads the TLS messages "Certificate, Certificate Verify, Finished" from the connection. # tshark -o "tls.desegment_ssl_records: TRUE" -o "tls.keylog_file:/tmp/keylog" -i any port 443 Running as user "root" and group "root". This could be dangerous. Capturing on 'any' 1 0.000000000 10.0.138.108 → 10.0.138.108 TCP 76 36794 → 443 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605228 TSecr=0 WS=128 2 0.000030627 10.0.138.108 → 10.0.138.108 TCP 76 443 → 36794 [SYN, ACK] Seq=0 Ack=1 Win=65483 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605228 TSecr=1228605228 WS=128 3 0.000040559 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=1228605228 TSecr=1228605228 4 0.000704884 10.0.138.108 → 10.0.138.108 TLSv1 585 Client Hello 5 0.000721951 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=1 Ack=518 Win=65024 Len=0 TSval=1228605229 TSecr=1228605229 6 0.002084036 10.0.138.108 → 10.0.138.108 TLSv1.3 3171 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished 7 0.002100614 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=518 Ack=3104 Win=63360 Len=0 TSval=1228605230 TSecr=1228605230 8 0.002553815 10.0.138.108 → 10.0.138.108 TLSv1.3 148 Change Cipher Spec, Finished 9 0.002567863 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=3104 Ack=598 Win=65536 Len=0 TSval=1228605231 TSecr=1228605231 10 0.002756021 10.0.138.108 → 10.0.138.108 HTTP 277 GET /ca/agent/ca/displayBySerial HTTP/1.1 11 0.002759835 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=3104 Ack=807 Win=65408 Len=0 TSval=1228605231 TSecr=1228605231 12 0.002776709 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket 13 0.002784604 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=807 Ack=3407 Win=65280 Len=0 TSval=1228605231 TSecr=1228605231 14 0.002870580 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket 15 0.002875193 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=807 Ack=3710 Win=65024 Len=0 TSval=1228605231 TSecr=1228605231 16 0.003138218 10.0.138.108 → 10.0.138.108 TLSv1.3 235 Certificate Request 17 0.003151800 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=807 Ack=3877 Win=64896 Len=0 TSval=1228605231 TSecr=1228605231 18 0.004778969 10.0.138.108 → 10.0.138.108 TLSv1.3 2664 Certificate, Certificate Verify, Finished 19 0.004791816 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36794 [ACK] Seq=3877 Ack=3403 Win=63616 Len=0 TSval=1228605233 TSecr=1228605233 20 0.005226573 10.0.138.108 → 10.0.138.108 TLSv1.3 1379 New Session Ticket 21 0.005239616 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=3403 Ack=5188 Win=64256 Len=0 TSval=1228605233 TSecr=1228605233 22 0.005363821 10.0.138.108 → 10.0.138.108 TLSv1.3 1379 New Session Ticket 23 0.005367288 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=3403 Ack=6499 Win=64256 Len=0 TSval=1228605233 TSecr=1228605233 24 0.020897783 10.0.138.108 → 10.0.138.108 HTTP 1922 HTTP/1.1 200 200 (text/html) 25 0.020904818 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [ACK] Seq=3403 Ack=8353 Win=64000 Len=0 TSval=1228605249 TSecr=1228605249 26 0.021939091 10.0.138.108 → 10.0.138.108 TCP 68 36794 → 443 [FIN, ACK] Seq=3403 Ack=8353 Win=65536 Len=0 TSval=1228605250 TSecr=1228605249 27 0.022087729 10.0.138.108 → 10.0.138.108 TLSv1.3 92 Alert (Level: Warning, Description: Close Notify) 28 0.022112779 10.0.138.108 → 10.0.138.108 TCP 56 36794 → 443 [RST] Seq=3404 Win=0 Len=0 29 0.030330047 10.0.138.108 → 10.0.138.108 TCP 76 36796 → 443 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605258 TSecr=0 WS=128 30 0.030348996 10.0.138.108 → 10.0.138.108 TCP 76 443 → 36796 [SYN, ACK] Seq=0 Ack=1 Win=65483 Len=0 MSS=65495 SACK_PERM=1 TSval=1228605258 TSecr=1228605258 WS=128 31 0.030357883 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=1228605258 TSecr=1228605258 32 0.030889786 10.0.138.108 → 10.0.138.108 TLSv1 585 Client Hello 33 0.030906029 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=1 Ack=518 Win=65024 Len=0 TSval=1228605259 TSecr=1228605259 34 0.032502581 10.0.138.108 → 10.0.138.108 TLSv1.3 3171 Server Hello, Change Cipher Spec, Encrypted Extensions, Certificate, Certificate Verify, Finished 35 0.032521963 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=518 Ack=3104 Win=63360 Len=0 TSval=1228605261 TSecr=1228605261 36 0.032968162 10.0.138.108 → 10.0.138.108 TLSv1.3 148 Change Cipher Spec, Finished 37 0.032981319 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=3104 Ack=598 Win=65536 Len=0 TSval=1228605261 TSecr=1228605261 38 0.033130757 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket 39 0.033135265 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=598 Ack=3407 Win=65280 Len=0 TSval=1228605261 TSecr=1228605261 40 0.033148744 10.0.138.108 → 10.0.138.108 TLSv1.3 298 [TLS segment of a reassembled PDU] 41 0.033151866 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=3407 Ack=828 Win=65408 Len=0 TSval=1228605261 TSecr=1228605261 42 0.033165106 10.0.138.108 → 10.0.138.108 HTTP 114 POST /ca/agent/ca/displayBySerial HTTP/1.1 43 0.033167675 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=3407 Ack=874 Win=65408 Len=0 TSval=1228605261 TSecr=1228605261 44 0.033180766 10.0.138.108 → 10.0.138.108 TLSv1.3 371 New Session Ticket 45 0.033183758 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=874 Ack=3710 Win=65024 Len=0 TSval=1228605261 TSecr=1228605261 46 0.033299383 10.0.138.108 → 10.0.138.108 TLSv1.3 235 Certificate Request 47 0.033316804 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=874 Ack=3877 Win=64896 Len=0 TSval=1228605261 TSecr=1228605261 48 0.033399513 10.0.138.108 → 10.0.138.108 HTTP 570 HTTP/1.1 403 Forbidden (text/html) 49 0.033402002 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=874 Ack=4379 Win=64768 Len=0 TSval=1228605261 TSecr=1228605261 50 0.034499630 10.0.138.108 → 10.0.138.108 TLSv1.3 2664 Certificate, Certificate Verify, Finished 51 0.034504331 10.0.138.108 → 10.0.138.108 TCP 68 443 → 36796 [ACK] Seq=4379 Ack=3470 Win=63616 Len=0 TSval=1228605263 TSecr=1228605263 52 0.035032328 10.0.138.108 → 10.0.138.108 TLSv1.3 1379 New Session Ticket 53 0.035038891 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [ACK] Seq=3470 Ack=5690 Win=64256 Len=0 TSval=1228605263 TSecr=1228605263 54 0.035091948 10.0.138.108 → 10.0.138.108 TCP 68 36796 → 443 [RST, ACK] Seq=3470 Ack=5690 Win=65536 Len=0 TSval=1228605263 TSecr=1228605263
Created attachment 1638448 [details] tshark dump for another connection attempt
Created attachment 1638449 [details] keylog file for the pcap file
Ah, yes, I see the problem, the code to buffer the request body is not invoked in the TLSv1.3 reneg path. Thanks for the report.
You are welcome! Is the bug fix simple or complicated? The bug affects our FIPS compliance effort. In IdM we would like to purely rely on crypto policy instead of config flags for ciphers and TLS versions. The crypto policies in RHEL and Fedora have TLS 1.3 enabled.
Fairly simple, I am testing it right now.
https://github.com/apache/httpd/pull/75
Commit: https://src.fedoraproject.org/rpms/httpd/c/401f5d1ca77b71fa5dd446d556ad9c795f3261b0
FEDORA-2019-ae1dd32c5f has been submitted as an update to Fedora 31. https://bodhi.fedoraproject.org/updates/FEDORA-2019-ae1dd32c5f
Can you test this build with FreeIPA? https://koji.fedoraproject.org/koji/buildinfo?buildID=1415659
Sure! I'll test it tomorrow after the build has reached testing.
httpd-2.4.41-9.fc31 has been pushed to the Fedora 31 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-ae1dd32c5f
FreeIPA PR https://github.com/freeipa/freeipa/pull/3911with TLS 1.3 enabled is now passing basic tests. TLS 1.3 post-handshake client cert auth of POST requests is working as expected. Could you please backport the fix to Fedora 30, too? Thanks for the quick fix!
I'm setting F30 as target. The TLS 1.3 change are going to land in IPA 4.8.3 and we plan to release 4.8.3 for F30, too.
Commit: https://src.fedoraproject.org/rpms/httpd/c/1f4a25cb669818590d44b0ddd1ffa5127ac99d8d.
Package: httpd-2.4.41-6.1.fc30 Build: https://koji.fedoraproject.org/koji/buildinfo?buildID=1415917
FEDORA-2019-d54e892077 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-d54e892077
httpd-2.4.41-6.1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-d54e892077
httpd-2.4.41-9.fc31 has been pushed to the Fedora 31 stable repository. If problems still persist, please make note of it in this bug report.
httpd-2.4.41-6.1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.
I filed the backport bug https://bz.apache.org/bugzilla/show_bug.cgi?id=64242 to get it merged into httpd-2.4.x.