Bug 1785509
Summary: | cockpit-ws cannot process longer HTTPS request than 4096 bytes | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Masahiro Matsuya <mmatsuya> |
Component: | cockpit | Assignee: | Martin Pitt <mpitt> |
Status: | CLOSED ERRATA | QA Contact: | Jan Ščotka <jscotka> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 8.0 | CC: | mpitt, release-test-team-automation |
Target Milestone: | rc | Keywords: | Patch |
Target Release: | 8.2 | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | 1785497 | Environment: | |
Last Closed: | 2020-04-28 16:51:35 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1785497 | ||
Bug Blocks: |
Description
Masahiro Matsuya
2019-12-20 04:46:15 UTC
Thanks for your report! I did an initial attempt at reproducing this, with the debug patch below. But that doesn't reproduce it. A small request with $ curl -H "Authorization: Negotiate AAAAA" http://localhost:9999/cockpit/login is correctly plumbed through the web server to cockpit-session: cockpit-session: authorize message: Negotiate AAAAA But so is a large request > 4KiB: $ curl -H "Authorization: Negotiate $(printf '%0.5000i' 1)" http://localhost:9999/cockpit/login cockpit-session: authorize message: Negotiate 00[.... 5000 more '0's ...]01 It only breaks down at 8 KiB: $ curl -H "Authorization: Negotiate $(printf '%0.8200i' 1)" http://localhost:9999/cockpit/login curl: (52) Empty reply from server cockpit-protocol-Message: 08:27:09.360: received HTTP request that was too large But this is not affected by your patch, it's enforced in parse_and_process_request(): /* The hard input limit, we just terminate the connection */ if (request->buffer->len > cockpit_webserver_request_maximum * 2) { g_message ("received HTTP request that was too large"); goto out; } (cockpit_webserver_request_maximum is 4096). So it seems what's going on is slightly more complicated than this naïve attempt at reproducing. --- src/ws/cockpitauth.c +++ src/ws/cockpitauth.c @@ -53,7 +53,7 @@ /* Some tunables that can be set from tests */ const gchar *cockpit_ws_session_program = - PACKAGE_LIBEXEC_DIR "/cockpit-session"; + "./cockpit-session"; const gchar *cockpit_ws_ssh_program = PACKAGE_LIBEXEC_DIR "/cockpit-ssh"; diff --git src/ws/session-utils.c src/ws/session-utils.c index b5d074926..6bd871b9b 100644 --- src/ws/session-utils.c +++ src/ws/session-utils.c @@ -79,6 +79,7 @@ read_authorize_response (const char *what) len -= auth_prefix_size + auth_response_size + auth_suffix_size; memmove (message, message + auth_prefix_size + auth_response_size, len); message[len] = '\0'; + debug ("authorize message: %s", message); return (char *)message; } diff --git src/ws/session-utils.h src/ws/session-utils.h index a64c37858..eb9a68936 100644 --- src/ws/session-utils.h +++ src/ws/session-utils.h @@ -38,7 +38,7 @@ #include "common/cockpitauthorize.h" #include "common/cockpitmemory.h" -#define DEBUG_SESSION 0 +#define DEBUG_SESSION 1 #define EX 127 #define DEFAULT_PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" and run `G_MESSAGES_DEBUG=all ./cockpit-ws --no-tls -p 9999` If there is more data to read from the request than 4KiB, on_request_input() is just being called several times. After the first 4K, the header is still incomplete, and this is supposed to be caught by this in parse_and_process_request(): off2 = web_socket_util_parse_headers ((const gchar *)request->buffer->data + off1, request->buffer->len - off1, &headers); if (off2 == 0) { again = TRUE; goto out; } i. e. if this hits, it ignores the currently pending buffer, and goes back to reading more input. web_socket_util_parse_headers() returns 0 mainly if there is no line ending (\n) in the request. I stepped through the code with various scenarios such as adding more headers before and after "Authorization:", but I still can't see how this fails. Increasing the block size to 8192 fixes your case "by accident", but with slightly different data or network I/O patterns (fragmenting the packets differently) a large request might still be split into multiple smaller send()/recv() blocks, so I don't think this is a robust solution. As a next step I tried this in a real IdM (FreeIPA) environment with kerberos negotiation. With no functional changes, just some extra debugging, it works fine (as expected, the Negotiate header is 867 bytes): $ sudo -u cockpit-wsinstance G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ws --port 9999 --no-tls $ curl --negotiate -u: http://x0.cockpit.lan:9999/cockpit/login {"csrf-token":"3fe28d3fe7d9beee2277abc63aad7ee8359a0f87b097e9daa43df0ce349930f5"} cockpit-session: authorize message: Negotiate YIIC[...]GLE So now I tried to reproduce it with going the opposite direction -- the read chunk length that you bumped from 4096 to 8192 I changed to 512, so that my 867 byte Negotiate header does not fit in any more. But this still worked, I get a valid login token, and the header was parsed correctly in multiple chunks: (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.466: XXX on_request_input current length 0; adding 512 (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.466: XXX on_request_input current read 512 bytes (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input set new buffer size to 512 (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX parse_and_process_request, current size 512 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 27 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Host val 'x0.cockpit.lan:9999' (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers no EOL in buffer, ignoring (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input current length 512; adding 512 (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input current read 467 bytes (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input set new buffer size to 979 (cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX parse_and_process_request, current size 979 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 27 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Host val 'x0.cockpit.lan:9999' (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 883 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Authorization val 'Negotiate YIICfgYGKwYBBQUCoIICcjCCAm6gDTALBgkqhkiG9xIBAgKiggJbBIICV2CCAlMGCSqGSIb3EgECAgEAboICQjCCAj6gAwIBBaEDAgEOogcDBQAgAAAAo4IBVmGCAVIwggFOoAMCAQWhDRsLQ09DS1BJVC5MQU6iITAfoAMCAQOhGDAWGwRIVFRQGw54MC5jb2NrcGl0LmxhbqOCARMwggEPoAMCARKhAwIBAaKCAQEEgf4VpQgWov+ySGYYXPWiIunKbgt093FZpMtWO/M8XTbuZLIHLT1IqHN6KXaJ7rYDxMzTV55i1mMbbaM9sw79Z89kOPNl4hl8/Q1JoBVFm4Ssad5DpymO6Kby9pi/AsPsmAvcYfFHz9owg+J7N6q79ZHhwEMdBv1OzMuR7NnvVJsYZZFt76Sv+mys70wgvIKjZUzeJDJbaxzcMecmznSJtaQ+K4lOtJWvBy3XBH85XkCL5eTWyHurGcG8GjPP+F0rwaHMdcC/BEgp8a7CSW7hDpk82/kB4+mKCZ4jv0TkI9mSLuerN94tvwHr76m+KZ+F6J3aAi5/6+djyG/BgnYYnKSBzjCBy6ADAgESooHDBIHAU9PK3SkE/ppNiNs3GFvcpO5mO3bPCIzm5yF0rTOkk/8smh33dG0Vn8evLixxpwotZzkn6wTklGcwZRgwBsmxGoBPGrTLLG6W1qFElWw8Trq17EQemXWM0LtVXeZIaMJif9zzckyvkjBOn6WfMYvqC8LTKWoOgXNLDVYXj8xWudDXz9zgCpzlB5DYQuwR+wziyEi0eGtxBtVXfZFtcAuLJFRPCVQDOD30TaEN6XU4a+r6229Wu8bD0258Ei7VAtDZ' (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 25 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header User-Agent val 'curl/7.66.0' (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 13 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Accept val '*/*' (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 2 (cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got empty line I also tried with 128 bytes, and it still works. (last patch: https://gist.github.com/martinpitt/80502247072aee4199e8b2c7942ede67) So I'm afraid I still cannot reproduce this. Masahiro, do you still have access to the affected environment? Any chance that I could get access to that, or perhaps you can try and apply the above gist patch and send me the output? Note that you don't have to put this into /usr actually, you can just run cockpit-ws as normal user out of the build tree. It won't be able to call cockpit-session then and auth will fail, but the interesting parts about the header parsing will have happened by then already. Thanks! Hello Martin, Thanks for your update. Can you try it with TLS enabled? I could reproduce the problem with that. 1) Start the cockpit-ws without --no-tls. 2) # curl -H "Authorization: Negotiate $(printf '%0.5000i' 1)" --insecure https://localhost:9999/cockpit/login ==> Anything is not output. 3) # curl -H "Authorization: Negotiate $(printf '%0.3000i' 1)" --insecure https://localhost:9999/cockpit/login ==> HTML data including "<title>Loading...</title>" was output. Thanks! Masahiro I built the debug package with the gist patch, and tested it on RHEL8.1 with TLS enabled. # sudo -u cockpit-ws G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ws --port 9999 (cockpit-ws:1827): cockpit-ws-DEBUG: 14:06:24.216: loaded 1 certificates from /etc/cockpit/ws-certs.d/0-self-signed.cert cockpit-ws-INFO: 14:06:24.216: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.737: XXX on_request_input current length 0; adding 4096 (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.738: XXX on_request_input current read -1 bytes (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.738: XXX on_request_input current length 0; adding 4096 (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.738: XXX on_request_input current read -1 bytes (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX on_request_input current length 0; adding 4096 (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX on_request_input current read 4096 bytes (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX on_request_input set new buffer size to 4096 (cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX parse_and_process_request, current size 4096 (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.745: XXX web_socket_util_parse_headers: got line of len 22 (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.745: XXX web_socket_util_parse_headers: got header Host val 'localhost:9999' (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got line of len 25 (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got header User-Agent val 'curl/7.61.1' (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got line of len 13 (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got header Accept val '*/*' (cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers no EOL in buffer, ignoring cockpit-protocol-Message: 14:07:30.900: request timed out, closing (cockpit-ws:1827): cockpit-ws-DEBUG: 14:07:54.894: auth is idle # curl -H "Authorization: Negotiate $(printf '%0.5000i' 1)" --insecure https://localhost:9999/cockpit/login curl: (52) Empty reply from server Thanks, Thanks Masahiro! Indeed it reproduces well with https, I didn't originally think that this would make a difference. So this is a bug/impedance mismatch between the block sizes acquired from TLS, versus the block sizes read by on_request_input(). This got "accidentally" fixed in RHEL 8.2 due to moving out TLS termination into cockpit-tls, so that cockpit-ws only ever speaks http. But as we need a fix for cockpit-ws for 7.8 anyway, let's just apply the fix to 8.2 as well, so that custom setups that call cockpit-ws directly work properly. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1837 |