Thanks for your report!

I did an initial attempt at reproducing this, with the debug patch below. But that doesn't reproduce it. A small request with

    $ curl -H "Authorization: Negotiate AAAAA" http://localhost:9999/cockpit/login

is correctly plumbed through the web server to cockpit-session:

    cockpit-session: authorize message: Negotiate AAAAA


But so is a large request > 4KiB:

   $ curl -H "Authorization: Negotiate $(printf '%0.5000i' 1)" http://localhost:9999/cockpit/login

   cockpit-session: authorize message: Negotiate 00[.... 5000 more '0's ...]01

It  only breaks down at 8 KiB:

   $ curl -H "Authorization: Negotiate $(printf '%0.8200i' 1)" http://localhost:9999/cockpit/login
   curl: (52) Empty reply from server

   cockpit-protocol-Message: 08:27:09.360: received HTTP request that was too large

But this is not affected by your patch, it's enforced in parse_and_process_request():

  /* The hard input limit, we just terminate the connection */
  if (request->buffer->len > cockpit_webserver_request_maximum * 2)
    {
      g_message ("received HTTP request that was too large");
      goto out;
    }

(cockpit_webserver_request_maximum is 4096).

So it seems what's going on is slightly more complicated than this naïve attempt at reproducing.



--- src/ws/cockpitauth.c
+++ src/ws/cockpitauth.c
@@ -53,7 +53,7 @@
 
 /* Some tunables that can be set from tests */
 const gchar *cockpit_ws_session_program =
-    PACKAGE_LIBEXEC_DIR "/cockpit-session";
+    "./cockpit-session";
 
 const gchar *cockpit_ws_ssh_program =
     PACKAGE_LIBEXEC_DIR "/cockpit-ssh";
diff --git src/ws/session-utils.c src/ws/session-utils.c
index b5d074926..6bd871b9b 100644
--- src/ws/session-utils.c
+++ src/ws/session-utils.c
@@ -79,6 +79,7 @@ read_authorize_response (const char *what)
   len -= auth_prefix_size + auth_response_size + auth_suffix_size;
   memmove (message, message + auth_prefix_size + auth_response_size, len);
   message[len] = '\0';
+  debug ("authorize message: %s", message);
   return (char *)message;
 }
 
diff --git src/ws/session-utils.h src/ws/session-utils.h
index a64c37858..eb9a68936 100644
--- src/ws/session-utils.h
+++ src/ws/session-utils.h
@@ -38,7 +38,7 @@
 #include "common/cockpitauthorize.h"
 #include "common/cockpitmemory.h"
 
-#define DEBUG_SESSION 0
+#define DEBUG_SESSION 1
 #define EX 127
 #define DEFAULT_PATH "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
 
and run `G_MESSAGES_DEBUG=all ./cockpit-ws --no-tls -p 9999`

If there is more data to read from the request than 4KiB, on_request_input() is just being called several times. After the first 4K, the header is still incomplete, and this is supposed to be caught by this in parse_and_process_request():

  off2 = web_socket_util_parse_headers ((const gchar *)request->buffer->data + off1,
                                        request->buffer->len - off1,
                                        &headers);
  if (off2 == 0)
    {
      again = TRUE;
      goto out;
    }

i. e. if this hits, it ignores the currently pending buffer, and goes back to reading more input. web_socket_util_parse_headers() returns 0 mainly if there is no line ending (\n) in the request. I stepped through the code with various scenarios such as adding more headers before and after "Authorization:", but I still can't see how this fails.

Increasing the block size to 8192 fixes your case "by accident", but with slightly different data or network I/O patterns (fragmenting the packets differently) a large request might still be split into multiple smaller send()/recv() blocks, so I don't think this is a robust solution.

As a next step I tried this in a real IdM (FreeIPA) environment with kerberos negotiation. With no functional changes, just some extra debugging, it works fine (as expected, the Negotiate header is 867 bytes):

    $ sudo -u cockpit-wsinstance G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ws --port 9999 --no-tls

    $ curl --negotiate -u: http://x0.cockpit.lan:9999/cockpit/login
    {"csrf-token":"3fe28d3fe7d9beee2277abc63aad7ee8359a0f87b097e9daa43df0ce349930f5"}


    cockpit-session: authorize message: Negotiate YIIC[...]GLE

So now I tried to reproduce it with going the opposite direction -- the read chunk length that you bumped from 4096 to 8192 I changed to 512, so that my 867 byte Negotiate header does not fit in any more. But this still worked, I get a valid login token, and the header was parsed correctly in multiple chunks:

(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.466: XXX on_request_input current length 0; adding 512
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.466: XXX on_request_input current read 512 bytes
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input set new buffer size to 512
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX parse_and_process_request, current size 512
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 27
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Host val 'x0.cockpit.lan:9999'
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers no EOL in buffer, ignoring
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input current length 512; adding 512
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input current read 467 bytes
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX on_request_input set new buffer size to 979
(cockpit-ws:31270): cockpit-protocol-DEBUG: 03:38:15.467: XXX parse_and_process_request, current size 979
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 27
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Host val 'x0.cockpit.lan:9999'
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 883
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Authorization val 'Negotiate YIICfgYGKwYBBQUCoIICcjCCAm6gDTALBgkqhkiG9xIBAgKiggJbBIICV2CCAlMGCSqGSIb3EgECAgEAboICQjCCAj6gAwIBBaEDAgEOogcDBQAgAAAAo4IBVmGCAVIwggFOoAMCAQWhDRsLQ09DS1BJVC5MQU6iITAfoAMCAQOhGDAWGwRIVFRQGw54MC5jb2NrcGl0LmxhbqOCARMwggEPoAMCARKhAwIBAaKCAQEEgf4VpQgWov+ySGYYXPWiIunKbgt093FZpMtWO/M8XTbuZLIHLT1IqHN6KXaJ7rYDxMzTV55i1mMbbaM9sw79Z89kOPNl4hl8/Q1JoBVFm4Ssad5DpymO6Kby9pi/AsPsmAvcYfFHz9owg+J7N6q79ZHhwEMdBv1OzMuR7NnvVJsYZZFt76Sv+mys70wgvIKjZUzeJDJbaxzcMecmznSJtaQ+K4lOtJWvBy3XBH85XkCL5eTWyHurGcG8GjPP+F0rwaHMdcC/BEgp8a7CSW7hDpk82/kB4+mKCZ4jv0TkI9mSLuerN94tvwHr76m+KZ+F6J3aAi5/6+djyG/BgnYYnKSBzjCBy6ADAgESooHDBIHAU9PK3SkE/ppNiNs3GFvcpO5mO3bPCIzm5yF0rTOkk/8smh33dG0Vn8evLixxpwotZzkn6wTklGcwZRgwBsmxGoBPGrTLLG6W1qFElWw8Trq17EQemXWM0LtVXeZIaMJif9zzckyvkjBOn6WfMYvqC8LTKWoOgXNLDVYXj8xWudDXz9zgCpzlB5DYQuwR+wziyEi0eGtxBtVXfZFtcAuLJFRPCVQDOD30TaEN6XU4a+r6229Wu8bD0258Ei7VAtDZ'
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 25
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header User-Agent val 'curl/7.66.0'
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 13
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got header Accept val '*/*'
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got line of len 2
(cockpit-ws:31270): WebSocket-DEBUG: 03:38:15.467: XXX web_socket_util_parse_headers: got empty line


I also tried with 128 bytes, and it still works. (last patch: https://gist.github.com/martinpitt/80502247072aee4199e8b2c7942ede67)

So I'm afraid I still cannot reproduce this.

Masahiro, do you still have access to the affected environment? Any chance that I could get access to that, or perhaps you can try and apply the above gist patch and send me the output? Note that you don't have to put this into /usr actually, you can just run cockpit-ws as normal user out of the build tree. It won't be able to call cockpit-session then and auth will fail, but the interesting parts about the header parsing will have happened by then already.

Thanks!

Hello Martin,

Thanks for your update.

Can you try it with TLS enabled?
I could reproduce the problem with that.

1) Start the cockpit-ws without --no-tls.
2) # curl -H "Authorization: Negotiate $(printf '%0.5000i' 1)" --insecure https://localhost:9999/cockpit/login

   ==> Anything is not output.

3) # curl -H "Authorization: Negotiate $(printf '%0.3000i' 1)" --insecure https://localhost:9999/cockpit/login

   ==> HTML data including "<title>Loading...</title>" was output.

Thanks!

Masahiro

I built the debug package with the gist patch, and tested it on RHEL8.1 with TLS enabled.

# sudo -u cockpit-ws G_MESSAGES_DEBUG=all /usr/libexec/cockpit-ws --port 9999
(cockpit-ws:1827): cockpit-ws-DEBUG: 14:06:24.216: loaded 1 certificates from /etc/cockpit/ws-certs.d/0-self-signed.cert
cockpit-ws-INFO: 14:06:24.216: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.737: XXX on_request_input current length 0; adding 4096
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.738: XXX on_request_input current read -1 bytes
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.738: XXX on_request_input current length 0; adding 4096
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.738: XXX on_request_input current read -1 bytes
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX on_request_input current length 0; adding 4096
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX on_request_input current read 4096 bytes
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX on_request_input set new buffer size to 4096
(cockpit-ws:1827): cockpit-protocol-DEBUG: 14:07:00.745: XXX parse_and_process_request, current size 4096
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.745: XXX web_socket_util_parse_headers: got line of len 22
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.745: XXX web_socket_util_parse_headers: got header Host val 'localhost:9999'
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got line of len 25
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got header User-Agent val 'curl/7.61.1'
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got line of len 13
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers: got header Accept val '*/*'
(cockpit-ws:1827): WebSocket-DEBUG: 14:07:00.746: XXX web_socket_util_parse_headers no EOL in buffer, ignoring
cockpit-protocol-Message: 14:07:30.900: request timed out, closing
(cockpit-ws:1827): cockpit-ws-DEBUG: 14:07:54.894: auth is idle


# curl -H "Authorization: Negotiate $(printf '%0.5000i' 1)" --insecure https://localhost:9999/cockpit/login
curl: (52) Empty reply from server


Thanks,

Thanks Masahiro! Indeed it reproduces well with https, I didn't originally think that this would make a difference. So this is a bug/impedance mismatch between the block sizes acquired from TLS, versus the block sizes read by on_request_input().

This got "accidentally" fixed in RHEL 8.2 due to moving out TLS termination into cockpit-tls, so that cockpit-ws only ever speaks http. But as we need a fix for cockpit-ws for 7.8 anyway, let's just apply the fix to 8.2 as well, so that custom setups that call cockpit-ws directly work properly.

https://github.com/cockpit-project/cockpit/pull/13351

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1837