Unbounded Per-Connection Queue Growth in WebSocketServerExtensionHandler Leads to Denial of Service

A public GitHub Security Advisory (GHSA-2g37-3h88-55hc) describes the following issue:

## Summary

`WebSocketServerExtensionHandler` keeps a per-channel `Queue<List<WebSocketServerExtension>>` field named `validExtensions`. It offers one entry for every inbound `HttpRequest` and polls one entry only when the application later writes an `HttpResponse`. Nothing bounds the queue. A remote, unauthenticated peer that uses HTTP/1.1 pipelining to send requests faster than the application produces responses grows the queue without limit until the JVM exhausts heap and dies with `OutOfMemoryError`.

The affected handler is the base class of `WebSocketServerCompressionHandler`, the standard handler applications add to enable `permessage-deflate`. Any server that supports WebSocket compression is exposed on its plain HTTP port, before any WebSocket upgrade completes and before any authentication the application may perform.

## Details

The queue is declared as per-connection state in `codec-http/src/main/java/io/netty/handler/codec/http/websocketx/extensions/WebSocketServerExtensionHandler.java`:

```java
private final Queue<List<WebSocketServerExtension>> validExtensions = new ArrayDeque<>(4);
```

It is filled on the inbound path, once per request, with no size check of any kind:

```java
protected void onHttpRequestChannelRead(ChannelHandlerContext ctx, HttpRequest request) throws Exception {
    List<WebSocketServerExtension> validExtensionsList = null;
    ...
    if (validExtensionsList == null) {
        validExtensionsList = Collections.emptyList();
    }
    validExtensions.offer(validExtensionsList);      // unbounded
    super.channelRead(ctx, request);
}
```

It is drained only on the outbound path, once per response:

```java
protected void onHttpResponseWrite(ChannelHandlerContext ctx, HttpResponse response, ChannelPromise promise)
        throws Exception {
    List<WebSocketServerExtension> validExtensionsList = validExtensions.poll();
    ...
}
```

Three properties make this remotely drivable rather than merely untidy.

The fill rate is under the attacker's direct control and the drain rate is not. `onHttpRequestChannelRead` runs on the I/O thread the instant bytes are decoded, whereas `onHttpResponseWrite` runs only when the application chooses to write a response. Any application that does asynchronous work before responding, which is the common case for a WebSocket endpoint that consults a database or an auth service, drains strictly slower than an attacker can fill.

The offer happens for every `HttpRequest`, not only for WebSocket upgrades. When the request is not an upgrade the handler still offers, using `Collections.emptyList()`. Plain `GET` requests to any path therefore grow the queue.

The handler removes itself from the pipeline only after a successful `101 Switching Protocols` response. A peer that never completes an upgrade keeps the handler, and its queue, alive for the whole life of the connection.

Supplying a valid `Sec-WebSocket-Extensions: permessage-deflate` offer makes each entry a real `ArrayList` holding a `PerMessageDeflateServerExtension` instance rather than the shared empty-list singleton, which is what raises the per-entry cost from a queue slot to roughly 100 bytes.

## Proof of concept

`PocPipelineQueue.java` in this report drives the released `io.netty:netty-codec-http:4.2.17.Final` artifact through `EmbeddedChannel` and reads the queue depth reflectively. It runs three stages. Build and run with the attached `pom.xml`:

```
mvn -q -B compile exec:java
```

Observed output:

```
[websocket] WebSocketServerCompressionHandler
   requests in            : 200000
   responses written      : 0
   validExtensions depth  : 200000
   retained heap growth   : 17.9 MiB (94 bytes/request)
   bound enforced         : NO

[control] HttpContentCompressor
   requests accepted      : 128
   refused with           : IllegalStateException: maxPipelineDepth exceeded: 128
   bound enforced         : yes

[raw wire] HttpServerCodec + WebSocketServerCompressionHandler
   pipelined requests     : 50000
   bytes sent on the wire : 9750000 (195 bytes/request)
   validExtensions depth  : 50000
   retained heap growth   : 4.8 MiB
   reachable from network : YES - decoded HTTP bytes alone drive the queue
```

The first stage shows the queue growing one-to-one with inbound requests and never stopping.

The second stage is a control. `HttpContentCompressor` maintains a structurally identical per-connection queue and already carries a depth bound, so running the same harness against it shows what a bounded handler does: it refuses the 129th request. Because the control refuses and the WebSocket handler does not, the difference is the missing bound in the WebSocket handler and not an artifact of how the harness feeds requests.

The third stage removes any doubt about network reachability. It feeds raw pipelined HTTP bytes through a real `HttpServerCodec`, exactly as they would arrive from a socket, with the application consuming each request and writing nothing back. Decoded HTTP bytes alone drive the queue to 50,000 entries.

## Impact

A remote, unauthenticated attacker holding one TCP connection open can force the server to retain roughly 100 bytes of heap per pipelined request, for the lifetime of that connection, with no upper limit. Memory is reclaimed only when the connection closes, so an attacker who keeps connections open and keeps pipelining drives the server to `OutOfMemoryError`. Spreading the same traffic across many connections multiplies the effect and keeps any single connection from looking anomalous.

This is unbounded accumulation rather than an amplification bomb. The per-request heap cost is slightly below the per-request wire cost, so the attacker spends bandwidth roughly in proportion to the memory consumed. What makes it a denial of service is that the accumulation has no ceiling and is never released while the connection lives, so the attacker converts

[truncated]

Affected:
- maven:io.netty:netty-codec-http affected >= 4.1.88.Final, <= 4.1.137.Final; fixed unknown
- maven:io.netty:netty-codec-http affected >= 4.2.0.Final, <= 4.2.17.Final; fixed unknown

Fixed versions: see advisory

Advisory: https://github.com/netty/netty/security/advisories/GHSA-2g37-3h88-55hc