The application server is supposed to have a size limit on incoming messages. If a message is too large, the server should reject it and close the connection. This limit is called maxMessageSize.

Two separate defects in the Undertow library (the web engine inside JBoss EAP) mean this limit is never actually enforced for text messages:

Defect 1 — The limit is never configured: When the server receives a text WebSocket message, it creates an internal buffer object called BufferedTextMessage to collect the incoming data. The code that creates
this object always passes -1 as the size limit, which means "no limit". This happens regardless of what limit the application developer specified in their code. The developer's setting is simply ignored and discarded.

For comparison, when the server handles binary messages, the code correctly reads the developer's configured limit and passes it to the buffer. Text messages were overlooked.

Defect 2 — The size check is never called during data arrival:
Even if the limit were correctly configured (which it is not, due to Defect 1), there is a second problem. When data arrives asynchronously — meaning the client sends data in chunks over time — the server processes each chunk via an event listener. That listener collects and stores each chunk of data but never calls the function that checks whether the accumulated size has exceeded the limit. The size check only runs when the connection ends, by which point all the data is already sitting in memory.

There is a open jira and a pull request for fixing defect 1 but that is still not enough to prevent DoS attack due to defect 2 :
https://redhat.atlassian.net/browse/UNDERTOW-2770
https://github.com/undertow-io/undertow/pull/1960
https://cwe.mitre.org/data/definitions/120.html