Bug 2037967 - Apache reverse proxy doesn't seem to handle websockets correctly [NEEDINFO]
Summary: Apache reverse proxy doesn't seem to handle websockets correctly
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: httpd
Version: 8.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Luboš Uhliarik
QA Contact: rhel-cs-infra-services-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-01-06 22:07 UTC by Verhoeckx
Modified: 2023-07-06 07:28 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: ---
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-07-06 07:28:06 UTC
Type: Bug
Target Upstream Version:
Embargoed:
j.verhoeckx: needinfo? (luhliari)
j.verhoeckx: needinfo? (luhliari)


Attachments (Terms of Use)
Apache error log on the host (2.42 MB, text/plain)
2022-01-06 22:07 UTC, Verhoeckx
no flags Details
Diagram of the software setup (382.90 KB, image/png)
2022-01-06 22:08 UTC, Verhoeckx
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-107027 0 None None None 2022-01-06 22:11:53 UTC

Description Verhoeckx 2022-01-06 22:07:41 UTC
Created attachment 1849354 [details]
Apache error log on the host

Created attachment 1849354 [details]
Apache error log on the host

Created attachment 1849354 [details]
Apache error log on the host

Created attachment 1849354 [details]
Apache error log on the host


Description of problem:
When trying to reach a container running inside a virtual machine, the websocket connection closes prematurely.
I believe this causes by the Apache reverse proxy (on the host) not handling the rewrite of the websockets correctly.



Steps to Reproduce / the setup:


Diagram
Internet <--> Host with a reverse proxy <--> virtual machine with a container


Host configuration:
RHEL 8.5
Apache 2.4.37


VirtualHost/reverse proxy on the host:
<VirtualHost *:3000>
    ServerName [my.domain.com]
    LogLevel trace8

    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule /(.*) ws://192.168.122.237:3000/$1 [P,L]

    ProxyPreserveHost On
    ProxyPass / http://192.168.122.237:3000/
    ProxyPassReverse / http://192.168.122.237:3000/

    SSLCertificateFile /etc/letsencrypt/live/[my.domain.com]/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/[my.domain.com]/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLEngine on
</VirtualHost>


Virtual machine configuration:
RHEL 8.5
A container with OpenVSCode (an open source online IDE)

The container is run with the following command:
sudo podman run -it --init -p 3000:3000 -v "$(pwd):/var/www/html:cached" gitpod/openvscode-server



When I open the address:
https://[my.domain.com]:3000/?tkn=[security token]

I get the following error message: 
"The workbench failed to connect to the server (Error: WebSocket close with status code 1006)"



After a discussion with the developers of OpenVSCode (see links below) and a specialist from the Apache mailing list, I have to conclude that the Apache rewrites rules don't seem to work correctly in combination with websockets.

With the rewrite rules I mean the following directives in the VirtualHost / reverse proxy:
RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule /(.*) ws://192.168.122.237:3000/$1 [P,L]



After sending my Apache error log (1. see attachment, 2. LogLevel trace8 was enabled), this is what the specialist from the Apache mailing list said: "It appears the backend closes the websockets tunnel immediately."


According to the developers of OpenVSCode my reverse proxy configuration is fine and the problem can only be solved by upgrading to Apache 2.4.48 or later. However, I can't do that because RHEL 8.5 only supports Apache 2.4.37.


I don't know what exactly goes wrong in the background and what solved the problem in later versions of Apache (if this is true), but I do hope that the fix (whatever it is/was) can be backported to version 2.4.37.




Discussion on Github with the developers of OpenVSCode:
https://github.com/gitpod-io/openvscode-server/discussions/278
https://github.com/gitpod-io/openvscode-server/discussions/96

Comment 1 Verhoeckx 2022-01-06 22:08:19 UTC
Created attachment 1849355 [details]
Diagram of the software setup

Comment 2 Verhoeckx 2022-01-06 22:34:19 UTC
The complete documentation of my setup can be found here:
https://jeroenverhoeckx.com/build-your-own-online-ide.html


I don't write this bug report in order to solve my problem, but I sincerely believe that it's a more general problem.

Comment 3 Verhoeckx 2022-01-21 16:52:58 UTC
Let me know if there something that I could do to help solve this possible bug!
And if it's not a bug, I also would like to hear it (and hopefully find a workaround)!

Comment 4 Verhoeckx 2022-04-19 17:37:11 UTC
Will this problem be solved in RHEL 8.6?
How can I get attention for the bug report?

Comment 6 RHEL Program Management 2023-07-06 07:28:06 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.


Note You need to log in before you can comment on or make changes to this bug.