Bug 2037967

Created attachment 1849354 [details]
Apache error log on the host

Created attachment 1849354 [details]
Apache error log on the host

Created attachment 1849354 [details]
Apache error log on the host

Created attachment 1849354 [details]
Apache error log on the host


Description of problem:
When trying to reach a container running inside a virtual machine, the websocket connection closes prematurely.
I believe this causes by the Apache reverse proxy (on the host) not handling the rewrite of the websockets correctly.



Steps to Reproduce / the setup:


Diagram
Internet <--> Host with a reverse proxy <--> virtual machine with a container


Host configuration:
RHEL 8.5
Apache 2.4.37


VirtualHost/reverse proxy on the host:
<VirtualHost *:3000>
    ServerName [my.domain.com]
    LogLevel trace8

    RewriteEngine On
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule /(.*) ws://192.168.122.237:3000/$1 [P,L]

    ProxyPreserveHost On
    ProxyPass / http://192.168.122.237:3000/
    ProxyPassReverse / http://192.168.122.237:3000/

    SSLCertificateFile /etc/letsencrypt/live/[my.domain.com]/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/[my.domain.com]/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLEngine on
</VirtualHost>


Virtual machine configuration:
RHEL 8.5
A container with OpenVSCode (an open source online IDE)

The container is run with the following command:
sudo podman run -it --init -p 3000:3000 -v "$(pwd):/var/www/html:cached" gitpod/openvscode-server



When I open the address:
https://[my.domain.com]:3000/?tkn=[security token]

I get the following error message: 
"The workbench failed to connect to the server (Error: WebSocket close with status code 1006)"



After a discussion with the developers of OpenVSCode (see links below) and a specialist from the Apache mailing list, I have to conclude that the Apache rewrites rules don't seem to work correctly in combination with websockets.

With the rewrite rules I mean the following directives in the VirtualHost / reverse proxy:
RewriteEngine On
RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule /(.*) ws://192.168.122.237:3000/$1 [P,L]



After sending my Apache error log (1. see attachment, 2. LogLevel trace8 was enabled), this is what the specialist from the Apache mailing list said: "It appears the backend closes the websockets tunnel immediately."


According to the developers of OpenVSCode my reverse proxy configuration is fine and the problem can only be solved by upgrading to Apache 2.4.48 or later. However, I can't do that because RHEL 8.5 only supports Apache 2.4.37.


I don't know what exactly goes wrong in the background and what solved the problem in later versions of Apache (if this is true), but I do hope that the fix (whatever it is/was) can be backported to version 2.4.37.




Discussion on Github with the developers of OpenVSCode:
https://github.com/gitpod-io/openvscode-server/discussions/278
https://github.com/gitpod-io/openvscode-server/discussions/96