Description of problem: Fedora 35 is unable to tunnel SSH sessions through an SSH bastion connection using a reliable method promoted by the *nix community that has worked until Fedora 35. Specifically, this command fails in Fedora 35 but works in Fedora 33/34 and other *nix systems: $ ssh -o 'ProxyCommand=ncat --proxy-type="socks4" --proxy localhost:9998 %h %p' user@private-node - - - First, a primary SSH connection is made to a bastion and the DynamicForward (-D) feature is used to start a tunnel listening on a specific port (ex:9998) on the client. TCP clients can then send traffic through localhost:9998 to tunnel traffic through the bastion to the bastion's private network resources. In Fedora 35, the command shown above that uses the ncat command to create a proxy from SSH to the tunnel will fail. Other clients can successfully use the tunnel including Firefox with the FoxyProxy extension and curl using the proxychains utility to proxy curl traffic through the tunnel. This appears to be something broken between nmap-ncat and OpenSSH in the Fedora 35 release. I've tested the same commands to work on Fedora 33 and Fedora 34 out of the box without issue. Version-Release number of selected component (if applicable): Fedora 35 openssh-8.7p1-2.fc35.x86_64 openssh-clients-8.7p1-2.fc35.x86_64 openssh-server-8.7p1-2.fc35.x86_64 nmap-ncat-7.91-8.fc35.x86_64 How reproducible: I can consistently reproduce this with a clean Fedora 35 install using only the files provided on the Fedora 35 release ISO. Steps to Reproduce: - You will need three nodes: A Fedora 35 workstation client, a bastion host and a third 'private' host to attempt to access via a tunnel through the bastion. 1. Use the Fedora 35 Workstation ISO to do a basic install of Fedora 35. The ISO provides OpenSSH and nmap-ncat by default with no third-party repos required. 2. Establish a primary connection with the bastion and create the tunnel. (Ex: port 9998) $ ssh -vvv -l username -p 22 -D 9998 bastion-node.domain Leave this session running as it just maintains the tunnel. 3. Attempt to create an SSH connection between the client workstation and the private node. * The SSH client will execute ncat to proxy SSH through the tunnel to the remote user@private-node target. $ ssh -vvv -o 'ProxyCommand=ncat --proxy-type="socks4" --proxy localhost:9998 %h %p' user@private-node Actual results: With verbose debugging enabled, the SSH client attempting to use the tunnel (tunnel-user) will process its config file (if exists), announce the Local version string, then *immediately* report that the connection has been closed: debug1: Local version string SSH-2.0-OpenSSH_8.7 kex_exchange_identification: Connection closed by remote host Connection closed by UNKNOWN port 65535 $ If the SSH to bastion (tunnel-operator) has verbose debugging enabled, It will report 'open confirm' indicating a client attempting to use the tunnel: ...(more not shown) debug2: channel 3: open confirm rwindow 2097152 rmax 32768 But *immediately* afterwards, the SSH tunnel-operator reports it is closing its side of the connection: debug2: channel 3: read<=0 rfd 9 len 0 debug2: channel 3: read failed debug2: chan_shutdown_read: channel 3: (i0 o0 sock 9 wfd 9 efd -1 [closed]) debug2: channel 3: input open -> drain ...(more deleted) Expected results: On a working Fedora 33 system the SSH tunnel-user client connects and can send traffic through the tunnel. The SSH tunnel-operator's debug statements look similar to: [root@bank ~]# debug1: Connection to port 9998 forwarding to socks port 0 requested. debug2: fd 11 setting TCP_NODELAY debug2: fd 11 setting O_NONBLOCK debug3: fd 11 is O_NONBLOCK debug1: channel 3: new [dynamic-tcpip] debug2: channel 3: pre_dynamic: have 0 debug2: channel 3: pre_dynamic: have 14 debug2: channel 3: decode socks4 debug2: channel 3: socks4a request debug2: channel 3: decode socks4: user /0 debug2: channel 3: decode socks4a: host stor/4 debug2: channel 3: dynamic request: socks4 host stor port 22 command 1 debug3: send packet: type 90 debug3: receive packet: type 91 debug2: channel 3: open confirm rwindow 2097152 rmax 32768 (The client is now sending data through the tunnel and no debug lines are reported) When the working SSH tunnel-user client disconnects from the remote private node, the SSH tunnel-operator debug shows the connection being shutdown: 347 [root@bank ~]# debug2: channel 3: read<=0 rfd 11 len 0 348 debug2: channel 3: read failed 349 debug2: channel 3: chan_shutdown_read (i0 o0 sock 11 wfd 11 efd -1 [closed]) 350 debug2: channel 3: input open -> drain 351 debug2: channel 3: ibuf empty 352 debug2: channel 3: send eof 353 debug3: send packet: type 96 354 debug2: channel 3: input drain -> closed 355 debug3: receive packet: type 96 356 debug2: channel 3: rcvd eof 357 debug2: channel 3: output open -> drain 358 debug2: channel 3: obuf empty 359 debug2: channel 3: chan_shutdown_write (i3 o1 sock 11 wfd 11 efd -1 [closed]) 360 debug2: channel 3: output drain -> closed 361 debug3: receive packet: type 97 362 debug2: channel 3: rcvd close 363 debug3: channel 3: will not send data after close 364 debug2: channel 3: send close 365 debug3: send packet: type 97 366 debug2: channel 3: is dead 367 debug2: channel 3: garbage collecting 368 debug1: channel 3: free: direct-tcpip: listening port 9998 for stor port 22, connect from ::1 port 4 8736 to ::1 port 9998, nchannels 4 369 debug3: channel 3: status: The following connections are open: 370 #2 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 8/9/10 sock -1 cc -1) 371 #3 direct-tcpip: listening port 9998 for stor port 22, connect from ::1 port 48736 to ::1 port 999 8 (t4 r1 i3/0 o3/0 e[closed]/0 fd 11/11/-1 sock 11 cc -1) Additional info: * Using the steps to reproduce above: * Success using nmap-ncat with Fedora 33 Workstation using only ISO installation files openssh-8.4p1-1.1.fc33.x86_64 openssh-clients-8.4p1-1.1.fc33.x86_64 openssh-server-8.4p1-1.1.fc33.x86_64 nmap-ncat-7.80-5.fc33.x86_64 * Success using nmap-ncat with Fedora 34 Workstation using only ISO installation files * The SSH tunnel appears to operate successfully when the clients are not nmap-ncat * Firefox with the FoxyProxy extension using the tunnel via localhost:9998 works fine * Curl with a proxychains config using localhost:9998 works fine. * I see the same results using a bastion and private node in my home lab and using AWS cloud based bastions and private instances. * The DynamicForward (-D) feature reported here is critical to opening a single tunnel which can then be used by a variety of clients. Workarounds below using -J to get SSH through the tunnel don't offer work-arounds for browsers and CLI clients. * Providing attachment files for SSH -vvv debug output for a working connection example on Fedora 33 and a failing example on Fedora 35. Workaround(s): * Installing OpenBSD 'netcat' and using that instead of nmap-ncat appears to work fine. This suggests perhaps an issue with nmap-ncat? $ ssh -vvv -o "ProxyCommand=netcat -X 5 -x localhost:9998 %h %p" root@stor" * The 'netcat' work-around is good, but it would be nice to stick with the nmap-ncat package deployed by default with the base OS. * Newer versions of SSH have a -J option which works in Fedora 35. * ssh -J user@bastion user@private
Created attachment 1842222 [details] SSH debug output for Fedora 33 client to bastion (Successful Example) Success example: Debug output of an SSH connection establishing a connection to a bastion and a tunnel on a fresh install of Fedora 33.
Created attachment 1842226 [details] Fedora 33 Client to bastion creating tunnel output (success) Fedora 33, Successful example: SSH client to bastion creating a tunnel and showing tunnel connection activity.
Created attachment 1842227 [details] Fedora 33 Client through tunnel to private node (success) Fedora 33, Successful example: SSH client using nmap-ncat to tunnel through bastion to private node.
Created attachment 1842228 [details] Fedora 35 Client to bastion creating tunnel output (Bug scenario) Failure example: Fedora 35 SSH client to bastion node creating a tunnel. Tunnel works fine Includes activity of SSH client that attempts to go through the tunnel and immediately fails.
Created attachment 1842229 [details] Fedora 35 Client through tunnel to private node (Bug scenario, Failure) Failure example: Fedora 35 SSH client through tunnel to private node attempt Fails very quickly at "kex_exchange_identification".
I confirm that this bug impacts fedora 35 and the issue resides in the default /bin/nc executable. My workflow is using a socks5 proxy and using it to make a ssh connection using the default nc in fedora 35 will fail with debug1: kex_exchange_identification: write: Broken pipe banner exchange: Connection to UNKNOWN port 65535: Broken pipe Which clearly indicate an issue with the underlying connection as netcat is the one in charge this was my first guess and by doing the following I confirmed it Changing debug1: Executing proxy command: exec /bin/nc --proxy-type socks5 --proxy 0.0.0.0:1080 git.dev********* to debug1: Executing proxy command: exec netcat -X 5 -x 0.0.0.0:1080 git.dev********* Fixed the issue. This was introduces between fedora 34 and 35 as this only affects my newer VM Fedora 34 (fine) [r@fedora ~]$ nc -v Ncat: Version 7.80 ( https://nmap.org/ncat ) Ncat: You must specify a host to connect to. QUITTING. [r@fedora ~]$ Fedora 35 (not fine) [r@fedora ~]$ nc -v Ncat: Version 7.91 ( https://nmap.org/ncat ) Ncat: You must specify a host to connect to. QUITTING. [r@fedora ~]$ Using netcat package (not from nmap on fedora 35) fixed it: Last metadata expiration check: 0:24:24 ago on Wed 17 Nov 2021 10:50:05 AM CET. Installed Packages netcat.x86_64 1.218-2.fc35 @updates Regards, Neil
I bisected nmap repository and found the fix of this issue in the commit 169d7e5a922ef8e63b51ee2bdf4fd4ad60ed2689 If you do : git clone https://github.com/nmap/nmap cd nmap/ git reset f4fb91912d975f20499b4d903a14491471522398 --hard ./configure && make -j 12 && sudo cp ./ncat/ncat /usr/bin/ncat Then by doing any git connection using /bin/nc (which is a symlink to /etc/alternative/ncat -> /usr/bin/ncat) then it will fail with debug1: kex_exchange_identification: write: Broken pipe banner exchange: Connection to UNKNOWN port 65535: Broken pipe However if you do git clone https://github.com/nmap/nmap cd nmap/ git reset 169d7e5a922ef8e63b51ee2bdf4fd4ad60ed2689 --hard ./configure && make -j 12 && sudo cp ./ncat/ncat /usr/bin/ncat then the connection will go perfectly fine This commit add this patch if (o.ssl) { /* connect_handler creates stdin_nsi and calls post_connect */ nsock_reconnect_ssl(mypool, cs.sock_nsi, connect_handler, o.conntimeout, NULL, NULL); } else { /* Create IOD for nsp->stdin */ if ((cs.stdin_nsi = nsock_iod_new2(mypool, 0, NULL)) == NULL) bye("Failed to create stdin nsiod."); post_connect(mypool, cs.sock_nsi); } https://github.com/nmap/nmap/commit/169d7e5a922ef8e63b51ee2bdf4fd4ad60ed2689 Cheers, Neil
I tend to change the component
Thanks for the report, this is a known issue with 7.91. See https://github.com/nmap/nmap/issues/2149 *** This bug has been marked as a duplicate of bug 2020453 ***