Description of problem: I updated from openssh-clients-7.7p1 to 7.8p1-1.fc28 yesterday. I'm no longer able to open ssh connections to other systems. Especially connections to Debian systems and Fedora systems are not possible. Connections to CentOS systems still work. I use public key authentication. Version-Release number of selected component (if applicable): Working: openssh-clients-7.7p1 Broken: 7.8p1-1.fc28 How reproducible: console: ssh <debian-system> => broken console: ssh <centos-system> => OK Actual results: packet_write_wait: Connection to 192.168.58.10 port 22: Broken pipe Expected results: SSH session. Additional info: The TCP session is resetted by the destination system. I attached two tcpdumps: bug.trace: A tcpdump of the new package working.trace: A tcpdump of the old package Both dumps were created using "tcpdump -i any -w /output/path.trace"
Created attachment 1480102 [details] tcpdump of old version (working)
Workarround: sudo yum downgrade openssh-clients-7.7p1 This will downgrade openssh packages to the old version.
Created attachment 1480157 [details] tcpdump of new version (bug)
I get the same symptom on Rawhide, but for different conditions. I'm running my Linux nodes as VMware Workstation Player VMs hosted on Win7. On Win7, I have the Cygwin64 sshd listening on Win's interface to the restricted (no forwarding) VMNET1 virtual LAN. Normally all hosts can talk SSH to each other. From my Rawhide host I get immediate broken pipe connecting to the Cygwin64 OpenSSH 7.8p1 server running on Win7. I have no problem connecting to Debian Sid (Unstable) nor to openSUSE Tumbleweed, nor through loopback to the same Rawhide host. Using verbose options to ssh and comparing to working clients, it appears I'm getting past authentication and it should be about at the point where a PTY would be set up on the server end. We're not getting a type 99 packet that I see on working clients. For what it's worth, I get exactly the same symptom with the Debin Sid OpenSSH 7.8p1 client, which was just upgraded yesterday. In the Debian case, I know ssh client 7.7p1 was working just the day before the upgrade, so it appears to be a 7.8p1 client issue. On the other hand, my Cygwin64 OpenSSH has also recently upgraded to 7.8p1, and I have no problems connecting from Win7 to any of the Linux VMs. Same broken pipe using scp(1) and sftp(1) as well as ssh(1) in both login mode and remote command execution mode.
I downgraded openssh package and it worked again so I stopped investigating further. But when I read the comment from Ron, who mentioned VMware Workstation Player I did some tests on a native Fedora installation. The problem does not occur (with same destination systems) when trying to connect from a native Fedora host. On the system where I can reproduce the problem I'm running Win7 as hostsystem with VMWare Workstation. The ssh client system is connected to the destination network using NAT. Looks like both mentioned setups are nearly identical. Unsure how to proceed. Close this bug report and file a new one at VMWare? Btw: The destination system logs something like "pubkey for user xy accepted". So the problem occurs in the post-authentication phase.
Workarround 2: Add the following to the ssh client configuration (e.g. ~/.ssh/config) Host * IPQoS lowdelay throughput It seems really to be a bug on the VMWare side, especially in the NAT implementation. References: https://groups.google.com/forum/#!msg/opensshunixdev/5FK67SCpPg8/Y_QsxNkQBAAJ https://groups.google.com/forum/#!msg/opensshunixdev/uNd48nGOe7A/EgZPg2CvDgAJ
Florian, Thanks for posting the workaround to ~/.ssh/config. Restoring the pre-7.8p1 defaults does work around the issue on my 7.8p1 clients. And thanks for the references to ongoing discussions, which I read. It does look like a VMware-specific issue triggered by the DSCP defaults change in OpenSSH 7.8p1. Best regards, Ron
Sorry about that. This is indeed a change in OpenSSH, but the bug is in WMWare, which is not willing to handle the new (valid) IPQoS configuration. I will leave this bug open as a landing page for others who will probably hit the same issue, but I don't think it makes sense to revert this change from upstream since it works in the rest of the world.
Agreed. Presumably the upstream change does some good. It will be good to have an active bug which supplies the workaround for those afflicted. Thanks.
Thank you indeed for leaving this bug in place. I definitely ran into this issue recently while running F28 as a VM in VMware Workstation 12. The workaround indeed worked for me as well. Thank you again!
Thanks alot for posting the workaround, I ran into this on F28 running in VMWare Workstation 14.
Posted bugreport about this to vmware community: https://communities.vmware.com/message/2803219#2803219
Disabling IPQoS server side also seems to be a valid workaround. Having all the clients who connect to our servers change their local config was not ideal. So server side workaround it is. 1) Adding "IPQoS 0x00" to /etc/ssh/sshd_config 2) restart sshd
Hi John, Could you help create a internal VMware bug to track this? as this is a vmware side issue. Thanks
The issue still persists in Fedora 29 + openssh-7.9p1-4.fc29 + VMWare 15.0.2. Should someone bump the version here?
I can confirm it's still an issue for Fedora Rawhide, Debian Sid (Unstable), and openSUSE Tumbleweed (all using OpenSSH 7.1p1), when connecting from a VMware guest to the VMware host system's interface (in my case the Win7 interface on vmnet1). That's as far as my Linux guests can see in my config, but I believe several others have reported a more general issue connecting from VMware guests to nodes outside the LAN provided by VMware. I'm currently working around it with the client-side IPQoS workaround. That's not an issue for me because I control all the affected clients. For other users/owners it could be quite a burden. Thank you Fedora team for pursuing what appears to be an area where VMware has fallen behind on support of current DSCP configurations.
I have filed VMware internal bug (bug#2307798)
Since this report seems to be useful for those seeking a workaround, it should be mentioned in this report that the referenced discussions indicate that the issue is with NAT configurations of the vmnet subnet, and bridged mode works. See the https://communities.vmware.com/message/2803219#2803219 discussion. I haven't actually tried that myself.
(In reply to Ron Lovell from comment #16) > I can confirm it's still an issue for Fedora Rawhide, Debian Sid (Unstable), > and > openSUSE Tumbleweed (all using OpenSSH 7.1p1), when connecting from a VMware > guest > to the VMware host system's interface (in my case the Win7 interface on > vmnet1). > That's as far as my Linux guests can see in my config, but I believe several > others > have reported a more general issue connecting from VMware guests to nodes > outside > the LAN provided by VMware. I'm currently working around it with the > client-side > IPQoS workaround. That's not an issue for me because I control all the > affected > clients. For other users/owners it could be quite a burden. > > Thank you Fedora team for pursuing what appears to be an area where VMware > has fallen > behind on support of current DSCP configurations. Correction to typo: I'm using the OpenSSH 7.9p1 client on all hosts, not 7.1p1.
This message is a reminder that Fedora 28 is nearing its end of life. On 2019-May-28 Fedora will stop maintaining and issuing updates for Fedora 28. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '28'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 28 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.