This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1478213 - ssh port 22 connection refused error
ssh port 22 connection refused error
Status: CLOSED NOTABUG
Product: Fedora
Classification: Fedora
Component: openssh (Show other bugs)
26
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Jakub Jelen
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-08-03 22:05 EDT by Gregory David Martinez
Modified: 2017-08-07 03:32 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-07 03:32:04 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Gregory David Martinez 2017-08-03 22:05:10 EDT
Description of problem:
But after upgrading to Fedora 26, I now get the following error when trying to ssh into a remote computer:

Connection closed by 169.232.151.211 port 22

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Install Fedora 26
2. ssh into any computer that you were able to ssh into before upgrading

Actual results:
ssh into computer

Expected results:
Connection closed by 169.232.151.211 port 22

Additional info:

This is what I know:

* It's not a router issue since I am able to login with other non-Fedora 26 computers. 

* It's not a firewall issue since the problem persists if I shutdown the firewall (via service firewalld stop). 

* It's not a selinux issue because the problem persists is a set selinux to permissive.

* This is specific to Fedora 26 as Fedora 25 works fine.
Comment 1 Jakub Jelen 2017-08-04 02:38:38 EDT
Please, post a debug logs from the client and server. Also check the status of the sshd service. From what you wrote so far, it is not possible to help you.
Comment 2 Gregory David Martinez 2017-08-04 02:49:33 EDT
Here is the output of "ssh -vvvv":

OpenSSH_7.5p1, OpenSSL 1.1.0f-fips  25 May 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: /etc/ssh/ssh_config line 56: Including file /etc/ssh/ssh_config.d/05-redhat.conf depth 0
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug3: /etc/ssh/ssh_config.d/05-redhat.conf line 2: Including file /etc/crypto-policies/back-ends/openssh.config depth 1
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug3: gss kex names ok: [gss-gex-sha1-,gss-group14-sha1-]
debug3: kex names ok: [curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1]
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug2: resolving "ixchel.astro.ucla.edu" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ixchel.astro.ucla.edu [169.232.151.213] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/gmartine/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH* compat 0x04000000
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to ixchel.astro.ucla.edu:22 as 'gmartine'
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc,3des-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc,3des-cbc
debug2: MACs ctos: umac-128-etm@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1,hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha2-512-etm@openssh.com,hmac-sha2-512
debug2: MACs stoc: umac-128-etm@openssh.com,umac-128@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha1,hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha2-512-etm@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: <implicit> compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32
debug1: kex: diffie-hellman-group-exchange-sha256 need=32 dh_need=32
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
Connection closed by 169.232.151.213 port 22

Here is the output of "service sshd status":

# service sshd status
Redirecting to /bin/systemctl status sshd.service
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-08-03 23:45:18 PDT; 3s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 18109 (sshd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/sshd.service
           └─18109 /usr/sbin/sshd -D

Aug 03 23:45:18 MrComputer systemd[1]: Starting OpenSSH server daemon...
Aug 03 23:45:18 MrComputer sshd[18109]: Server listening on 0.0.0.0 port 22.
Aug 03 23:45:18 MrComputer sshd[18109]: Server listening on :: port 22.
Aug 03 23:45:18 MrComputer systemd[1]: Started OpenSSH server daemon.

Here is the output of "/usr/sbin/sshd -D -d":

# /usr/sbin/sshd -D -d
debug1: sshd version OpenSSH_7.5, OpenSSL 1.1.0f-fips  25 May 2017
debug1: private host key #0: ssh-rsa SHA256:oDBKXsUs3LgRjBPcmL71i+CSlRrz5xnbOTc8eKCE1uo
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:DY+0Ldo7yXDpd0uHhHWyQbgNtxZNNz1Zh1uW3Z3bH9k
debug1: private host key #2: ssh-ed25519 SHA256:WS0CVVQGtp8np5vO4TaTcUMd0t+AHiGpZX7UVRpchdI
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
Comment 3 Jakub Jelen 2017-08-04 03:08:18 EDT
I would need the logs from the sever preferably when you are connecting there and when you see the failure in the client, preferably with LogLevel DEBUG3 in sshd_config.
Comment 4 Gregory David Martinez 2017-08-04 15:13:45 EDT
Hello,

Sorry for the delay, I needed to be physically at the computer to get the logs.  When trying to ssh into the computer, the sshd seems to be giving:

fatal: matching cipher is not supported: aes256-gcm@openssh.com [preauth]


I checked "man sshd_config" and the "aes256-gcm@openssh.com" cipher is suppose to be supported on the server side.  And I was able to log in with another cipher.  It seems that when using openssh client version 7.4p1, the aes128-ctr cipher is used, but with the 7.5p1 client the aes256-gcm@openssh.com cipher is used.

So, I tried using the "aes256-gcm@openssh.com" cipher using a 7.4p1 client and it failed with the same error, which makes me think the problem is on the server side.  So I checked the server openssh version, which is 7.1p2 (we are mostly using macs at work, and it seems that mac don't regularly update their openssh version).  To confirm that it was on the server side, I tried logging into a newer computer with a openssh version of 7.4p1 using the "aes256-gcm@openssh.com" cipher, and it worked fine.

Thus, in short, it seems what's going on is that there is something wrong with using the "aes256-gcm@openssh.com" cipher with a openssh 7.1p2 server.  And this problem appeared in Federa 26 because the openssh 7.5p1 client defaulted to using this cipher whereas previous openssh client versions defaulted to using the "aes128-ctr" cipher instead.
Comment 5 Gregory David Martinez 2017-08-04 17:03:35 EDT
Update, I changed the line in /etc/ssh/ssh_config from

# Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc

to 

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc

figuring it didn't include the dreaded "aes256-gcm@openssh.com" cipher.  Now it works fine.
Comment 6 Jakub Jelen 2017-08-07 03:32:04 EDT
Sigh ... this is a problem of a server, that is offering a cipher it does not know how to use. If I remember I saw this issue with some Suse server (in this case OpenSSH 6.2) with missing support for GCM ciphers in OpenSSL.

You should really contact your server administrator, who should work on fixing this somehow. Sooner or later, there will be more people hitting the same problem. There is no way how we can fix broken servers in clients.

Note You need to log in before you can comment on or make changes to this bug.