Bug 1332082

Summary: server host key signature fails after update to 7.2p2
Product: [Fedora] Fedora Reporter: mathieu.lacage
Component: opensshAssignee: Jakub Jelen <jjelen>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 22CC: jjelen, mattias.ellert, mgrepl, plautrba, tmraz
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openssh-6.9p1-12.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-05-20 23:52:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description mathieu.lacage 2016-05-02 07:18:01 UTC
Description of problem: When I ssh root@XXXXXX, I get "ssh_dispatch_run_fatal: Connection to XXXXX port 22: incorrect signature" after upgrading my client to openssh 7.2p2


Version-Release number of selected component (if applicable): 7.2p2

Last known working version: 7.1p2

How reproducible: Always

Steps to Reproduce:
1. Server: OpenSSH 6.9p1 OpenSSL 1.0.1k-fips 8 Jan 2015
2. Client: OpenSSH_7.2p2, OpenSSL 1.0.2g-fips  1 Mar 2016
3. 

Actual results: ssh_dispatch_run_fatal: Connection to XXXXX port 22: incorrect signature


Expected results: no error, connection success.


Additional info:
Full debug logs from client-side:

[mathieu@xps13 code]$ ssh -vvv root@ovh6
OpenSSH_7.2p2, OpenSSL 1.0.2g-fips  1 Mar 2016
debug1: Reading configuration data /home/mathieu/.ssh/config
debug1: /home/mathieu/.ssh/config line 5: Applying options for *
debug1: /home/mathieu/.ssh/config line 71: Applying options for ovh6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "XXXXXXXXXXXXXXXXXXxx" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to XXXXXXXXXXXXXXXXXXXXX [XXXXXXXXXXXXXXXX] port 22.
debug1: Connection established.
debug1: identity file /home/mathieu/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mathieu/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.9
debug1: match: OpenSSH_6.9 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to XXXXXXXXXXXXXXXXXXXXX:22 as 'root'
debug3: hostkeys_foreach: reading file "/home/mathieu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/mathieu/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from XXXXXXXXXXXXXXXXXXXXXXX
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-rsa-cert-v01,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01,ecdsa-sha2-nistp384-cert-v01,ecdsa-sha2-nistp521-cert-v01,ssh-ed25519-cert-v01,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,aes256-gcm,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm,aes256-gcm,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm,umac-128-etm,hmac-sha2-256-etm,hmac-sha2-512-etm,hmac-sha1-etm,umac-64,umac-128,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm,umac-128-etm,hmac-sha2-256-etm,hmac-sha2-512-etm,hmac-sha1-etm,umac-64,umac-128,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib,zlib
debug2: compression stoc: none,zlib,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: aes256-gcm,aes128-gcm,aes256-ctr,aes128-ctr
debug2: ciphers stoc: aes256-gcm,aes128-gcm,aes256-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm,hmac-sha2-256-etm,umac-128-etm,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160
debug2: MACs stoc: hmac-sha2-512-etm,hmac-sha2-256-etm,umac-128-etm,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160
debug2: compression ctos: none,zlib
debug2: compression stoc: none,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: umac-128-etm compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-128-etm compression: none
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
debug3: receive packet: type 31
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 1548/3072
debug3: send packet: type 32
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug3: receive packet: type 33
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:3dFTH1Rbkf+Y4PyYWfyPFkmZoxepgsdJnKgSikaSqEs
debug3: hostkeys_foreach: reading file "/home/mathieu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/mathieu/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from XXXXXXXXXXXXXXXXXXXXXXXXXXXX
debug3: hostkeys_foreach: reading file "/home/mathieu/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/mathieu/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from XXXXXXXXXXXXXXXx
debug1: Host 'XXXXXXXXXXXXXXXXX' is known and matches the RSA host key.
debug1: Found key in /home/mathieu/.ssh/known_hosts:4
debug2: bits set: 1575/3072
ssh_dispatch_run_fatal: Connection to XXXXXXXXXXXXXX port 22: incorrect signature

Server-side logs:
May 02 09:16:01 XXXXXXXXXX sshd[10936]: Set /proc/self/oom_score_adj to 0
May 02 09:16:01 XXXXXXXXXX sshd[10883]: debug1: Forked child 10936.
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: inetd sockets after dupping: 3, 3
May 02 09:16:01 XXXXXXXXXX sshd[10936]: Connection from XXXXXXXXXX port 65505 on 37.187.149.109 port 22
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: Client protocol version 2.0; client software version OpenSSH_7.2
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: match: OpenSSH_7.2 pat OpenSSH* compat 0x04000000
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: Enabling compatibility mode for protocol 2.0
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: Local version string SSH-2.0-OpenSSH_6.9
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: SELinux support enabled [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: ssh_selinux_change_context: setting context from 'system_u:system_r:sshd_t:s0-s0:c0.c1023' to 'system_u:system_r:sshd_net_t:s0-s0:c0.c1023' [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: permanently_set_uid: 74/74 [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: list_hostkey_types: ssh-rsa [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: SSH2_MSG_KEXINIT sent [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: SSH2_MSG_KEXINIT received [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: kex: client->server aes128-ctr umac-128-etm none [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: kex: server->client aes128-ctr umac-128-etm none [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: kex: diffie-hellman-group-exchange-sha256 need=16 dh_need=16 [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: expecting SSH2_MSG_KEX_DH_GEX_REQUEST [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
May 02 09:16:01 XXXXXXXXXX sshd[10936]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
May 02 09:16:02 XXXXXXXXXX sshd[10936]: Connection closed by XXXXXXXXXXXX [preauth]
May 02 09:16:02 XXXXXXXXXX sshd[10936]: debug1: do_cleanup [preauth]
May 02 09:16:02 XXXXXXXXXX sshd[10936]: debug1: monitor_read_log: child log fd closed
May 02 09:16:02 XXXXXXXXXX sshd[10936]: debug1: do_cleanup
May 02 09:16:02 XXXXXXXXXX sshd[10936]: debug1: Killing privsep child 10937

Comment 1 Jakub Jelen 2016-05-02 07:44:25 UTC
Seems to be related similar problem like the bug #1323622 and related to the bump of min DH key size in openssh-7.2p1.

Hash for diffie-hellman-group-exchange-sha256 is also computed from:

     uint32  min, minimal size in bits of an acceptable group
     uint32  n, preferred size in bits of the group the server will send
     uint32  max, maximal size in bits of an acceptable group

and they are different between server and client in these versions. It seems to be a problem even with upstream version. I will keep you informed.

Other key exchange menthods seems to work fine.

Comment 2 Jakub Jelen 2016-05-02 12:04:04 UTC
This is a bug only in Fedora 22 openssh server, which handles DH group exchange in wrong way. This patch for the server fixes the issue for me:

diff --git a/kexgexs.c b/kexgexs.c
index a81fd1e..f69068c 100644
--- a/kexgexs.c
+++ b/kexgexs.c
@@ -81,7 +81,7 @@ input_kex_dh_gex_request(int type, u_int32_t seq, void *ctxt)
 	    (r = sshpkt_get_end(ssh)) != 0)
 		goto out;
 	kex->nbits = nbits;
-	kex->min = min = FIPS_mode() ? DH_GRP_MIN_FIPS : DH_GRP_MIN;
+	kex->min = min;
 	kex->max = max;
 	min = MAX(FIPS_mode() ? DH_GRP_MIN_FIPS : DH_GRP_MIN, min);
 	max = MIN(DH_GRP_MAX, max);

I will issue updates soon.

Comment 3 Jakub Jelen 2016-05-02 12:23:07 UTC
Can you verify it with this f22 (candidate) build [1]?

[1] http://koji.fedoraproject.org/koji/taskinfo?taskID=13887985

Comment 4 Fedora Update System 2016-05-02 12:34:46 UTC
openssh-6.9p1-12.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e99389f35d

Comment 5 mathieu.lacage 2016-05-02 12:55:49 UTC
I can confirm that this rpm fixes the connection problem for me.

Comment 6 Fedora Update System 2016-05-03 09:25:15 UTC
openssh-6.9p1-12.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e99389f35d

Comment 7 Fedora Update System 2016-05-20 23:51:55 UTC
openssh-6.9p1-12.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.