Bug 1323622 - Upgrading to 7.2p2-1 breaks GSSAPI Authentication "Hash's MIC didn't verify"
Summary: Upgrading to 7.2p2-1 breaks GSSAPI Authentication "Hash's MIC didn't verify"
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: openssh
Version: 23
Hardware: All
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Jakub Jelen
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-04-04 09:35 UTC by Colin.Simpson
Modified: 2016-04-08 17:00 UTC (History)
6 users (show)

Fixed In Version: openssh-7.2p2-2.fc23 openssh-7.2p2-2.fc24
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-04-08 15:50:40 UTC


Attachments (Terms of Use)
Client Session (6.66 KB, text/plain)
2016-04-04 09:35 UTC, Colin.Simpson
no flags Details
Server Session (13.68 KB, text/plain)
2016-04-04 09:41 UTC, Colin.Simpson
no flags Details
Proposed patch (1.22 KB, patch)
2016-04-05 14:24 UTC, Jakub Jelen
no flags Details | Diff

Description Colin.Simpson 2016-04-04 09:35:25 UTC
Created attachment 1143276 [details]
Client Session

Description of problem:
When I upgrade openssh to openssh-7.2p2-1.fc23 

Version-Release number of selected component (if applicable):
openssh-7.2p2-1.fc23

How reproducible:
Everytime

Steps to Reproduce:
1.With a valid Kerberos ticket ssh to F23 machine running 7.2p2-1
2.Client errors with "Disconnecting: Hash's MIC didn't verify" and doesn't connect


Actual results:
"Disconnecting: Hash's MIC didn't verify"

Expected results:
A SSH session to this machine

It works again if downgraded to openssh-7.2p2-1.fc23

Additional info:
Attached the debug from client and server.

Comment 1 Colin.Simpson 2016-04-04 09:41:22 UTC
Created attachment 1143277 [details]
Server Session

Comment 2 Colin.Simpson 2016-04-04 09:52:08 UTC
Sorry typo, should have read "it works again if we downgrade to openssh-7.1p1-3.fc23"

Comment 3 Jakub Jelen 2016-04-04 10:35:36 UTC
Hi,
thank you for the report. There was a lot of releases since openssh-7.1p1-3.fc23 (since 2015-09-25). All of the versions since that date do not work? Can  you please verify that? There was a lot of changes to go through.

I will try to reproduce it in our environment. I am wondering how did it slip through our testing, because we certainly have test for GSSAPI with MIC and it looks like a reproducible one.

Comment 4 Jakub Jelen 2016-04-04 11:49:50 UTC
You are using non-default "GSSAPIDelegateCredentials" which makes a difference from my default test case. But even though I can't reproduce your behaviour with latest openssh (I am connecting from openssh-7.2p2-1.

Can you please provide what other non-default configuration do you have on client and server? I tried various configurations and I am always able to connect without any problem.

Comment 5 Colin.Simpson 2016-04-04 12:40:15 UTC
I can't see how you get the other intermediate versions?


yum list openssh --showduplicates
Yum command has been deprecated, redirecting to '/usr/bin/dnf list openssh --showduplicates'.
See 'man dnf' and 'man yum2dnf' for more information.
To transfer transaction metadata from yum to DNF, run:
'dnf install python-dnf-plugins-extras-migrate && dnf-2 migrate'

Last metadata expiration check: 2:18:43 ago on Mon Apr  4 11:17:54 2016.
Installed Packages
openssh.x86_64                                      7.2p2-1.fc23                                       @updates
Available Packages
openssh.x86_64                                      7.1p1-3.fc23                                       fedora  
openssh.x86_64                                      7.2p2-1.fc23                                       @updates
openssh.x86_64                                      7.2p2-1.fc23                                       updates 

I thought this would be it, and can't see them in the repo directly from a browser.

What did you have in mind for non-standard options? I'm thinking the sshd_config and ssh_config and maybe krb5.conf? Can there be much else that would influence this error?

Comment 6 Jakub Jelen 2016-04-04 13:00:21 UTC
Sorry I forgot to add a link to koji.

It looks like the previous versions are gone from mirrors already and I don't know why there is only that old one (the version packaged when Fedora 23 was released?).

You can download all available versions from koji [1], but you need to go through the builds to your architecture and download packages by hand (at least openssh-clients, openssh-server and openssh rpms, depending on your installation) and then (example version somewhere in the middle):

    dnf update *7.1p2-4.fc23.x86_64.rpm

or 

    dnf downgrade *7.1p2-4.fc23.x86_64.rpm

respective if you are going up or down in numbers. I would try to bisect the interval of releases.

Yes, about the config I meant client ssh_config, server sshd_config (it is partially visible from the log) and kerberos (krb5.conf) probably too.

[1] http://koji.fedoraproject.org/koji/packageinfo?packageID=96

Comment 7 Colin.Simpson 2016-04-04 13:18:00 UTC
Works with openssh-7.1p2-4 breaks with openssh-7.2p1-1

Comment 8 Jakub Jelen 2016-04-04 13:35:47 UTC
Thanks for narrowing the problem to one update. openssh 7.2 version reworked rekey handling which is probably causing the issue (still I am wondering why it does not cause problems in my setup).

Can you verify that it works for you if you drop "GSSAPIDelegateCredentials" from your your client? Basically you can do it with inline switch:

    ssh -vvv -k fedtest

Comment 9 Colin.Simpson 2016-04-04 14:14:48 UTC
Ahh maybe why you can't replicate is that it works if you used OpenSSH 7.2 to 7.2 but if one end is an earlier version 7.1 (or the version on RHEL7), it errors. 

Maybe you have both ends (clients and server) on the new version?

Comment 10 Jakub Jelen 2016-04-05 14:24:30 UTC
Created attachment 1143859 [details]
Proposed patch

Yes, you are right. We test only on the single machine with same client and server versions. Now I set up kerberos principals between rhel7 and fedora23 and I see your described behaviour. 

Digging deep enough showed, that part of MIC hash is also min

    uint32    min, minimal size in bits of an acceptable group

which is increased in openssh-7.2 and therefore the MIC does not verify (client and server are using different minimal values).

From RFC4462

   The server keeps a list of safe primes and corresponding generators
   that it can select from.  These are chosen as described in Section 3
   of [GROUP-EXCHANGE].  The client requests a modulus from the server,
   indicating the minimum, maximum, and preferred sizes; the server
   responds with a suitable modulus and generator.  The exchange then
   proceeds as described in Section 2.1 above.

  [...]

   o  min and max are the minimal and maximal sizes of p in bits that
      are acceptable to the client

This means that there is a bug in server code, sending wrong MIC signature (not based on client range, but based on subset of client and server ranges). If I read the RFC right, there is no way for server to enforce minimal key size and doing:

    min = MAX(DH_GRP_MIN, min);

is plain wrong in this case. Server can only verify, that the  nbits  value is in the server proposal range, but it can not propose different range for a client.

See proposed patch. It fixed the issue for me. Can you verify it with this scratch build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=13565607

Comment 11 Colin.Simpson 2016-04-05 15:03:19 UTC
Your patched packages work on the test systems I have here too. Both F23 to F23 plus RHEL7 to F23 work fine. 

Thanks

Comment 12 Fedora Update System 2016-04-06 11:48:35 UTC
openssh-7.2p2-2.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-8629d3fbb0

Comment 13 Fedora Update System 2016-04-06 12:10:52 UTC
openssh-7.2p2-2.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-90c69a9e2a

Comment 14 Fedora Update System 2016-04-07 16:54:46 UTC
openssh-7.2p2-2.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-8629d3fbb0

Comment 15 Fedora Update System 2016-04-07 21:19:45 UTC
openssh-7.2p2-2.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-90c69a9e2a

Comment 16 Fedora Update System 2016-04-08 15:50:32 UTC
openssh-7.2p2-2.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 17 Fedora Update System 2016-04-08 17:00:04 UTC
openssh-7.2p2-2.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.