Bug 1261414 - can't ssh to RHEL7 with GSSAPIKeyExchange
can't ssh to RHEL7 with GSSAPIKeyExchange
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: openssh (Show other bugs)
22
x86_64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: Jakub Jelen
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-09 05:33 EDT by RudraB
Modified: 2015-10-05 14:47 EDT (History)
6 users (show)

See Also:
Fixed In Version: openssh-7.1p1-3.fc23 openssh-6.9p1-8.fc22
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-10-04 18:49:24 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
ssh -vvv output (6.30 KB, text/plain)
2015-09-09 05:33 EDT, RudraB
no flags Details
ssh log (11.00 KB, text/plain)
2015-09-22 06:11 EDT, Soumyajyoti Haldar
no flags Details

  None (edit)
Description RudraB 2015-09-09 05:33:24 EDT
Created attachment 1071659 [details]
ssh -vvv output

Description of problem:
SSH login session fails with error "ssh_dispatch_run_fatal: Connection to 193.11.167.133: unexpected internal error"


Version-Release number of selected component (if applicable):
OpenSSH_6.9p1, OpenSSL 1.0.1k-fips 8 Jan 2015


How reproducible:
Everytime

Steps to Reproduce:
1. try to login to a kerberos system
2.
3.

Actual results:
Can't login. ssh -vvv output added

Expected results:


Additional info:
This seems to be similar to https://bugzilla.redhat.com/show_bug.cgi?id=1228323. 
But we have followed https://bugzilla.redhat.com/show_bug.cgi?id=1228323#c2 without any change in result.
Comment 1 Jakub Jelen 2015-09-09 10:19:52 EDT
Hi,
it would be nice to hear if the issue was introduced with last update and what version (rpm -q openssh) are you using.

Also it would be nice to hear what software is used on remote side. I see, OpenSSH_5.3, which is totally not a RHEL7 default. RHEL6 maybe?

I was just working on similar issue, so can you test with this scratch build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=11018165

If it would not help. can you provide some more information as described above?
Comment 2 Soumyajyoti Haldar 2015-09-17 06:12:08 EDT
(In reply to Jakub Jelen from comment #1)
> Hi,
> it would be nice to hear if the issue was introduced with last update and
> what version (rpm -q openssh) are you using.
> 
> Also it would be nice to hear what software is used on remote side. I see,
> OpenSSH_5.3, which is totally not a RHEL7 default. RHEL6 maybe?
> 
> I was just working on similar issue, so can you test with this scratch build:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=11018165
> 
> If it would not help. can you provide some more information as described
> above?

Hi Jakub, 

Me and RudraB together had filed this bug since we work together and had same problem as described in the bug. 

Here is the output of "rpm -q openssh"
openssh-6.9p1-7.fc22.x86_64

We have tried to downgrade to openssh-6.8p1-5.fc22.x86_64 but still the problem exist. 

We have tried two remote side machine. One machine runs OpenSSH_6.6.1 (CentOS7) and other runs OpenSSH_5.3 (Cray Linux). We face the same problem on both machines. 

I have also tried your scratch build but that does not help either.
Comment 3 Jakub Jelen 2015-09-17 10:10:18 EDT
Thank you for the report. 

Now I see the problem. You are connecting to server without host keys (PROPOSAL_SERVER_HOST_KEY_ALGS is null). I was trying to set up similar environment, but without success.

I found one part of patch that was not applied during rebase which is probably causing this issue. Can you try with this build from master?
http://koji.fedoraproject.org/koji/taskinfo?taskID=11123401

If it doesn't help or in any case I would be interested in the configuration how to reproduce it to avoid another problem in future. Do you have access to server configuration and can you share it?

Do you have some other system where you can connect successfully? I suspect it should work on openssh provided by Fedora 21 and RHEL7. Can you provide successful connect log?
Comment 4 Soumyajyoti Haldar 2015-09-22 06:11 EDT
Created attachment 1075759 [details]
ssh log

Hi Jakub, 

I tried your patch which based on openssh7.1. I got following error
"no matching key exchange method found. Their offer: gss-gex-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-group1-sha1-toWM5Slw5Ew8Mqkay+al2g==,gss-group14-sha1-toWM5Slw5Ew8Mqkay+al2g==" 

I tried some of the possible solution that I can find in different forums but it did not work. 

I do not have access to server configurations since these machines are in HPC super computing facilities. So unfortunately I can not share it with you. 

I have a working ubuntu version from which I can log in to the system.  It is OpenSSH_6.9p1 Ubuntu-2, OpenSSL 1.0.2d 9 Jul 2015 version. Here I also attach the ssh log file of the successful connection.
Comment 5 Jakub Jelen 2015-09-22 11:48:34 EDT
Hi,
thank you for your time trying. There is no difference between successful log and failed logs until the failure.

The updated version shows the same error as the old one, but is more verbose, if I am right. Can you, please, have a look at the verbose output, if the proposed gss key exchange method is in the list offered by client (or post the verbose log here again)?

I will check the patch once more, if there are any more missing pieces.
Comment 6 Jakub Jelen 2015-09-24 05:21:47 EDT
Ok. I have got the reproducer now. Removing host keys and disabling automatic creation makes the server behave as you describe.

The client is at, the moment, not offering gss key exchange methods (even if the debug contains this information). openssh-6.9 from your first report was filling them correctly. Fixing GSSAPI key exchange patch to reflect this change.

Also it appears that there is problem with server-without-keys, because it is failing to send new-hostkeys (hostkey-rotation feature). This is also fixed now. Can you give it one more try now, before I will push it to Fedora? Here is another scratch build:
http://koji.fedoraproject.org/koji/taskinfo?taskID=11208252
Comment 7 Soumyajyoti Haldar 2015-09-24 05:59:24 EDT
(In reply to Jakub Jelen from comment #6)
> Ok. I have got the reproducer now. Removing host keys and disabling
> automatic creation makes the server behave as you describe.
> 
> The client is at, the moment, not offering gss key exchange methods (even if
> the debug contains this information). openssh-6.9 from your first report was
> filling them correctly. Fixing GSSAPI key exchange patch to reflect this
> change.
> 
> Also it appears that there is problem with server-without-keys, because it
> is failing to send new-hostkeys (hostkey-rotation feature). This is also
> fixed now. Can you give it one more try now, before I will push it to
> Fedora? Here is another scratch build:
> http://koji.fedoraproject.org/koji/taskinfo?taskID=11208252

Great. This version of the scratch build is working as expected and I can log in to the server now.
Comment 8 Fedora Update System 2015-09-25 08:52:29 EDT
openssh-7.1p1-3.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2015-8774808146
Comment 9 Fedora Update System 2015-09-26 20:55:19 EDT
openssh-7.1p1-3.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update openssh'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-8774808146
Comment 10 Fedora Update System 2015-10-01 04:48:43 EDT
openssh-6.9p1-8.fc22 has been submitted as an update to Fedora 22. https://bodhi.fedoraproject.org/updates/FEDORA-2015-686c9d0140
Comment 11 Fedora Update System 2015-10-01 12:01:37 EDT
openssh-7.1p1-3.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.
Comment 12 Fedora Update System 2015-10-03 21:37:13 EDT
openssh-6.9p1-8.fc22 has been pushed to the Fedora 22 testing repository. If problems still persist, please make note of it in this bug report.
If you want to test the update, you can install it with
$ su -c 'dnf --enablerepo=updates-testing update openssh'
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2015-686c9d0140
Comment 13 Fedora Update System 2015-10-04 18:49:21 EDT
openssh-6.9p1-8.fc22 has been pushed to the Fedora 22 stable repository. If problems still persist, please make note of it in this bug report.
Comment 14 Jason Tibbitts 2015-10-05 14:47:56 EDT
Just a note that this fix appears to have completely broken ssh for me unless I destroy all of my kerberos principals.  I filed https://bugzilla.redhat.com/show_bug.cgi?id=1268968

Note You need to log in before you can comment on or make changes to this bug.