RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1450609 - RHEL6 openssl can't talk to Fedora git server anymore?
Summary: RHEL6 openssl can't talk to Fedora git server anymore?
Keywords:
Status: CLOSED CANTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: openssh
Version: 6.9
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jakub Jelen
QA Contact: BaseOS QE Security Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-05-13 22:06 UTC by Tom Lane
Modified: 2017-05-17 11:58 UTC (History)
0 users

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-05-17 11:58:48 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
stderr of ssh -vvv tgl@pkgs.fedoraproject.org test (6.06 KB, text/plain)
2017-05-15 14:14 UTC, Tom Lane
no flags Details

Description Tom Lane 2017-05-13 22:06:42 UTC
Description of problem:
When I try to do something with Fedora package sources on my RHEL6 machine, I get

$ git pull
key_from_blob: remaining bytes in key blob 36
cert_parse: Invalid signature key type unknown (11)
key_from_blob: can't parse cert data
cannot decode server_host_key_blob
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This used to work, and it still does work on my Fedora 25 laptop with the same
keys and configuration.  I surmise that Fedora has updated their server key or something like that so that it no longer works with RHEL6's version of openssl.
Is that correct, and if so is there any chance of modernizing RHEL6's package?

(It's certainly possible that I'm misassigning blame here, but out-of-date openssl
seems like the best theory at the moment.)

Version-Release number of selected component (if applicable):
openssl-1.0.1e-57.el6.x86_64

How reproducible:
100% on this machine

Steps to Reproduce:
1. Install usual Fedora packager infrastructure on a RHEL6 machine.
2. "fedpkg co", or anything else that tries to access the Fedora git repo.

Actual results:
fail

Expected results:
work

Additional info:
I last tried to do something with the Fedora repos on this machine on March 10,
and it worked then.  I have not changed any configuration since then.  I tried regenerating my Fedora certificate, and that succeeded, but it didn't help.

Comment 2 Tomas Mraz 2017-05-15 09:59:17 UTC
This is probably not openssl but openssh related.

Comment 3 Jakub Jelen 2017-05-15 10:55:44 UTC
Just tried the same with up-to-date RHEL6.9 and it looks like it works for me. Can you provide your debug logs from the following command:

$ ssh -vvv fedora_username.org test

What is the version of openssh?

$ rpm -q openssh

Comment 4 Tom Lane 2017-05-15 14:14:57 UTC
Created attachment 1279007 [details]
stderr of ssh -vvv tgl.org test

Comment 5 Tom Lane 2017-05-15 14:17:02 UTC
(In reply to Jakub Jelen from comment #3)
> Just tried the same with up-to-date RHEL6.9 and it looks like it works for
> me. Can you provide your debug logs from the following command:
> $ ssh -vvv fedora_username.org test

Attached.  Looking briefly at the log, it seems odd that it's complaining about
my key file; "openssl rsa" recognizes the file as a passphrase-protected RSA key
without any problem.  But maybe that's unrelated.

> What is the version of openssh?

openssh-5.3p1-122.el6.x86_64

Comment 6 Jakub Jelen 2017-05-15 15:03:32 UTC
Thank you for the information.

> debug2: key_type_from_name: unknown key type 'ssh-ed25519'

It complains about the ed25519 host key (this type is not known to RHEL6 yet) that is being send by the remote server from pkgs.fedoraproject.org (RHEL 7 equivalent) as part of the SSH_MSG_KEX_DH_GEX_REPLY. IMHO, that key type signature should not be send in the first place, since it is not negotiated in the client PK_ALG proposal:

> debug2: kex_parse_kexinit: ssh-rsa-cert-v01,ssh-dss-cert-v01,ssh-rsa-cert-v00,ssh-dss-cert-v00,ssh-rsa,ssh-dss

but even server does not offer this PK_ALG in its proposal:

> debug2: kex_parse_kexinit: ssh-rsa,ssh-rsa-cert-v01


I am able to reproduce the behavior against this server now I am able to workaround the problem by telling the client that I really want RSA host keys (so at least it does not render rhel6 completely unusable):

ssh -o HostKeyAlgorithms=ssh-rsa -vvv jjelen.org test

This can be stored in the ~/.ssh/config as a workaround:

    Host pkgs.fedoraproject.org
      HostKeyAlgorithms ssh-rsa,ssh-rsa-cert-v01


But still I can not reproduce the same against my RHEL7 machine. 

From my point of view, this is some bug in the server (either implementation or configuration), but I will probably need more information how is it configured. I will have a look into that.

Comment 7 Tom Lane 2017-05-15 15:55:21 UTC
Thanks, I confirm that the HostKeyAlgorithms workaround works for me.
But it does seem like the server is doing something wrong.

Comment 8 Jakub Jelen 2017-05-16 08:50:57 UTC
Respective log from RHEL7 machine looks like this:

  debug1: ssh_ed25519_verify: signature correct
  debug1: Server host key: RSA-CERT fe:2e:6a:86:f3:41:e7:03:95:ea:9c:7f:75:9c:ce:9d
  debug3: load_hostkeys: loading entries for host "pkgs.fedoraproject.org" from file "/root/.ssh/known_hosts"

There is obviously some ed25519 in action, even though it is RSA-CERT. It is the certificate authority using ED25519, which was used to sign the RSA host key. This confuses the old clients and well ... there is no switch to negotiate certificate authorities key types (having that would be very over-engineered).

What can be done?

 * OpenSSH should not send certificates signed by the keys that are not supported by the peers (fix for RHEL7+)

but I am not sure how upstream will be willing to accept this change: interoperability between 5.3 (with backported certificates) and 7.5. Lets see.

 * There is no reasonable thing we can do about that in RHEL6 client. The server sends the host key based on key types that were negotiated (certificates have priority over keys). Ignoring the certificate is probably not what is expected.

 * Fedora release-engineering should be notified about this issue, but I don't expect they would re-issue the CA key to support RHEL6 clients.

Comment 9 Jakub Jelen 2017-05-16 11:20:27 UTC
>  * OpenSSH should not send certificates signed by the keys that are not
> supported by the peers (fix for RHEL7+)

This is not possible -- the key exchange expect that both client and server select the same PK_ALG and therefore server can not send raw RSA key if RSA_CERT is first in the supported list.

Comment 10 Jakub Jelen 2017-05-17 10:42:18 UTC
Upstream comment to this issue:

http://lists.mindrot.org/pipermail/openssh-unix-dev/2017-May/036032.html

Well, this is a problem, but we don't have a sane way to fix it in RHEL6 nor RHEL7. This is an addition to the protocol, which is not known to the old clients. The solution will be probably not to use this type of keys in infrastructure where RHEL6 should be supported.

Comment 11 Jakub Jelen 2017-05-17 11:58:48 UTC
Filled a releng ticked:
https://pagure.io/releng/issue/6798

Since there is no reasonable way to resolve this in RHEL6 nor upstream, I will close this bug and keep it tracked in Fedora releng Issue. Lets see what they can do about that.


Note You need to log in before you can comment on or make changes to this bug.