Bug 1837221

Summary:	[RFE] Allow using other than RSA SHA-1/SHA-2 public keys for SSH connections between RHVM and hypervisors
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Martin Perina <mperina>
Component:	ovirt-engine	Assignee:	Artur Socha <asocha>
Status:	CLOSED ERRATA	QA Contact:	Petr Matyáš <pmatyas>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.3.0	CC:	cshao, dfodor, gdeolive, jjelen, lsvaty, mavital, mtessun, peyu, qiyuan, sbonazzo, sgoodman, shlei, weiwang, yaniwang, yturgema
Target Milestone:	ovirt-4.4.5	Keywords:	FutureFeature, ZStream
Target Release:	4.4.5
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ovirt-engine-4.4.5.5	Doc Type:	Enhancement
Doc Text:	Previously, the Manager was able to connect to hypervisors only using RSA public keys for SSH connection. With this update, the Manager can also use EcDSA and EdDSA public keys for SSH. Previously, RHV used only the fingerprint of an SSH public key to verify the host. Now that RHV can use EcDSA and EdDSA public keys for SSH, the whole public SSH key must be stored in the RHV database. As a result, using the fingerprint of an SSH public key is deprecated. When adding a new host to the Manager, the Manager will always use the strongest public key that the host offers, unless an administrator provides another specific public key to use. For existing hosts, the Manager stores the entire RSA public key in its database on the next SSH connection. For example, if an administrator moves the host to maintenance mode and executes an enroll certificate or reinstalls the host, to use a different public key for the host, the administrator can provide a custom public key using the REST API or by fetching the strongest public key in the Edit host dialog in the Administration Portal.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-04-14 11:39:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Infra	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1931474
Bug Blocks:	1901752, 1930198

Description Martin Perina 2020-05-19 06:23:35 UTC

We are using apache-sshd to execute commands on hypervisors from engine using SSH protocol (for example copy SSH public key to hypervisors, getting VDSM ID, restarting/stopping hosts over SSH, ...).

During connection from engine to host we are checking the fingerprint of the host. But from historical reasons all our code around generating fingerprint from SSH key is built on RSA public keys:

https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/uutils/src/main/java/org/ovirt/engine/core/uutils/ssh/OpenSSHUtils.java#L177
https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/uutils/src/main/java/org/ovirt/engine/core/uutils/ssh/OpenSSHUtils.java#L211

That's why currently we allowing only RSA keys:

https://github.com/oVirt/ovirt-engine/blob/master/backend/manager/modules/uutils/src/main/java/org/ovirt/engine/core/uutils/ssh/SSHClient.java#L74

Of course apache-sshd supports also other public keys:

http://mina.apache.org/sshd-project/apidocs/org/apache/sshd/common/signature/BuiltinSignatures.html

We have found out that due to above limitations we cannot connect to EL8 hosts with FIPS security hardening enabled.

This problem is solvable, but it requires significant code changes and probably even breaking backward compatibility around host SSH fingerprints:

https://github.com/oVirt/ovirt-engine-api-model/blob/master/src/main/java/types/Host.java#L242
https://github.com/oVirt/ovirt-engine-api-model/blob/master/src/main/java/types/Ssh.java

We would need to perform following:

1. Save all existing SSH public keys of the host into database
2. If one of above keys is RSA, generate fingerprint and save it to database
3. During SSH connection don't validate fingerprints, but compare stored SSH keys
4. Extend our RESTAPI to provide not only RSA public key fingerprint, but all SSH public keys we have fetched from the host
5. If the host won't have RSA public key available, fingerprint will be empty -> this is breaking backward compatibility for existing clients which relies on existence of fingerprint

Comment 1 Jakub Jelen 2020-05-19 07:33:53 UTC

Just for the record, there is really nothing wrong with RSA keys as long as they can be used with SHA2 digests per RFC 8332 [1], but it looks like this RFC is not implemented in apache-sshd (from my fast check of few files).

Therefore accepting different key types (generally ecdsa, which is also supported on apache-sshd side) is something we should strive to in short-term. Implementation of the above RFC can be tricky in many ways, but it would ultimate goal to extend compatibility.

[1] https://tools.ietf.org/html/rfc8332

Comment 2 Martin Perina 2020-05-19 08:01:01 UTC

(In reply to Jakub Jelen from comment #1)
> Just for the record, there is really nothing wrong with RSA keys as long as
> they can be used with SHA2 digests per RFC 8332 [1], but it looks like this
> RFC is not implemented in apache-sshd (from my fast check of few files).
> 

Hmm, right now we are using version 2.2.0, but if I take a look at 2.3.0 release notes, I can see also support for RSA-SHA256 and RSA-SHA512:

https://gitbox.apache.org/repos/asf?p=mina-sshd.git;a=blob;f=docs/changes/2.3.0.md;h=4c3fa4bb62c78c446bc7320c231f21c8ec66c96b;hb=HEAD#l109

So maybe it's also worth trying to upgrade first

> Therefore accepting different key types (generally ecdsa, which is also
> supported on apache-sshd side) is something we should strive to in
> short-term. Implementation of the above RFC can be tricky in many ways, but
> it would ultimate goal to extend compatibility.
> 
> [1] https://tools.ietf.org/html/rfc8332

Comment 3 Jakub Jelen 2020-05-19 08:28:40 UTC

Thanks for sharing the link. I did not find the changelog myself. The official project page is very brief [1].

It indeed looks like they implemented this RFC so in that case, updating to current version (and maybe adjusting your code to support also the rsa-sha2 signatures) would be better as a short-term fix for he previously mentioned issue. Adding support for different key types (ecdsa) would be certainly good for a long-term but would not be a must

[1] https://mina.apache.org/sshd-project/features.html

Comment 4 Martin Perina 2020-05-20 14:46:54 UTC

I've created BZ1838159 to track upgrade to apache-sshd 2.4.0 to support RSA-SHA256 and and RSA-SHA512 public keys, so moving current bug to 4.5

Comment 5 Sandro Bonazzola 2020-05-26 07:47:55 UTC

Martin shouldn't this be re-targeted to 4.4 too?

Comment 6 Martin Perina 2020-05-26 09:06:26 UTC

(In reply to Sandro Bonazzola from comment #5)
> Martin shouldn't this be re-targeted to 4.4 too?

If BZ1838159 will fix the issue, then I'd rather leave this for 4.5 or some 4.4.z

Comment 8 Petr Matyáš 2021-02-17 11:40:19 UTC

Verified on ovirt-engine-4.4.5.5-0.13.el8ev.noarch

Comment 15 Yedidyah Bar David 2021-04-14 09:49:58 UTC

*** Bug 1619649 has been marked as a duplicate of this bug. ***

Comment 16 errata-xmlrpc 2021-04-14 11:39:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) 4.4.z [ovirt-4.4.5] security, bug fix, enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1169