Bug 1901752 - AddVds fails as FIPS host rejects SSH with ssh-rsa, failing HostedEngine deployment
Summary: AddVds fails as FIPS host rejects SSH with ssh-rsa, failing HostedEngine depl...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.4.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ovirt-4.4.5
: 4.4.5
Assignee: Artur Socha
QA Contact: Pavol Brilla
URL:
Whiteboard:
Depends On: 1837221
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-26 00:12 UTC by Germano Veit Michel
Modified: 2021-05-24 21:48 UTC (History)
8 users (show)

Fixed In Version: ovirt-engine-4.4.5.5
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-04-14 11:39:56 UTC
oVirt Team: Infra
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 5604651 0 None None None 2020-12-29 21:10:53 UTC
Red Hat Product Errata RHSA-2021:1169 0 None None None 2021-04-14 11:40:29 UTC

Description Germano Veit Michel 2020-11-26 00:12:43 UTC
Description of problem:

When deploying HE, at the step when the deploy host is added to HostedEngineLocal:

2020-11-25 23:31:17,835Z INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [fc2c1be] EVENT_ID: USER_VDC_LOGIN(30), User admin@internal-authz connecting from '192.168.222.1' using session 'WmBsFpMQUjTmkgbgF3CM8zN+5KNNQK5mghf/Gpyp0nfKeMcx7PHpmTW6AwjDln9E0o8gaL3nKJ7rz+4mDHaB+A==' logged in.
2020-11-25 23:31:54,563Z ERROR [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1) [6af37be3-8650-429c-baa9-7133945693f8] Failed to authenticate session with host 'host2.kvm': SSH authentication to 'root' failed. Please verify provided credentials. Make sure key is authorized at host
2020-11-25 23:31:54,564Z WARN  [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1) [6af37be3-8650-429c-baa9-7133945693f8] Validation of action 'AddVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server host2.kvm,VDS_CANNOT_AUTHENTICATE_TO_SERVER
2020-11-25 23:31:54,677Z ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot add Host. SSH authentication failed, verify authentication parameters are correct (Username/Password, public-key etc.) You may refer to the engine.log file for further details.]

Then the HE deploy fails waiting for the "Host to be up", but it wasn't even added.

On the host we have signs that the SSH failed too:
Nov 25 23:31:54 host2 sshd[10520]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Nov 25 23:31:54 host2 sshd[10520]: Connection closed by authenticating user root 192.168.222.28 port 60740 [preauth]

If I SSH to HostedEngineLocal and try to SSH to the host in a similar way to what the SSHClient Java code in the engine does, I get this:
# ssh -i /etc/pki/ovirt-engine/keys/engine_id_rsa -oHostKeyAlgorithms=ssh-rsa root
Unable to negotiate with 192.168.100.2 port 22: no matching host key type found. Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256

And the exact same error on the host:
Nov 25 23:46:35 host2 sshd[12412]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Nov 25 23:46:35 host2 sshd[12412]: Connection closed by authenticating user root 192.168.222.28 port 60950 [preauth]

# cat /etc/crypto-policies/back-ends/openssh.config
Ciphers aes256-gcm,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc
MACs hmac-sha2-256,hmac-sha1,hmac-sha2-512
GSSAPIKeyExchange no
KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256
PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512
CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512

Version-Release number of selected component (if applicable):
RHEL 8.3
rhvm-appliance-4.4-20201117.0.el8ev.x86_64
redhat-release-virtualization-host-4.4.3-1.el8ev.x86_64
crypto-policies-20200713-1.git51d1222.el8.noarch

How reproducible:
Always

Steps to Reproduce:
1. Deploy fresh RHVH 4.4.3 with DISA STIG
2. Deploy Hosted-Engine (see Additional info to workaround another BZ, otherwise it will fail earlier)

Actual results:
* HE deploy fails
* Host add fails

Expected results:
* Host add succeeds

Additional info:
* To get to this point you need to workaround BZ1894852 by doing this on the Host before the step "Wait for SSH to restart on the local VM"
  $ chmod 644 /etc/crypto-policies/back-ends/openssh.config

Comment 2 Martin Perina 2021-02-08 10:07:02 UTC
(In reply to Germano Veit Michel from comment #0)
> Description of problem:
> 
> When deploying HE, at the step when the deploy host is added to
> HostedEngineLocal:
> 
> 2020-11-25 23:31:17,835Z INFO 
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (default task-1) [fc2c1be] EVENT_ID: USER_VDC_LOGIN(30), User
> admin@internal-authz connecting from '192.168.222.1' using session
> 'WmBsFpMQUjTmkgbgF3CM8zN+5KNNQK5mghf/
> Gpyp0nfKeMcx7PHpmTW6AwjDln9E0o8gaL3nKJ7rz+4mDHaB+A==' logged in.
> 2020-11-25 23:31:54,563Z ERROR
> [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1)
> [6af37be3-8650-429c-baa9-7133945693f8] Failed to authenticate session with
> host 'host2.kvm': SSH authentication to 'root' failed. Please
> verify provided credentials. Make sure key is authorized at host
> 2020-11-25 23:31:54,564Z WARN 
> [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1)
> [6af37be3-8650-429c-baa9-7133945693f8] Validation of action 'AddVds' failed
> for user admin@internal-authz. Reasons:
> VAR__ACTION__ADD,VAR__TYPE__HOST,$server
> host2.kvm,VDS_CANNOT_AUTHENTICATE_TO_SERVER
> 2020-11-25 23:31:54,677Z ERROR
> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default
> task-1) [] Operation Failed: [Cannot add Host. SSH authentication failed,
> verify authentication parameters are correct (Username/Password, public-key
> etc.) You may refer to the engine.log file for further details.]
> 
> Then the HE deploy fails waiting for the "Host to be up", but it wasn't even
> added.
> 
> On the host we have signs that the SSH failed too:
> Nov 25 23:31:54 host2 sshd[10520]: userauth_pubkey: key type ssh-rsa not in
> PubkeyAcceptedKeyTypes [preauth]
> Nov 25 23:31:54 host2 sshd[10520]: Connection closed by authenticating user
> root 192.168.222.28 port 60740 [preauth]
> 
> If I SSH to HostedEngineLocal and try to SSH to the host in a similar way to
> what the SSHClient Java code in the engine does, I get this:
> # ssh -i /etc/pki/ovirt-engine/keys/engine_id_rsa
> -oHostKeyAlgorithms=ssh-rsa root
> Unable to negotiate with 192.168.100.2 port 22: no matching host key type
> found. Their offer: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256
> 
> And the exact same error on the host:
> Nov 25 23:46:35 host2 sshd[12412]: userauth_pubkey: key type ssh-rsa not in
> PubkeyAcceptedKeyTypes [preauth]
> Nov 25 23:46:35 host2 sshd[12412]: Connection closed by authenticating user
> root 192.168.222.28 port 60950 [preauth]
> 
> # cat /etc/crypto-policies/back-ends/openssh.config
> Ciphers
> aes256-gcm,aes256-ctr,aes256-cbc,aes128-gcm,aes128-
> ctr,aes128-cbc
> MACs hmac-sha2-256,hmac-sha1,hmac-sha2-512
> GSSAPIKeyExchange no
> KexAlgorithms
> ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-
> group14-sha256
> PubkeyAcceptedKeyTypes
> ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-
> sha2-512
> CASignatureAlgorithms
> ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-
> sha2-512

That's very strange, rsa-sha2 support is enabled from RHV 4.4 GA (BZ1838159) and I've just verified that latest master works fine with host providing rsa-sha2 keys. Anyway as a part of RHV 4.4.5 we are adding support also for EcDSA and EdDSA keys (BZ1837221), so I'm suggesting to retest with newer version.

Comment 3 Martin Perina 2021-02-09 14:05:22 UTC
Syncing status with BZ1837221

Comment 6 Wei Wang 2021-02-20 04:27:08 UTC
1. Test with latest RHVH build RHVH-4.4-20210216.0-RHVH-x86_64-dvd1.iso
Test version:
RHVH-4.4-20210216.0-RHVH-x86_64-dvd1.iso
rhvm-appliance-4.4-20201117.0.el8ev.x86_64
crypto-policies-20200713-1.git51d1222.el8.noarch

Test steps:
1. Clean install RHVH with STIG security profile
2. Yum installrhvm-appliance
3. Deploy hosted engine via cockpit UI

Result:
HE deploy fails, it can be reproduced with latest version. And the issue has been traced by https://bugzilla.redhat.com/show_bug.cgi?id=1909956

2. Test with RHEL-Host regular host
Test version:
RHEL-8.3.0-20201009.2-x86_64-dvd1.iso
crypto-policies-20200713-1.git51d1222.el8.noarch
cockpit-ovirt-dashboard-0.14.19-1.el8ev.noarch
ovirt-hosted-engine-setup-2.4.9-4.el8ev.noarch
ovirt-hosted-engine-ha-2.4.6-1.el8ev.noarch

Test steps:
1. Clean install RHEL8.3 with STIG security profile
2. Subscribe and enable to the right channels
3. Yum install cockpit-ovirt-dashboard
4. Yum install rhvm-appliance from CDN(rhv-4-mgmt-agent-for-rhel-8-x86_64-rpms repository)
4. Deploy hosted engine via cockpit UI

Result:
HE deploy fails, it can be reproduced with latest version.
2021-02-20 10:50:34,227+08 ERROR [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1) [f653ccfb-f128-4314-ae82-fdc919222698] Failed to authenticate session with host 'hp-dl388g9-05.lab.eng.pek2.redhat.com': SSH authentication to 'root.eng.pek2.redhat.com' failed. Please verify provided credentials. Make sure key is authorized at host
2021-02-20 10:50:34,228+08 WARN  [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1) [f653ccfb-f128-4314-ae82-fdc919222698] Validation of action 'AddVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server hp-dl388g9-05.lab.eng.pek2.redhat.com,VDS_CANNOT_AUTHENTICATE_TO_SERVER
2021-02-20 10:50:34,262+08 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot add Host. SSH authentication failed, verify authentication parameters are correct (Username/Password, public-key etc.) You may refer to the engine.log file for further details.] 

Feb 20 10:50:34 hp-dl388g9-05.lab.eng.pek2.redhat.com sshd[25336]: error: Unable to load host key: /etc/ssh/ssh_host_ed25519_key
Feb 20 10:50:34 hp-dl388g9-05.lab.eng.pek2.redhat.com sshd[25336]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Feb 20 10:50:34 hp-dl388g9-05.lab.eng.pek2.redhat.com sshd[25336]: Connection closed by authenticating user root 192.168.222.196 port 40638 [preauth]

[root@hp-dl388g9-05 ~]# cat /etc/crypto-policies/back-ends/openssh.config
Ciphers aes256-gcm,aes256-ctr,aes256-cbc,aes128-gcm,aes128-ctr,aes128-cbc
MACs hmac-sha2-256,hmac-sha1,hmac-sha2-512
GSSAPIKeyExchange no
KexAlgorithms ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256
PubkeyAcceptedKeyTypes ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512
CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-256,rsa-sha2-512 

More info:
# rpm -qa|grep appliance
rhvm-appliance-4.4-20201117.0.el8ev.x86_64
# ssh root.222.196
[root@rhevh-hostedengine-vm-06 ~]# rpm -qa|grep ovirt-engine
ovirt-engine-4.4.3.12-0.1.el8ev.noarch
[root@rhevh-hostedengine-vm-06 ~]# rpm -qa|grep apache-sshd
apache-sshd-2.5.1-1.el8ev.noarch

So ovirt-engine is not the latest one, QE will verfiy this bug until updating rhvm-appliace with ovirt-engine-4.4.5.5

Comment 7 Martin Perina 2021-02-20 07:07:32 UTC
As mentioned in BZ1837221 you need to test in on ovirt-engine-4.4.5.5 or later. It's not required to test this in hosted, it can easily be tested on standalone engine by adding FIPS enabled host.

If you really need to test it in hosted engine, then please update your hosted engine appliance to latest version during installation, AFAIK appliance can be updated to latest version even before adding the 1st host to engine. Am I right Asaf?

Comment 8 Asaf Rachmani 2021-02-21 10:03:56 UTC
(In reply to Martin Perina from comment #7)
> As mentioned in BZ1837221 you need to test in on ovirt-engine-4.4.5.5 or
> later. It's not required to test this in hosted, it can easily be tested on
> standalone engine by adding FIPS enabled host.
> 
> If you really need to test it in hosted engine, then please update your
> hosted engine appliance to latest version during installation, AFAIK
> appliance can be updated to latest version even before adding the 1st host
> to engine. Am I right Asaf?

Yes, there are a few options described in [1]. 

[1] https://github.com/oVirt/ovirt-ansible-collection/tree/master/roles/hosted_engine_setup#make-changes-in-the-engine-vm-during-the-deployment

Comment 9 Wei Wang 2021-02-22 04:36:25 UTC
According to comment 7, QE test on standalone engine(ovirt-engine-4.4.5.5-0.13.el8ev.noarch) by adding FIPS enabled host.

The FIPS host can be added successfully, the bug is fixed, move it to "VERIFIED"

Comment 15 errata-xmlrpc 2021-04-14 11:39:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) 4.4.z [ovirt-4.4.5] security, bug fix, enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1169

Comment 16 Marina Kalinin 2021-05-24 21:29:10 UTC
Why there is no release notes for this bug?

Comment 17 Martin Perina 2021-05-24 21:48:31 UTC
(In reply to Marina Kalinin from comment #16)
> Why there is no release notes for this bug?

There is no need to add anything specific for this bug, all important informations are included in dependent BZ1837221


Note You need to log in before you can comment on or make changes to this bug.