Created attachment 1741293 [details] issue log files Description of problem: Hosted engine setup with STIG security profile, QE detect below issue: [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Host is not up, please check logs, perhaps also on the engine machine"} .... [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure] [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} ----------- engine.log ----------- 2020-12-18 11:52:07,873+08 ERROR [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1) [bf22a318-16bc-43c8-81bc-52584955a06b] Failed to authenticate session with host 'hp-dl388g9-04.lab.eng.pek2.redhat.com': SSH authentication to 'root.eng.pek2.redhat.com' failed. Please verify provided credentials. Make sure key is authorized at host 2020-12-18 11:52:07,874+08 WARN [org.ovirt.engine.core.bll.hostdeploy.AddVdsCommand] (default task-1) [bf22a318-16bc-43c8-81bc-52584955a06b] Validation of action 'AddVds' failed for user admin@internal-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__HOST,$server hp-dl388g9-04.lab.eng.pek2.redhat.com,VDS_CANNOT_AUTHENTICATE_TO_SERVER 2020-12-18 11:52:07,914+08 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [Cannot add Host. SSH authentication failed, verify authentication parameters are correct (Username/Password, public-key etc.) You may refer to the engine.log file for further details.] Version-Release number of selected component (if applicable): RHVH-4.4-20201217.0-RHVH-x86_64-dvd1.iso cockpit-ws-224.2-1.el8.x86_64 cockpit-224.2-1.el8.x86_64 cockpit-ovirt-dashboard-0.14.17-1.el8ev.noarch cockpit-bridge-224.2-1.el8.x86_64 cockpit-system-224.2-1.el8.noarch cockpit-storaged-224.2-1.el8.noarch subscription-manager-cockpit-1.27.16-1.el8.noarch cockpit-dashboard-224.2-1.el8.noarch ovirt-hosted-engine-setup-2.4.9-2.el8ev.noarch ovirt-hosted-engine-ha-2.4.5-1.el8ev.noarch rhvm-appliance-4.4-20201117.0.el8ev.x86_64 How reproducible: 100% Steps to Reproduce: 1. Select STIG profile during RHVH installation 2. Deploy hosted engine via cockpit Actual results: Failed to authenticate ssh session with host during hosted engine deployment Expected results: Hosted engine should be deployed successfully. Additional info:
Issue is similar but not the same as in bug #1902646
Deployment fails because of "Add host" task when using public key [1]. From the host: # journalctl -u sshd ... Dec 27 15:18:28 rhvh444.asrachmani.com sshd[6262]: rexec line 144: Deprecated option UsePrivilegeSeparation Dec 27 15:18:28 rhvh444.asrachmani.com sshd[6262]: error: Unable to load host key: /etc/ssh/ssh_host_ed25519_key Dec 27 15:18:29 rhvh444.asrachmani.com sshd[6262]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth] ... Once 'ssh-rsa,ssh-rsa-cert-v01' added to /etc/crypto-policies/back-ends/opensshserver.config file, the deployment succeeds. Seems like an issue with the engine public key, perhaps we need to use different SSH key types [2]. [1] https://github.com/oVirt/ovirt-ansible-collection/blob/fc77289aa316494b9e1f791d95f20900298a25b4/roles/hosted_engine_setup/tasks/bootstrap_local_vm/05_add_host.yml#L121 [2] https://access.redhat.com/solutions/4906221
ssh-rsa is not supported on FIPS enabled host, but FIPS should still offer ssh-rsa-256 and ssh-rsa-512, which we have added in 4.4.1 BZ1838159
Moving to infra for further investigation, no idea why ssh-rsa-2 public key is not negotiated. BZ1837221 might help, but we should still investigate, why ssh-rsa-2 doesn't work
Created attachment 1748766 [details] host_sshd_output
Created attachment 1748767 [details] engine_ssh_keyscan
(In reply to Asaf Rachmani from comment #2) > Deployment fails because of "Add host" task when using public key [1]. > > From the host: > # journalctl -u sshd > ... > Dec 27 15:18:28 rhvh444.asrachmani.com sshd[6262]: rexec line 144: > Deprecated option UsePrivilegeSeparation > Dec 27 15:18:28 rhvh444.asrachmani.com sshd[6262]: error: Unable to load > host key: /etc/ssh/ssh_host_ed25519_key > Dec 27 15:18:29 rhvh444.asrachmani.com sshd[6262]: userauth_pubkey: key type > ssh-rsa not in PubkeyAcceptedKeyTypes [preauth] > ... > > Once 'ssh-rsa,ssh-rsa-cert-v01' added to > /etc/crypto-policies/back-ends/opensshserver.config file, the deployment > succeeds. Hmm, it might be related to https://github.com/apache/mina-sshd/pull/158 We will try to upgrade to apache-sshd-2.6.0
(In reply to Martin Perina from comment #7) > (In reply to Asaf Rachmani from comment #2) > > Deployment fails because of "Add host" task when using public key [1]. > > > > From the host: > > # journalctl -u sshd > > ... > > Dec 27 15:18:28 rhvh444.asrachmani.com sshd[6262]: rexec line 144: > > Deprecated option UsePrivilegeSeparation > > Dec 27 15:18:28 rhvh444.asrachmani.com sshd[6262]: error: Unable to load > > host key: /etc/ssh/ssh_host_ed25519_key > > Dec 27 15:18:29 rhvh444.asrachmani.com sshd[6262]: userauth_pubkey: key type > > ssh-rsa not in PubkeyAcceptedKeyTypes [preauth] > > ... > > > > Once 'ssh-rsa,ssh-rsa-cert-v01' added to > > /etc/crypto-policies/back-ends/opensshserver.config file, the deployment > > succeeds. > > Hmm, it might be related to https://github.com/apache/mina-sshd/pull/158 > We will try to upgrade to apache-sshd-2.6.0 Sorry, wrong link, it should have been https://issues.apache.org/jira/browse/SSHD-1053
Syncing status with BZ1919555
QE will verify this bug when the updated rhvm-appliance include ovirt-engine-4.4.5.5 is coming.
Test with rhvm-appliance-4.4-20210309.0 and rhvm-appliance-4.4-20210309.0, bug is not fixed. QE will verify this bug when the new rhvm-appliance include ovirt-engine-4.4.5.5 is coming.
Test with RHVH-4.4-20210331.0-RHVH-x86_64-dvd1.iso and rhvm-appliance-4.4-20210401.0.el8ev.x86_64 Bug is fixed, move it to "VERIFIED"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: RHV RHEL Host (ovirt-host) 4.4.z [ovirt-4.4.5] security, bug fix, enhancement), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1184