Bug 1845271 - Better documentation for host ssh requirements
Summary: Better documentation for host ssh requirements
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Documentation
Version: 2.4.4
Hardware: Unspecified
OS: Unspecified
medium
medium vote
Target Milestone: ovirt-4.4.9
: ---
Assignee: Steve Goodman
QA Contact: Richard Hoch
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-08 19:26 UTC by Wart
Modified: 2021-10-05 09:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-05 09:42:40 UTC
oVirt Team: Integration


Attachments (Terms of Use)

Description Wart 2020-06-08 19:26:49 UTC
Description of problem:  The self-hosted engine setup requires ssh pubkey access from the engine VM into the root account of the bare metal host/node.  If the user sets 'PermitRootLogin no' in sshd_config of the host/node, then the engine deployment will hang (and eventually time out) at 'Waiting for the host to be up'.  There are no clear error messages in the logs about the possible cause of the problem.


Version-Release number of selected component (if applicable):
# rpm -qf /usr/sbin/hosted-engine
ovirt-hosted-engine-setup-2.4.4-1.el8.noarch


How reproducible:
Always


Steps to Reproduce:
1. Per the ovirt 4.4 installation instructions, install a minimal CentOS 8 host, or an oVirt node.
2. Set 'PermitRootLogin no' in /etc/ssh/sshd_config and restart sshd
3. Continue the ovirt 4.4 installation and run 'hosted-engine --deploy'

Actual results:
Installation hangs, and eventually times and fails directing the user to check the ansible logs:
[ INFO ] TASK [ovirt.hosted_engine_setup : Obtain SSO token using username/pass
word credentials]
[ INFO ] ok: [localhost]
[ INFO ] TASK [ovirt.hosted_engine_setup : Wait for the host to be up]
[ ERROR ] fatal: [localhost]: FAILED! => {"attempts": 120,
"changed": false, "ov
irt_hosts": []}
[...cleanup...]
[ INFO ] TASK [ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
"The system may not be provisioned according to the playbook results: please check
the logs for the issue, fix accordingly or re-deploy from scratch.\n"}

Expected results:
Installation fails with a message (either on stdout or in a log file) that says there is a ssh problem between the newly deployed engine and the bare metal host/node.

Additional info:
In addition to clarifying the error message/logs in the hosted-engine application, it would be nice to have the online documentation describe the sshd config requirements for the host/node.  Something along the lines of:

"The oVirt engine requires root ssh access into the hypervisor host/node.  ssh is done using pubkey authentication.  At a minimum you will need to allow ssh pubkey auth into the root account of the hypervisor host/node with the following ssh configuration:

Match Address engine.ip.address
    PermitRootLogin without-password
    PubkeyAuthentication yes

Setting a less restrictive 'PermitRootLogin yes' at the global level will also work.
"

Comment 1 Yedidyah Bar David 2020-06-09 08:20:52 UTC
Following another post, by Paul-Erik Törrönen, on the thread leading to creation of this bug:

We should also mention that sshd must allow using RSA keypairs for authentication. You can't use e.g. ed25519.

Comment 2 Steve Goodman 2020-11-26 09:28:50 UTC
Comment 0 describes three different bugs:
1. There are no clear error messages in the logs about the possible cause of the problem.
2. The deployment script should state that ssh public key access to root is required.
3. The documentation should state pubkey access as a prerequisite.

This current bug only addresses documentation.

I can very easily add a prerequisite to the following topics:
- Deploying the Self-hosted Engine using Cockpit [1]
- Deploying the Self-Hosted Engine Using the Command Line [2]

How's this for the text of the prerequisite:

----
The self-hosted engine setup script requires ssh public key access from the engine virtual machine to the root account of its bare metal host. In `/etc/ssh/sshd_config`, ensure that these values are set as follows:
 * `PermitRoolLogin` is set to `without-password` or `yes`
 * `PubkeyAuthentication` is set to `yes`
 * `HostKey` is set to `/etc/ssh/ssh_host_rsa_key`
----

[1] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_cockpit_web_interface/index#Deploying_the_Self-Hosted_Engine_Using_Cockpit_install_RHVM
[2] https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_command_line/index#Deploying_the_Self-Hosted_Engine_Using_the_CLI_install_RHVM

Comment 3 RHEL Program Management 2020-11-26 09:29:01 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 4 Yedidyah Bar David 2020-11-26 10:03:13 UTC
(In reply to Steve Goodman from comment #2)
> Comment 0 describes three different bugs:
> 1. There are no clear error messages in the logs about the possible cause of
> the problem.

That's indeed a separate issue, perhaps worth its own bug. I hope to have at least partially improved this in bug 1893385, let's see.

> 2. The deployment script should state that ssh public key access to root is
> required.

Perhaps this makes sense as well, but let's assume that it's a doc issue and is part of current bug.

> 3. The documentation should state pubkey access as a prerequisite.
> 
> This current bug only addresses documentation.
> 
> I can very easily add a prerequisite to the following topics:
> - Deploying the Self-hosted Engine using Cockpit [1]
> - Deploying the Self-Hosted Engine Using the Command Line [2]
> 
> How's this for the text of the prerequisite:
> 
> ----
> The self-hosted engine setup script requires ssh public key access from the
> engine virtual machine to the root account of its bare metal host. In
> `/etc/ssh/sshd_config`, ensure that these values are set as follows:
>  * `PermitRoolLogin` is set to `without-password` or `yes`
>  * `PubkeyAuthentication` is set to `yes`

+1

>  * `HostKey` is set to `/etc/ssh/ssh_host_rsa_key`

Not sure this is the point raised in comment 1 (and [1], originally).

He didn't provide details, and it's a bit hard to guess. I think he changed PubkeyAcceptedKeyTypes, perhaps via crypto-policies (which is also mentioned in sshd_config manpage for this option).

I'd say something more general, such as:

"... requires ssh public key access using 2048 bit RSA keys", and then mention crypto-policies, e.g.

* PubkeyAcceptedKeyTypes should allow 2048 bit RSA keys. It is controlled by default using system-wide crypto policies. See also the manual page crypto-policies(7).

[1] https://lists.ovirt.org/archives/list/users@ovirt.org/message/PYY7KT3UYCLDPPPBEK2ZDFEY3AEKYJW5/

> ----
> 
> [1]
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/
> html-single/installing_red_hat_virtualization_as_a_self-
> hosted_engine_using_the_cockpit_web_interface/index#Deploying_the_Self-
> Hosted_Engine_Using_Cockpit_install_RHVM
> [2]
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/
> html-single/installing_red_hat_virtualization_as_a_self-
> hosted_engine_using_the_command_line/index#Deploying_the_Self-
> Hosted_Engine_Using_the_CLI_install_RHVM

Comment 5 Steve Goodman 2020-12-01 16:14:57 UTC
> >  * `HostKey` is set to `/etc/ssh/ssh_host_rsa_key`
> 
> Not sure this is the point raised in comment 1 (and [1], originally).
> 
> He didn't provide details, and it's a bit hard to guess. I think he changed
> PubkeyAcceptedKeyTypes, perhaps via crypto-policies (which is also mentioned
> in sshd_config manpage for this option).
> 
> I'd say something more general, such as:
> 
> "... requires ssh public key access using 2048 bit RSA keys", and then
> mention crypto-policies, e.g.
> 
> * PubkeyAcceptedKeyTypes should allow 2048 bit RSA keys. It is controlled by
> default using system-wide crypto policies. See also the manual page
> crypto-policies(7).
> 
> [1]
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/
> PYY7KT3UYCLDPPPBEK2ZDFEY3AEKYJW5/
> 
> > ----
> > 
> > [1]
> > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/
> > html-single/installing_red_hat_virtualization_as_a_self-
> > hosted_engine_using_the_cockpit_web_interface/index#Deploying_the_Self-
> > Hosted_Engine_Using_Cockpit_install_RHVM
> > [2]
> > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/
> > html-single/installing_red_hat_virtualization_as_a_self-
> > hosted_engine_using_the_command_line/index#Deploying_the_Self-
> > Hosted_Engine_Using_the_CLI_install_RHVM

Wart, can you comment?

Comment 6 Wart 2020-12-01 16:44:54 UTC
(In reply to Yedidyah Bar David from comment #4)
> (In reply to Steve Goodman from comment #2)
> > Comment 0 describes three different bugs:
> > 1. There are no clear error messages in the logs about the possible cause of
> > the problem.
> 
> That's indeed a separate issue, perhaps worth its own bug. I hope to have at
> least partially improved this in bug 1893385, let's see.
> 
> > 2. The deployment script should state that ssh public key access to root is
> > required.
> 
> Perhaps this makes sense as well, but let's assume that it's a doc issue and
> is part of current bug.
> 
> > 3. The documentation should state pubkey access as a prerequisite.
> > 
> > This current bug only addresses documentation.
> > 
> > I can very easily add a prerequisite to the following topics:
> > - Deploying the Self-hosted Engine using Cockpit [1]
> > - Deploying the Self-Hosted Engine Using the Command Line [2]
> > 
> > How's this for the text of the prerequisite:
> > 
> > ----
> > The self-hosted engine setup script requires ssh public key access from the
> > engine virtual machine to the root account of its bare metal host. In
> > `/etc/ssh/sshd_config`, ensure that these values are set as follows:
> >  * `PermitRoolLogin` is set to `without-password` or `yes`
> >  * `PubkeyAuthentication` is set to `yes`
> 
> +1

+1 from me as well.

> >  * `HostKey` is set to `/etc/ssh/ssh_host_rsa_key`
> 
> Not sure this is the point raised in comment 1 (and [1], originally).
> 
> He didn't provide details, and it's a bit hard to guess. I think he changed
> PubkeyAcceptedKeyTypes, perhaps via crypto-policies (which is also mentioned
> in sshd_config manpage for this option).
[...]

In my case, it was the 'PermitRootLogin = no' that was causing the login failure.  I had not made any modifications to the HostKey setting, but it would be good to state if there are requirements for that as well.

Comment 7 Sandro Bonazzola 2021-06-10 06:36:34 UTC
Re-targeting to 4.4.7 since oVirt 4.4.6 was released.

Comment 8 Sandro Bonazzola 2021-09-24 07:53:58 UTC
This issue is past ovirt-4.4.8-1 development cycle. Moving to ovirt-4.4.9.

If you believe this issue should be closed or moved back to 4.4.8-1 please do so.

Otherwise, I would like to ask for updating tickets ahead of the development cycle deadline (release date).

Comment 14 Steve Goodman 2021-10-05 06:07:05 UTC
Richard, can you please do a peer review?

Comment 15 Richard Hoch 2021-10-05 08:23:25 UTC
Two small comments -- otherwise, LGTM.

Comment 16 Steve Goodman 2021-10-05 08:58:58 UTC
Merged.


Note You need to log in before you can comment on or make changes to this bug.