Bug 1847547

Summary: SSH connection disabled during ISO installation
Product: Red Hat Enterprise Virtualization Manager Reporter: Jiri Macku <jmacku>
Component: redhat-virtualization-hostAssignee: Lev Veyde <lveyde>
Status: CLOSED ERRATA QA Contact: cshao <cshao>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.4.0CC: ahadas, cshao, lleistne, lsvaty, lveyde, mavital, michal.skrivanek, nlevy, pelauter, peyu, qiyuan, sbonazzo, shlei, weiwang, yaniwang
Target Milestone: ovirt-4.4.2Keywords: Reopened, ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: redhat-virtualization-host-productimg-4.4.2-2.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-05 13:09:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jiri Macku 2020-06-16 15:27:34 UTC
Description of problem:
During the installation when anaconda kicks in the ssh should be enabled to be able to log in and inspect the installation progress. The RHVH 4.3 and plain RHEL enable this feature. The RHVH 4.4 does not.

Version-Release number of selected component (if applicable):
RHVH 4.4

How reproducible:
100%

Steps to Reproduce:
1. Start ISO installation
2. When anaconda kicks in try to ssh into the host


Actual results:
The ssh connection is refused:
Connection reset by peer

Expected results:
The ssh connection is allowed the login successful.

Additional info:

Comment 2 Michal Skrivanek 2020-06-26 09:35:31 UTC
can you please describe why is this a regression and automation blocker?

Comment 3 Sandro Bonazzola 2020-06-26 13:38:19 UTC
Note for Nir: on oVirt Node sshd fails to start due to missing /etc/ssh/sshd_config.
I can confirm that also on a plain CentOS 8 install ISO the file is missing.

I verified on oVirt Node 4.3.9 and situation was exactly the same: sshd failing to start due to /etc/ssh/sshd_config.

If RHV-H 4.3 was allowing to start sshd providing a working /etc/ssh/sshd_config, it was a downstream only change.
If I have time today I'll check also status on RHV-H.

Comment 4 Sandro Bonazzola 2020-06-26 14:00:22 UTC
Verified that RHV-H 4.3.11 (RHVH-4.3-20200618.0-RHVH-x86_64-dvd1.iso) also fail to start sshd due to missing /etc/ssh/sshd_config.
Removing regression keyword and closing as not a bug.
If automation was able to run sshd in 4.3 and not in 4.4, it's an automation regression.

Comment 5 Jiri Macku 2020-06-30 08:52:33 UTC
The above mentioned sshd_config explanation is not relevant to the installation process while anaconda is running. While installation with anaconda is running there is special target active with other sshd service and configuration file. 

Comparing the RHEL 8.2 and RHVH 4.4 while installing.

Both systems have active anaconda target:
[anaconda root@lynx12 system]# systemctl get-default
anaconda.target

There is on both the anaconda-sshd.service, which consumes the /etc/ssh/sshd_config.anaconda configuration
[anaconda root@lynx12 system]# cat anaconda-sshd.service
[Unit]
Description=OpenSSH server daemon
Before=anaconda.target
After=syslog.target network.target
ConditionKernelCommandLine=|sshd
ConditionKernelCommandLine=|inst.sshd
ConditionKernelCommandLine=!inst.sshd=0
ConditionKernelCommandLine=!sshd=0
# TODO: use ConditionArchitecture in systemd v210 or later
ConditionPathIsDirectory=|/sys/hypervisor/s390

[Service]
EnvironmentFile=/etc/sysconfig/sshd
ExecStartPre=/usr/sbin/sshd-keygen
ExecStartPre=/usr/sbin/handle-sshpw
ExecStart=/usr/sbin/sshd -D $OPTIONS -f /etc/ssh/sshd_config.anaconda
ExecReload=/bin/kill -HUP $MAINPID

The /etc/ssh/sshd_config.anaconda configuration file is present on both systems and is identical.

But while on RHEL 8.2 the anaconda-sshd service is running:
[anaconda root@lynx12 system]# systemctl status anaconda-sshd
● anaconda-sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/anaconda-sshd.service; static; vendor preset: disabled)
   Active: active (running) since Mon 2020-06-29 17:06:09 UTC; 1min 52s ago
  Process: 1951 ExecStartPre=/usr/sbin/handle-sshpw (code=exited, status=0/SUCCESS)
  Process: 1889 ExecStartPre=/usr/sbin/sshd-keygen (code=exited, status=0/SUCCESS)
 Main PID: 2023 (sshd)
   CGroup: /system.slice/anaconda-sshd.service
           └─2023 /usr/sbin/sshd -D -f /etc/ssh/sshd_config.anaconda

on RHVH is NOT running:
[anaconda root@lynx12 ~]# systemctl status anaconda-sshd
● anaconda-sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/anaconda-sshd.service; static; vendo>
   Active: failed (Result: exit-code) since Mon 2020-06-29 17:25:27 UTC; 38s ago
  Process: 2419 ExecStart=/usr/sbin/sshd -D $OPTIONS -f /etc/ssh/sshd_config.an>
  Process: 2288 ExecStartPre=/usr/sbin/handle-sshpw (code=exited, status=0/SUCC>
 Main PID: 2419 (code=exited, status=255)

because the port is already taken:
at.com systemd[1]: Started OpenSSH server daemon.
at.com sshd[2419]: /etc/ssh/sshd_config.anaconda line 11: Deprecated option RSA>
at.com sshd[2419]: error: Bind to port 22 on 0.0.0.0 failed: Address already in>
at.com sshd[2419]: error: Bind to port 22 on :: failed: Address already in use.
at.com sshd[2419]: fatal: Cannot bind any address.
at.com systemd[1]: anaconda-sshd.service: Main process exited, code=exited, sta>
at.com systemd[1]: anaconda-sshd.service: Failed with result 'exit-code'.

On RHVH the port is taken by sshd.socket:
[anaconda root@lynx12 ~]# systemctl list-sockets | grep ssh
[::]:22                               sshd.socket                     sshd

because the sshd.socket is enabled:
[anaconda root@lynx12 ~]# systemctl status sshd.socket
● sshd.socket - OpenSSH Server Socket
   Loaded: loaded (/usr/lib/systemd/system/sshd.socket; enabled; vendor preset:>
   Active: active (listening) since Tue 2020-06-30 07:43:11 UTC; 3min 2s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
   Listen: [::]:22 (Stream)
 Accepted: 0; Connected: 0;
   CGroup: /system.slice/sshd.socket

On the other hand on the RHEL 8.2 the sshd.socket is disabled:
[anaconda root@lynx12 system]# systemctl status sshd.socket
● sshd.socket - OpenSSH Server Socket
   Loaded: loaded (/usr/lib/systemd/system/sshd.socket; disabled; vendor preset>
   Active: inactive (dead)
     Docs: man:sshd(8)
           man:sshd_config(5)
   Listen: [::]:22 (Stream)
 Accepted: 0; Connected: 0;

This makes the port 22 available for the anaconda-sshd service which could receive the incoming ssh connections.
Why is the socket in RHVH 4.4 in anaconda.target enabled and in RHEL disabled?

Comment 6 Lev Veyde 2020-08-10 18:42:53 UTC
Debugged this and built a new redhat-virtualization-host-productimg package that should fix this issue.

We'll need to build a new RHVH ISO so that we could test the fix.

Comment 9 Lev Veyde 2020-08-12 10:42:08 UTC
# systemctl status anaconda-sshd
● anaconda-sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/anaconda-sshd.service; static; vendor preset: disabled)
   Active: active (running) since Wed 2020-08-12 10:18:36 UTC; 20min ago
  Process: 2044 ExecStartPre=/usr/sbin/handle-sshpw (code=exited, status=0/SUCCESS)
 Main PID: 2050 (sshd)
    Tasks: 6 (limit: 50561)
   Memory: 15.8M
   CGroup: /system.slice/anaconda-sshd.service
           ├─2050 /usr/sbin/sshd -D -f /etc/ssh/sshd_config.anaconda
           ├─3307 sshd: root [priv]
           ├─3309 sshd: root@pts/5
           ├─3310 -bash
           ├─3334 systemctl status anaconda-sshd
           └─3335 less

Aug 12 10:37:32 <fqdn> sshd[3284]: rexec line 11: Deprecated option RSAAuthentication
Aug 12 10:37:33 <fqdn> sshd[3284]: reprocess config line 11: Deprecated option RSAAuthentication

Comment 11 Jiri Macku 2020-08-17 10:59:47 UTC
I can confirm, that in the latest RHVH iso the ssh is available during the anaconda is running.

Many thanks!

Comment 13 cshao 2020-08-26 11:37:32 UTC
Move to verified status according #c11.

Comment 15 errata-xmlrpc 2020-10-05 13:09:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Virtualization security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4172