Bug 1585325

> If the  /usr/sbin/sshd-keygen command is run first to create the keys then execution of the 'systemctl status sshd-keygen.service' command does not produce the above error.

If the sshd-keygen is executed manually, the keys are created and the service is not ran at all from systemd (see the content of the service file):

  ConditionFileNotEmpty=|!/etc/ssh/ssh_host_rsa_key

We did not update OpenSSH in RHEL7.6 so this must be something changed in systemd, in openssl or elsewhere.

Also I can not reproduce it on my virtual machine with the latest repo with the following test:

/CoreOS/openssh/Sanity/autocreate-server-keys-sanity-test

I have the following configuration of my VM:

Linux 3.10.0-862.3.2.el7.x86_64 #1 SMP Tue May 15 18:22:15 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
openssh-7.4p1-16.el7.x86_64
systemd-219-57.el7.x86_64
openssl-1.0.2k-12.el7.x86_64

Can you modify the sshd-keygen in your installation to produce more verbose logs (set -x) and rerun the test to get some more debug information?

Looks like I hit the same problem on my xen hvm guest.

I installed RHEL-7.6-20180528.n.0 and sshd did not start too.
After ran 'sshd-keygen' manually, then sshd can start normally.

[root@dhcp-3-19 ~]# systemctl status sshd -l
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2018-06-05 14:08:49 HKT; 7s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 3797 ExecStart=/usr/sbin/sshd -D $OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 3797 (code=exited, status=1/FAILURE)

Jun 05 14:08:49 dhcp-3-19.nay.redhat.com systemd[1]: sshd.service: main[ 5447.398309] serial8250: too much work for irq4
 process exited, code=exited, status=1/FAILURE
Jun 05 14:08:49 dhcp-3-19.nay.redhat.com sshd[3797]: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Jun 05 14:08:49 dhcp-3-19.nay.redhat.com sshd[3797]: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jun 05 14:08:49 dhcp-3-19.nay.redhat.com sshd[3797]: sshd: no hostkeys available -- exiting.
Jun 05 14:08:49 dhcp-3-19.nay.redhat.com systemd[1]: Failed to start OpenSSH server daemon.
Jun 05 14:08:49 dhcp-3-19.nay.redhat.com systemd[ 5447.504361] serial8250: too much work for irq4
[1]: Unit sshd.service entered failed state.
Jun 05 14:08:49 dhcp-3-19.nay.redhat.com systemd[1]: sshd.service failed.
[root@dhcp-3-19 ~]# sshd-keygen 
Generating SSH2 RSA host key: [  OK  ]
Generating SSH2 ECDSA host key: [  OK  ]
Generating SSH2 ED25519 host key: [  OK  ]
[root@dhcp-3-19 ~]# systemctl restart sshd
[root@dhcp-3-19 ~]# systemctl status sshd -l
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-06-05 14:12:18 HKT; 3min 0s ago
     Docs: man:sshd(8)
           man:sshd_config(5)
 Main PID: 3896 (sshd)
    Tasks: 1
   CGroup: /system.slice/sshd.service
           └─3896 /usr/sbin/sshd -D

Jun 05 14:12:18 dhcp-3-19.nay.redhat.com systemd[1]: Starting OpenSSH server daemon...
Jun 05 14:12:18 dhcp[ 5828.566147] serial8250: too much work for irq4
-3-19.nay.redhat.com sshd[3896]: Server listening on 0.0.0.0 port 22.
Jun 05 14:12:18 dhcp-3-19.nay.redhat.com sshd[3896]: Server listening on :: port 22.
Jun 05 14:12:18 dhcp-3-19.nay.redhat.com systemd[1]: Started OpenSSH server daemon.
[root@dhcp-3-19 ~]# uname -a
Linux dhcp-3-19.nay.redhat.com 3.10.0-893.el7.x86_64 #1 SMP Thu May 24 21:37:14 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Can somebody attach some useful debug information or provide me a reproducer (for example with beaker)? Does it work with SELinux permissive? Are there any AVCs?

Jakub - In the meantime, I am trying to set up a system so that you can troubleshoot this issue. Please stay tuned. Thanks - Karen

The SELinux is preventing us to do that. Additionally, even the auditd was not successfully started when I connected to the testing machine provided by Xiao (thanks!).

So after going to permissive, I was able to start both auditd and sshd. Then removing the keys in /etc/sshd/ssh_host_*, switching back to the enforcing and trying to start the service again, the following AVC appear.

# ausearch -m AVC
----
time->Wed Jun  6 05:15:34 2018
type=PROCTITLE msg=audit(1528276534.738:2065): proctitle="(null)"
type=SYSCALL msg=audit(1528276534.738:2065): arch=c000003e syscall=59 success=no exit=-13 a0=1dd0480 a1=1d91f50 a2=1dcbc90 a3=7ffe941ceae0 items=0 ppid=14720 pid=14724 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ssh-keygen" exe="/usr/bin/ssh-keygen" subj=system_u:system_r:sshd_keygen_t:s0 key=(null)
type=AVC msg=audit(1528276534.738:2065): avc:  denied  { map } for  pid=14724 comm="ssh-keygen" path="/usr/bin/ssh-keygen" dev="dm-0" ino=12845996 scontext=system_u:system_r:sshd_keygen_t:s0 tcontext=system_u:object_r:ssh_keygen_exec_t:s0 tclass=file permissive=0
----
time->Wed Jun  6 05:15:34 2018
type=PROCTITLE msg=audit(1528276534.738:2066): proctitle=2F62696E2F62617368002F7573722F7362696E2F737368642D6B657967656E
type=SYSCALL msg=audit(1528276534.738:2066): arch=c000003e syscall=269 success=no exit=-13 a0=ffffffffffffff9c a1=1dd47b0 a2=1 a3=7ffe941ce160 items=0 ppid=1 pid=14720 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sshd-keygen" exe="/usr/bin/bash" subj=system_u:system_r:sshd_keygen_t:s0 key=(null)
type=AVC msg=audit(1528276534.738:2066): avc:  denied  { execute } for  pid=14720 comm="sshd-keygen" name="plymouth" dev="dm-0" ino=12846391 scontext=system_u:system_r:sshd_keygen_t:s0 tcontext=system_u:object_r:plymouth_exec_t:s0 tclass=file permissive=0


I saw this issue in bug #1583531, but it was missing reproducer. Lukas, what can we try to figure what is going on there further?

# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.5 (Maipo)
# grep /functions `which sshd-keygen `
. /etc/rc.d/init.d/functions
# grep plymouth /etc/rc.d/init.d/functions
    if [ -x /bin/plymouth ]; then
        /bin/plymouth --update="$1"
    [ -x /bin/plymouth ] && /bin/plymouth --details
#

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3111