Bug 1427149

Summary: [downstream clone - 4.0.7] Sshd.service could not work normally after upgrade
Product: Red Hat Enterprise Virtualization Manager Reporter: rhev-integ
Component: imgbasedAssignee: Ryan Barry <rbarry>
Status: CLOSED ERRATA QA Contact: jianwu <jiawu>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: bugs, cshao, dfediuck, dguo, gklein, jiawu, leiwang, lsurette, mgoldboi, rbalakri, rbarry, Rhev-m-bugs, srevivo, weiwang, yaniwang, ycui, ykaul, ylavi, yzhao
Target Milestone: ovirt-4.0.7Keywords: Regression, ZStream
Target Release: ---Flags: rbarry: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: imgbased-0.8.15-0.1.el7ev Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1426151 Environment:
Last Closed: 2017-03-16 15:40:08 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1426151    
Bug Blocks:    

Description rhev-integ 2017-02-27 13:23:30 UTC
+++ This bug is an upstream to downstream clone. The original bug is: +++
+++   bug 1426151 +++
======================================================================

Description of problem:
After upgrade, sshd.service cannot work which happens only from redhat-virtualization-host-4.0-20161116.1.x86_64 to latest RHVH 4.0

Version-Release number of selected component (if applicable):
Before upgrade:
redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3)
After upgrade:
redhat-virtualization-host-4.0-20170222.0.x86_64
kernel-3.10.0-514.6.2.el7.x86_64
imgbased-0.8.13-0.1.el7ev.noarch

How reproducible:
100%
regression bug


Steps to Reproduce:
1. Install redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3) via anaconda
2. Reboot and log into this system
3. Set local repo and run #yum update
4. Reboot into new build
5. Run #systemctl status sshd

Actual results:
After step 5, sshd.service get something wrong and cannot access this host via ssh way.
#systemctl status sshd
service sshd status
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-02-23 15:28:43 CST; 1h 45min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 1465 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1585 (sshd)
   CGroup: /system.slice/sshd.service
           └─1585 /usr/sbin/sshd

Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: It is required that your private key files are NOT accessible by others.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: This private key will be ignored.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: bad permissions: ignore key: /etc/ssh/ssh_host_ecdsa_key
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: It is required that your private key files are NOT accessible by others.

Expected results:
After step 5, sshd.service run normally and could access this host via ssh way

Additional info:
1. It does not get wrong about sshd.service when upgrade from redhat-virtualization-host-4.0-20160919.0.x86_64(el7.2) to the latest RHVH 4.0
2. libssh is missing in latest 4.0.7 build. Not sure it is related for this bug
   *-libssh-0.7.1-2.el7.x86_64*

(Originally by Jian Wu)

Comment 1 rhev-integ 2017-02-27 13:23:42 UTC
No such issue happen in redhat-virtualization-host-4.0-20170201.0, so I thinks it is a regression bug.

(Originally by Jian Wu)

Comment 3 rhev-integ 2017-02-27 13:23:49 UTC
Created attachment 1256865 [details]
log 1 to RHVH

(Originally by Jian Wu)

Comment 4 rhev-integ 2017-02-27 13:23:56 UTC
libssh was added (and removed) as a dependency of cockpit.

I can't reproduce this, and the permissions here are exactly the same as a clean install. Were any additional steps taken? Some problem with the environment?

The logs look like there's an attempt to directly authenticate using /etc/ssh/ssh_host_ecdsa_key as a priv key, which will not even work on a base install at 0640 (these are the default permissions on EL distros and fedora)

Can you please post exact steps to reproduce and the complete "journalctl -u sshd.service" log?

---------------------------------------------------------------------------------------

[root@localhost ~]# imgbase layout
rhvh-4.0-0.20161116.0
 +- rhvh-4.0-0.20161116.0+1
rhvh-4.0-0.20170222.0
 +- rhvh-4.0-0.20170222.0+1
[root@localhost ~]# imgbase w
[INFO] You are on rhvh-4.0-0.20170222.0+1
[root@localhost ~]# ls -l /etc/ssh
total 276
-rw-r--r--. 1 root root     242153 Sep  6 09:30 moduli
-rw-r--r--. 1 root root       2208 Sep  6 09:30 ssh_config
-rw-------. 1 root root       4361 Sep  6 09:30 sshd_config
-rw-r-----. 1 root ssh_keys    227 Feb 23 07:07 ssh_host_ecdsa_key
-rw-r--r--. 1 root root        162 Feb 23 07:07 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys    387 Feb 23 07:07 ssh_host_ed25519_key
-rw-r--r--. 1 root root         82 Feb 23 07:07 ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys   1675 Feb 23 07:07 ssh_host_rsa_key
-rw-r--r--. 1 root root        382 Feb 23 07:07 ssh_host_rsa_key.pub
[root@localhost ~]# journalctl -u sshd.service
-- Logs begin at Thu 2017-02-23 08:15:52 MST, end at Thu 2017-02-23 11:07:55 MST. --
Feb 23 08:16:14 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
Feb 23 08:16:16 localhost.localdomain systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
Feb 23 08:16:16 localhost.localdomain sshd[1484]: Server listening on 0.0.0.0 port 22.
Feb 23 08:16:16 localhost.localdomain sshd[1484]: Server listening on :: port 22.
Feb 23 08:16:16 localhost.localdomain systemd[1]: Started OpenSSH server daemon.
Feb 23 10:44:07 localhost.localdomain sshd[3276]: Accepted password for root from 192.168.122.1 port 55050 ssh2

--------------------------------------------------------------------------------------

[root@localhost ~]# imgbase w
[INFO] You are on rhvh-4.0-0.20170222.0+1
[root@localhost ~]# imgbase layout
rhvh-4.0-0.20170222.0
 +- rhvh-4.0-0.20170222.0+1
[root@localhost ~]# ls -l /etc/ssh
total 276
-rw-r--r--. 1 root root     242153 Dec 20 09:27 moduli
-rw-r--r--. 1 root root       2208 Dec 20 09:27 ssh_config
-rw-------. 1 root root       4361 Dec 20 09:27 sshd_config
-rw-r-----. 1 root ssh_keys    227 Feb 23 08:25 ssh_host_ecdsa_key
-rw-r--r--. 1 root root        162 Feb 23 08:25 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys    387 Feb 23 08:25 ssh_host_ed25519_key
-rw-r--r--. 1 root root         82 Feb 23 08:25 ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys   1675 Feb 23 08:25 ssh_host_rsa_key
-rw-r--r--. 1 root root        382 Feb 23 08:25 ssh_host_rsa_key.pub
[root@localhost ~]# journalctl -u sshd.service
-- Logs begin at Thu 2017-02-23 08:24:54 MST, end at Thu 2017-02-23 11:08:16 MST. --
Feb 23 08:25:10 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
Feb 23 08:25:11 localhost.localdomain sshd[1481]: Server listening on 0.0.0.0 port 22.
Feb 23 08:25:11 localhost.localdomain sshd[1481]: Server listening on :: port 22.
Feb 23 08:25:11 localhost.localdomain systemd[1]: Started OpenSSH server daemon.
Feb 23 10:43:55 localhost.localdomain sshd[20818]: Accepted password for root from 192.168.122.1 port 34926 ssh2

(Originally by Ryan Barry)

Comment 5 rhev-integ 2017-02-27 13:24:04 UTC
Hi Ryan,
I have send you a email about this bug's detail log, and i will try to re-verify this problem on another machine, because we have reproduced this issue on our local machine 100%.

Jiawu
Thanks

(Originally by Jian Wu)

Comment 6 jianwu 2017-03-06 02:21:47 UTC
Hi,
I have verified this bug on redhat-virtualization-host-4.0-20170302.0.x86_64,

Version-Release number of selected component (if applicable):
Before upgrade:
redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3)
After upgrade:
redhat-virtualization-host-4.0-20170302.0
imgbased-0.8.15-0.1.el7ev.noarch
kernel-3.10.0-514.10.2.el7.x86_64

Steps to test:
1. Install redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3) via anaconda
2. Reboot and log into this system
3. Set local repo and run #yum update
4. Reboot into new build
5. Run #systemctl status sshd

Actual results:
After step 5, 
#systemctl status sshd
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-03-06 10:08:18 CST; 6min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 1483 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1540 (sshd)
   CGroup: /system.slice/sshd.service
           └─1540 /usr/sbin/sshd

So I think this bug is fixed, I will change status to VERIFIED.

Comment 9 errata-xmlrpc 2017-03-16 15:40:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2017-0549.html