Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1426151

Summary: Sshd.service could not work normally after upgrade
Product: [oVirt] ovirt-node Reporter: jianwu <jiawu>
Component: Installation & UpdateAssignee: Ryan Barry <rbarry>
Status: CLOSED CURRENTRELEASE QA Contact: jianwu <jiawu>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.0CC: bugs, cshao, dguo, gklein, huzhao, jiawu, leiwang, lsurette, mgoldboi, rbalakri, rbarry, Rhev-m-bugs, srevivo, weiwang, yaniwang, ycui, ykaul, ylavi, yzhao
Target Milestone: ovirt-4.1.1Keywords: Regression, ZStream
Target Release: 4.1Flags: rule-engine: ovirt-4.1+
rule-engine: blocker+
mgoldboi: planning_ack+
sbonazzo: devel_ack+
cshao: testing_ack+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: imgbased-0.9.17-0.1.el7ev Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1427149 (view as bug list) Environment:
Last Closed: 2017-04-21 09:33:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Node RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1427149    
Attachments:
Description Flags
log 1 to RHVH none

Description jianwu 2017-02-23 10:05:49 UTC
Description of problem:
After upgrade, sshd.service cannot work which happens only from redhat-virtualization-host-4.0-20161116.1.x86_64 to latest RHVH 4.0

Version-Release number of selected component (if applicable):
Before upgrade:
redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3)
After upgrade:
redhat-virtualization-host-4.0-20170222.0.x86_64
kernel-3.10.0-514.6.2.el7.x86_64
imgbased-0.8.13-0.1.el7ev.noarch

How reproducible:
100%
regression bug


Steps to Reproduce:
1. Install redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3) via anaconda
2. Reboot and log into this system
3. Set local repo and run #yum update
4. Reboot into new build
5. Run #systemctl status sshd

Actual results:
After step 5, sshd.service get something wrong and cannot access this host via ssh way.
#systemctl status sshd
service sshd status
● sshd.service - OpenSSH server daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-02-23 15:28:43 CST; 1h 45min ago
     Docs: man:sshd(8)
           man:sshd_config(5)
  Process: 1465 ExecStart=/usr/sbin/sshd $OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 1585 (sshd)
   CGroup: /system.slice/sshd.service
           └─1585 /usr/sbin/sshd

Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: Permissions 0640 for '/etc/ssh/ssh_host_ecdsa_key' are too open.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: It is required that your private key files are NOT accessible by others.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: This private key will be ignored.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: bad permissions: ignore key: /etc/ssh/ssh_host_ecdsa_key
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
Feb 23 15:29:28 dhcp-10-229.nay.redhat.com sshd[2898]: error: It is required that your private key files are NOT accessible by others.

Expected results:
After step 5, sshd.service run normally and could access this host via ssh way

Additional info:
1. It does not get wrong about sshd.service when upgrade from redhat-virtualization-host-4.0-20160919.0.x86_64(el7.2) to the latest RHVH 4.0
2. libssh is missing in latest 4.0.7 build. Not sure it is related for this bug
   *-libssh-0.7.1-2.el7.x86_64*

Comment 1 jianwu 2017-02-23 10:20:06 UTC
No such issue happen in redhat-virtualization-host-4.0-20170201.0, so I thinks it is a regression bug.

Comment 2 jianwu 2017-02-23 10:53:56 UTC
Created attachment 1256865 [details]
log 1 to RHVH

Comment 3 Ryan Barry 2017-02-23 18:15:11 UTC
libssh was added (and removed) as a dependency of cockpit.

I can't reproduce this, and the permissions here are exactly the same as a clean install. Were any additional steps taken? Some problem with the environment?

The logs look like there's an attempt to directly authenticate using /etc/ssh/ssh_host_ecdsa_key as a priv key, which will not even work on a base install at 0640 (these are the default permissions on EL distros and fedora)

Can you please post exact steps to reproduce and the complete "journalctl -u sshd.service" log?

---------------------------------------------------------------------------------------

[root@localhost ~]# imgbase layout
rhvh-4.0-0.20161116.0
 +- rhvh-4.0-0.20161116.0+1
rhvh-4.0-0.20170222.0
 +- rhvh-4.0-0.20170222.0+1
[root@localhost ~]# imgbase w
[INFO] You are on rhvh-4.0-0.20170222.0+1
[root@localhost ~]# ls -l /etc/ssh
total 276
-rw-r--r--. 1 root root     242153 Sep  6 09:30 moduli
-rw-r--r--. 1 root root       2208 Sep  6 09:30 ssh_config
-rw-------. 1 root root       4361 Sep  6 09:30 sshd_config
-rw-r-----. 1 root ssh_keys    227 Feb 23 07:07 ssh_host_ecdsa_key
-rw-r--r--. 1 root root        162 Feb 23 07:07 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys    387 Feb 23 07:07 ssh_host_ed25519_key
-rw-r--r--. 1 root root         82 Feb 23 07:07 ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys   1675 Feb 23 07:07 ssh_host_rsa_key
-rw-r--r--. 1 root root        382 Feb 23 07:07 ssh_host_rsa_key.pub
[root@localhost ~]# journalctl -u sshd.service
-- Logs begin at Thu 2017-02-23 08:15:52 MST, end at Thu 2017-02-23 11:07:55 MST. --
Feb 23 08:16:14 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
Feb 23 08:16:16 localhost.localdomain systemd[1]: PID file /var/run/sshd.pid not readable (yet?) after start.
Feb 23 08:16:16 localhost.localdomain sshd[1484]: Server listening on 0.0.0.0 port 22.
Feb 23 08:16:16 localhost.localdomain sshd[1484]: Server listening on :: port 22.
Feb 23 08:16:16 localhost.localdomain systemd[1]: Started OpenSSH server daemon.
Feb 23 10:44:07 localhost.localdomain sshd[3276]: Accepted password for root from 192.168.122.1 port 55050 ssh2

--------------------------------------------------------------------------------------

[root@localhost ~]# imgbase w
[INFO] You are on rhvh-4.0-0.20170222.0+1
[root@localhost ~]# imgbase layout
rhvh-4.0-0.20170222.0
 +- rhvh-4.0-0.20170222.0+1
[root@localhost ~]# ls -l /etc/ssh
total 276
-rw-r--r--. 1 root root     242153 Dec 20 09:27 moduli
-rw-r--r--. 1 root root       2208 Dec 20 09:27 ssh_config
-rw-------. 1 root root       4361 Dec 20 09:27 sshd_config
-rw-r-----. 1 root ssh_keys    227 Feb 23 08:25 ssh_host_ecdsa_key
-rw-r--r--. 1 root root        162 Feb 23 08:25 ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys    387 Feb 23 08:25 ssh_host_ed25519_key
-rw-r--r--. 1 root root         82 Feb 23 08:25 ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys   1675 Feb 23 08:25 ssh_host_rsa_key
-rw-r--r--. 1 root root        382 Feb 23 08:25 ssh_host_rsa_key.pub
[root@localhost ~]# journalctl -u sshd.service
-- Logs begin at Thu 2017-02-23 08:24:54 MST, end at Thu 2017-02-23 11:08:16 MST. --
Feb 23 08:25:10 localhost.localdomain systemd[1]: Starting OpenSSH server daemon...
Feb 23 08:25:11 localhost.localdomain sshd[1481]: Server listening on 0.0.0.0 port 22.
Feb 23 08:25:11 localhost.localdomain sshd[1481]: Server listening on :: port 22.
Feb 23 08:25:11 localhost.localdomain systemd[1]: Started OpenSSH server daemon.
Feb 23 10:43:55 localhost.localdomain sshd[20818]: Accepted password for root from 192.168.122.1 port 34926 ssh2

Comment 4 jianwu 2017-02-24 01:55:40 UTC
Hi Ryan,
I have send you a email about this bug's detail log, and i will try to re-verify this problem on another machine, because we have reproduced this issue on our local machine 100%.

Jiawu
Thanks

Comment 6 Huijuan Zhao 2017-03-14 05:55:01 UTC
Test version:
From:
redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3)
To:
redhat-virtualization-host-4.1-20170308.1.x86_64
imgbased-0.9.17-0.1.el7ev.noarch


Test Steps:
1. Install redhat-virtualization-host-4.0-20161116.1.x86_64(el7.3) via anaconda
2. Reboot and log into this system
3. Set local repos and run #yum update
4. Reboot into new build redhat-virtualization-host-4.1-20170308.1
5. Run #systemctl status sshd in host, ssh host from other machine

Test results:
In step5, service sshd status is normal, can ssh host successful from other machine.


But according to https://bugzilla.redhat.com/show_bug.cgi?id=1427468#c2, there is still sshd issue when upgrade twice, this issue is tracked by Bug 1427468.

So change this bug status to VERIFIED.