Bug 1248372 - node loses public key after reboot
node loses public key after reboot
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-node (Show other bugs)
3.5.4
Unspecified Linux
high Severity high
: ovirt-3.6.1
: 3.6.0
Assigned To: Douglas Schilling Landgraf
cshao
: Regression, ZStream
Depends On:
Blocks: 1263211 1279960
  Show dependency treegraph
 
Reported: 2015-07-30 03:56 EDT by Ido Barkan
Modified: 2016-03-09 09:34 EST (History)
11 users (show)

See Also:
Fixed In Version: ovirt-node-3.6.0-0.24.20151209gitc0fa931
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1263211 1279960 (view as bug list)
Environment:
Last Closed: 2016-03-09 09:34:08 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Node
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 44184 master MERGED persisting the ssh fingerprint keys Never
oVirt gerrit 44210 ovirt-3.5 MERGED persisting the ssh fingerprint keys Never
oVirt gerrit 44323 master MERGED Providing correct path to fingerprint keyes Never
oVirt gerrit 46177 ovirt-3.5 MERGED Providing correct path to fingerprint keyes Never
oVirt gerrit 48047 master MERGED Waiting for sshd to generate the keys Never
oVirt gerrit 48098 master MERGED semodule: Add rule ro AVC sshd_keygen_t plymouth_exec_t Never
oVirt gerrit 48369 ovirt-3.6 MERGED semodule: Add rule ro AVC sshd_keygen_t plymouth_exec_t Never
oVirt gerrit 48370 ovirt-3.6 MERGED Waiting for sshd to generate the keys Never
oVirt gerrit 48372 ovirt-3.5 MERGED semodule: Add rule ro AVC sshd_keygen_t plymouth_exec_t Never
oVirt gerrit 48373 ovirt-3.5 MERGED Waiting for sshd to generate the keys Never

  None (edit)
Description Ido Barkan 2015-07-30 03:56:17 EDT
Description of problem:
After each host reboot, the ssh public key is regenerated, causing ssh to the host fail.

Version-Release number of selected component (if applicable):
rhev-hypervisor7-7.1-20150728.1.iso

How reproducible:
always

Steps to Reproduce:
1.install node
2. ssh, answer yes to ssh prompt for saving the host udentification
3. reboot the hots
4. ssh again

Actual results:
ssh will complain that the host has another id then what is stored.

Expected results:
ssh should just work

Additional info:
to work around this, I persisted /etc/ssh/ssh_host_ecdsa_key and /config/etc/ssh/ssh_host_ecdsa_key.pub
Comment 1 Ying Cui 2015-07-30 04:42:23 EDT
Test version:
rhev-hypervisor7-7.1-20150728.1 
ovirt-node-3.2.3-14.el7

Only can reproduce this issue after _first_ time RHEV-H reboot.
After the second time or more time reboot RHEV-H, the SSH fingerprint is not changed.
Comment 2 Fabian Deutsch 2015-07-31 09:05:20 EDT
The attached patch does not fix the issue.

1. Install RHEV-H 7.1
2. Reboot and login to TUI
3. Drop to shell
4. grep ssh_host /config(files does _not_ list the files in the initial description
Comment 5 Ying Cui 2015-10-12 02:10:43 EDT
# rpm -qa ovirt-node
ovirt-node-3.3.0-0.13.20151008git03eefb5.el7ev.noarch

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor release 7.2 (20151009.0.el7ev)

Steps to Reproduce:
1. Installed RHEV-H 7.2 for 3.6.0
2. ssh, answer yes to ssh prompt for saving the host key.
3. restart the rhevh host
4. ssh host once more

step 4: ssh works well.
Comment 6 Ying Cui 2015-11-02 01:12:41 EST
The regression occurred between rhev-hypervisor7-7.1-20151009.0.el7ev and rhev-hypervisor7-7.2-20151025.0.el7ev.

This issue is fixed on rhev-hypervisor7-7.1-20151009.0.el7ev. But in build rhev-hypervisor7-7.2-20151025.0.el7ev, this issue happened again, the SSH fingerprint was changed after _first_ time RHEV-H reboot.

Steps to Reproduce:
1. installed rhevh
2. enable ssh
3. ssh via another host, answer yes to ssh prompt for saving the host udentification
4. reboot the rhevh host
5. ssh again

After the second time or more time reboot RHEV-H, the SSH fingerprint is not changed.
Comment 7 Anatoly Litovsky 2015-11-02 11:17:05 EST
Hi 

Cannot recreate it with rhev-hypervisor7-7.2-20151025.0.iso

Can you please set me with a machine that it recreates
Comment 10 Douglas Schilling Landgraf 2015-11-04 17:51:21 EST
(In reply to Ying Cui from comment #6)
> The regression occurred between rhev-hypervisor7-7.1-20151009.0.el7ev and
> rhev-hypervisor7-7.2-20151025.0.el7ev.
> 
> This issue is fixed on rhev-hypervisor7-7.1-20151009.0.el7ev. But in build
> rhev-hypervisor7-7.2-20151025.0.el7ev, this issue happened again, the SSH
> fingerprint was changed after _first_ time RHEV-H reboot.
> 
> Steps to Reproduce:
> 1. installed rhevh
> 2. enable ssh
> 3. ssh via another host, answer yes to ssh prompt for saving the host
> udentification
> 4. reboot the rhevh host
> 5. ssh again
> 
> After the second time or more time reboot RHEV-H, the SSH fingerprint is not
> changed.

I couldn't reproduce this report yet. However, testing some upgrade scenarios and I have found this semodule AVC and might be related.

semodule: Add rule ro AVC sshd_keygen_t plymouth_exec_t
https://gerrit.ovirt.org/#/c/48098/(

@Ying, I will try more times to reproduce it. Is it 100% of time for you? 
If yes, if you add selinux=0 in kargs is it solve the issue?

Thanks!
Comment 12 Ying Cui 2015-11-05 01:37:41 EST
(In reply to Douglas Schilling Landgraf from comment #10)
> (In reply to Ying Cui from comment #6)
> > The regression occurred between rhev-hypervisor7-7.1-20151009.0.el7ev and
> > rhev-hypervisor7-7.2-20151025.0.el7ev.
> > 
> > This issue is fixed on rhev-hypervisor7-7.1-20151009.0.el7ev. But in build
> > rhev-hypervisor7-7.2-20151025.0.el7ev, this issue happened again, the SSH
> > fingerprint was changed after _first_ time RHEV-H reboot.
> > 
> > Steps to Reproduce:
> > 1. installed rhevh
> > 2. enable ssh
> > 3. ssh via another host, answer yes to ssh prompt for saving the host
> > udentification
> > 4. reboot the rhevh host
> > 5. ssh again
> > 
> > After the second time or more time reboot RHEV-H, the SSH fingerprint is not
> > changed.
> 
> I couldn't reproduce this report yet. However, testing some upgrade
> scenarios and I have found this semodule AVC and might be related.
> 
> semodule: Add rule ro AVC sshd_keygen_t plymouth_exec_t
> https://gerrit.ovirt.org/#/c/48098/(
> 
> @Ying, I will try more times to reproduce it. Is it 100% of time for you? 
> If yes, if you add selinux=0 in kargs is it solve the issue?

@Douglas, Yes, it is 100% reproduce on QE's machines.

Steps to Reproduce:
1. installed rhevh
2. enable ssh
3. ssh via another host, answer yes to ssh prompt for saving the host udentification
4. reboot the rhevh host
5. ssh again --- # issue happen, the SSH fingerprint is changed, need resave SSH fingerprint
6. reboot the rhevh host once more
7. ssh again --- # the SSH fingerprint is not changed

this issue only happen after the first time reboot rhevh. After the second time or more time reboot RHEV-H, the SSH fingerprint is not changed.
Comment 13 Ying Cui 2015-11-05 04:23:26 EST
Tested RHEV-H 7.2 20151029.0 for rhev 3.5.6, this issue still here.
So this bug affect rhev 3.5.z as well. 
And this issue was fixed in RHEVH 3.5.5(rhev-hypervisor7-7.1-20150917.0.iso and rhev-hypervisor6-6.7-20150917.0.iso) yet. So the regression happen on RHEV-H 3.5.6.
Comment 15 Anatoly Litovsky 2015-11-05 07:50:09 EST
Please set me up with the environment containing install from comment 9
Comment 20 Douglas Schilling Landgraf 2015-11-06 12:15:11 EST
Hi,

I couldn't reproduce the report but below my findings with the last logs from 
10.66.9.205. First, it seems a different root cause then the original report. 
(If confirmed, better open a different bug, to avoid mix issues)

Based in the event logs:

#1 - sshd-keygen daemon started (/var/log/messages)
=================================================
Nov  6 11:24:39 localhost sshd-keygen: Generating SSH2 RSA host key: [  OK  ]
Nov  6 11:24:39 localhost sshd-keygen: Generating SSH2 ECDSA host key: [  OK  ]
Nov  6 11:24:39 localhost sshd-keygen: Generating SSH2 ED25519 host key: [  OK  ]
Nov  6 11:24:39 localhost systemd: Started OpenSSH Server Key Generation.


#2 - Now in the same messages file, we see an AVC from selinux preventing/denying sshd-keygen daemon.  *Please note* this AVC was not int audit/audit.log.

/var/log/messages
=========================================
Nov  6 11:27:09 localhost sshd-keygen: Generating SSH2 RSA host key: [FAILED]
Nov  6 11:27:09 localhost kernel: type=1400 audit(1446809229.521:41): avc:  denied  { execute } for  pid=1150 comm="sshd-keygen" name="plymouth" dev="dm-0" ino=4492 scontext=system_u:system_r:sshd_keygen_t:s0 tcontext=unconfined_u:object_r:plymouth_exec_t:s0 tclass=file
Nov  6 11:27:09 localhost kernel: type=1300 audit(1446809229.521:41): arch=c000003e syscall=269 success=no exit=-13 a0=ffffffffffffff9c a1=1e500c0 a2=1 a3=7ffc2297fb30 items=0 ppid=1 pid=1150 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sshd-keygen" exe="/usr/bin/bash" subj=system_u:system_r:sshd_keygen_t:s0 key=(null)
Nov  6 11:27:09 localhost systemd: sshd-keygen.service: main process exited, code=exited, status=1/FAILURE
Nov  6 11:27:09 localhost systemd: Failed to start OpenSSH Server Key Generation.
Nov  6 11:27:09 localhost systemd: Unit sshd-keygen.service entered failed state.



#3 - Later, looks like ssh daemon is enabled and restarted via TUI, generating the ssh keys.
================================================================

* (/var/log/ovirt-node.log)
<snip>
2015-11-06 11:29:29,299    WARNING File "/etc/ssh/sshd_config" had already been persisted
</snip>

* (in /var/log/messages at the same time):
<snip>
Nov  6 11:29:29 localhost systemd: Started OpenSSH Server Key Generation.
Nov  6 11:29:29 localhost systemd: Starting OpenSSH server daemon...
Nov  6 11:29:29 localhost systemd: Started OpenSSH server daemon.
Nov  6 11:29:29 localhost systemd: Stopping OpenSSH server daemon...
Nov  6 11:29:29 localhost systemd: Stopping OpenSSH Server Key Generation...
Nov  6 11:29:29 localhost systemd: Started OpenSSH Server Key Generation.
Nov  6 11:29:29 localhost systemd: Starting OpenSSH server daemon...
Nov  6 11:29:29 localhost systemd: Started OpenSSH server daemon.
</snip>

At the same time keys got generated.
# ls -la /etc/ssh/
total 271
drwxr-xr-x.   2 root root        220 Nov  6 11:29 .
drwxr-xr-x. 104 root root       4560 Nov  6 11:29 ..
-rw-r--r--.   1 root root     242153 May 12 19:41 moduli
-rw-r--r--.   1 root root       2233 Nov  5 14:25 ssh_config
-rw-------.   1 root root       4414 Nov  6 11:29 sshd_config
-rw-r-----.   1 root ssh_keys    227 Nov  6 11:29 ssh_host_ecdsa_key
-rw-r--r--.   1 root root        162 Nov  6 11:29 ssh_host_ecdsa_key.pub
-rw-r-----.   1 root ssh_keys    387 Nov  6 11:29 ssh_host_ed25519_key
-rw-r--r--.   1 root root         82 Nov  6 11:29 ssh_host_ed25519_key.pub
-rw-r-----.   1 root ssh_keys   1679 Nov  6 11:24 ssh_host_rsa_key
-rw-r--r--.   1 root root        382 Nov  6 11:24 ssh_host_rsa_key.pub


A possible explanation for the report from Ying in comment#12
================================================================
First boot selinux prevent sshd-keygen to generate the ssh keys and our init/boot script couldn't persist the keys in /etc/ssh. Later, ssh service enabled/restarted via TUI the keys got generated but not persisted. So, user could ssh to machine but the /etc/ssh keys were not persisted and next reboot the files were lost and regenerated.
Comment 21 Fabian Deutsch 2015-11-06 12:44:45 EST
To confirm DOuglas findings in comment 20, Ying, could you try to install in permissive mode and see if you can reproduce the bug there?
Comment 24 Ying Cui 2015-11-09 04:30:52 EST
(In reply to Fabian Deutsch from comment #21)
> To confirm DOuglas findings in comment 20, Ying, could you try to install in
> permissive mode and see if you can reproduce the bug there?

@Douglas, yes, I tested with enforcing=0, not help. still happened the issue.

see comment 23, the issue already fixed in ovirt-node-iso-3.6-0.999.201511081037.el7.centos.iso, if you think that is different root cause for the same reproduce steps, we can split a new bug, do we need?
Comment 26 Douglas Schilling Landgraf 2015-11-09 10:50:56 EST
(In reply to Ying Cui from comment #24)
> (In reply to Fabian Deutsch from comment #21)
> > To confirm DOuglas findings in comment 20, Ying, could you try to install in
> > permissive mode and see if you can reproduce the bug there?
> 
> @Douglas, yes, I tested with enforcing=0, not help. still happened the issue.
> 

Thanks for the testing.

> see comment 23, the issue already fixed in
> ovirt-node-iso-3.6-0.999.201511081037.el7.centos.iso, if you think that is
> different root cause for the same reproduce steps, we can split a new bug,
> do we need?

It's not needed at moment, I will discuss with the others developers if we should include the patch or not in rhev-h meetings. If requires, we can use the same bug as it's related topic (ssh-keygen).
Comment 29 Ying Cui 2015-11-16 22:56:45 EST
The last test RHEV-H 7.2 for 3.6 beta 1 build (rhev-hypervisor7-7.2-20151112.1.el7ev) did not include this bug fix, both ovirt-node and rhevh build with this bug fix is not built in brew yet.
We have to move this ON_QA bug to MODIFIED for pending on available downstream build to QE.

# rpm -qa ovirt-node
ovirt-node-3.6.0-0.20.20151103git3d3779a.el7ev.noarch

# cat /etc/redhat-release 
Red Hat Enterprise Virtualization Hypervisor (Beta) release 7.2 (20151112.1.el7ev)
Comment 31 yileye 2015-11-23 23:45:30 EST
I can‘ t verified this bug,please provide the Fixed In Version
Comment 32 yileye 2015-12-08 02:38:22 EST
Please provide the Fixed In Version
Comment 33 yileye 2015-12-11 02:45:58 EST
Test version:
RHEV-H 7.2-20151210.1.el7ev
ovirt-node-3.6.0-0.24.20151209gitc0fa931.el7ev.noarch

Steps to Reproduce:
1. Install node
2. Ssh, answer yes to ssh prompt for saving the host udentification
3. Reboot the hots
4. Ssh again

Test result:
After step4, ssh should just work

So the bug is fixed, change bug status to VERIFIED.
Comment 37 errata-xmlrpc 2016-03-09 09:34:08 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0378.html

Note You need to log in before you can comment on or make changes to this bug.