Bug 1951087

Summary: [Edge] SSH failed connected to RHEL for Edge VM after upgrade kernel to rt kernel from "normal" kernel
Product: Red Hat Enterprise Linux 8 Reporter: Xiaofeng Wang <xiaofwan>
Component: osbuildAssignee: Image Builder team <osbuilders>
Status: CLOSED ERRATA QA Contact: Xiaofeng Wang <xiaofwan>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 8.4CC: ckellner, elpereir, leiwang, perobins, tgunders, yih
Target Milestone: betaKeywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-09 18:46:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Xiaofeng Wang 2021-04-19 15:25:22 UTC
Description of problem:
After I run rpm-ostree upgrade from commit with "normal" kernel to commit with rt kernel, the edge VM can't be sshed.
When I login from console, I found the ssh service do not get started. The journalctl log reports the following error.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Unable to load host key "/etc/ssh/ssh_host_ed25519_key": bad permissions
Unable to load host key: /etc/ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

And I found file owner changed after upgrade to rt kernel
[admin@vm ~]$ ll /etc/ssh
total 600
-rw-r--r--. 1 root root    577388 Feb  3 15:18 moduli
-rw-r--r--. 1 root root      1770 Feb  3 15:18 ssh_config
drwxr-xr-x. 2 root root        28 Feb  3 15:18 ssh_config.d
-rw-------. 1 root polkitd    492 Feb  3 15:11 ssh_host_ecdsa_key
-rw-r--r--. 1 root root       162 Feb  3 15:11 ssh_host_ecdsa_key.pub
-rw-------. 1 root polkitd    387 Feb  3 15:11 ssh_host_ed25519_key
-rw-r--r--. 1 root root        82 Feb  3 15:11 ssh_host_ed25519_key.pub
-rw-------. 1 root polkitd   2578 Feb  3 15:11 ssh_host_rsa_key
-rw-r--r--. 1 root root       554 Feb  3 15:11 ssh_host_rsa_key.pub
-rw-------. 1 root root      4269 Feb  3 15:18 sshd_config

Version-Release number of selected component (if applicable):
python3-osbuild-27.1-1.el8.noarch
osbuild-composer-worker-28.3-1.el8.x86_64
osbuild-27.1-1.el8.noarch
osbuild-composer-core-28.3-1.el8.x86_64
osbuild-composer-28.3-1.el8.x86_64
osbuild-ostree-27.1-1.el8.noarch
osbuild-selinux-27.1-1.el8.noarch

How reproducible:

Steps to Reproduce:
1. Build a container image with "normal" kernel
$ cat container.toml
name = "container"
description = "A base rhel-edge container image"
version = "0.0.1"
modules = []
groups = []
[[packages]]
name = "python36"
version = "*"
[[customizations.user]]
name = "admin"
description = "Administrator account"
password = "$6$GRmb7S0p8vsYmXzH$o0E020S.9JQGaHkszoog4ha4AQVs3sk8q0DvLjSMxoxHBKnB2FBXGQ/OkwZQfW/76ktHd0NX5nls2LPxPuUdl."
key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC61wMCjOSHwbVb4VfVyl5sn497qW4PsdQ7Ty7aD6wDNZ/QjjULkDV/yW5WjDlDQ7UqFH0Sr7vywjqDizUAqK7zM5FsUKsUXWHWwg/ehKg8j9xKcMv11AkFoUoujtfAujnKODkk58XSA9whPr7qcw3vPrmog680pnMSzf9LC7J6kXfs6lkoKfBh9VnlxusCrw2yg0qI1fHAZBLPx7mW6+me71QZsS6sVz8v8KXyrXsKTdnF50FjzHcK9HXDBtSJS5wA3fkcRYymJe0o6WMWNdgSRVpoSiWaHHmFgdMUJaYoCfhXzyl7LtNb3Q+Sveg+tJK7JaRXBLMUllOlJ6ll5Hod root@localhost"
home = "/home/admin/"
groups = ["wheel"]

2. Install it on VM
3. Build an upgrade container image with rt kernel
$ cat container.toml
name = "upgrade"
description = "rhel-edge upgrade image"
version = "0.0.1"
modules = []
groups = []
[[packages]]
name = "python36"
version = "*"
[customizations.kernel]
name = "kernel-rt"
[[customizations.user]]
name = "admin"
description = "Administrator account"
password = "$6$GRmb7S0p8vsYmXzH$o0E020S.9JQGaHkszoog4ha4AQVs3sk8q0DvLjSMxoxHBKnB2FBXGQ/OkwZQfW/76ktHd0NX5nls2LPxPuUdl."
key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC61wMCjOSHwbVb4VfVyl5sn497qW4PsdQ7Ty7aD6wDNZ/QjjULkDV/yW5WjDlDQ7UqFH0Sr7vywjqDizUAqK7zM5FsUKsUXWHWwg/ehKg8j9xKcMv11AkFoUoujtfAujnKODkk58XSA9whPr7qcw3vPrmog680pnMSzf9LC7J6kXfs6lkoKfBh9VnlxusCrw2yg0qI1fHAZBLPx7mW6+me71QZsS6sVz8v8KXyrXsKTdnF50FjzHcK9HXDBtSJS5wA3fkcRYymJe0o6WMWNdgSRVpoSiWaHHmFgdMUJaYoCfhXzyl7LtNb3Q+Sveg+tJK7JaRXBLMUllOlJ6ll5Hod root@localhost"
home = "/home/admin/"
groups = ["wheel"]
4. Run rpm-ostree upgrade on VM
5. SSH to VM

Actual results:
Can't SSH to VM

Expected results:
SSH to VM without any error

Additional info:

Comment 4 Xiaofeng Wang 2021-08-31 15:04:32 UTC
Verified on the following builds:
osbuild-composer-worker-33-1.el8.x86_64
osbuild-35-1.el8.noarch
osbuild-composer-core-33-1.el8.x86_64
osbuild-selinux-35-1.el8.noarch
osbuild-composer-33-1.el8.x86_64
osbuild-ostree-35-1.el8.noarch
python3-osbuild-35-1.el8.noarch

Comment 6 errata-xmlrpc 2021-11-09 18:46:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (osbuild bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4273