1951087 – [Edge] SSH failed connected to RHEL for Edge VM after upgrade kernel to rt kernel from "normal" kernel

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1951087 - [Edge] SSH failed connected to RHEL for Edge VM after upgrade kernel to rt kernel from "normal" kernel

Summary: [Edge] SSH failed connected to RHEL for Edge VM after upgrade kernel to rt ke...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 8
Classification:	Red Hat
Component:	osbuild
Sub Component:
Version:	8.4
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	beta
Target Release:	---
Assignee:	Image Builder team
QA Contact:	Xiaofeng Wang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-04-19 15:25 UTC by Xiaofeng Wang
Modified:	2021-11-10 01:17 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-11-09 18:46:58 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	osbuild osbuild-composer issues 1222	None	open	GIDs should be consistent for common groups in OSTree commits	2021-04-19 15:33:48 UTC
Github	osbuild osbuild-composer pull 1659	None	Closed	CRI-O issue - Failed to inspect image with size for layer unknown	2022-05-28 06:22:26 UTC
Github	osbuild osbuild pull 620	None	open	Fix issue with changing content in /etc/passwd and /etc/groups in OSTree deployments	2021-04-19 15:33:48 UTC
Red Hat Product Errata	RHBA-2021:4273	None	None	None	2021-11-09 18:47:14 UTC

Description Xiaofeng Wang 2021-04-19 15:25:22 UTC

Description of problem:
After I run rpm-ostree upgrade from commit with "normal" kernel to commit with rt kernel, the edge VM can't be sshed.
When I login from console, I found the ssh service do not get started. The journalctl log reports the following error.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0640 for '/etc/ssh/ssh_host_ed25519_key' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Unable to load host key "/etc/ssh/ssh_host_ed25519_key": bad permissions
Unable to load host key: /etc/ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

And I found file owner changed after upgrade to rt kernel
[admin@vm ~]$ ll /etc/ssh
total 600
-rw-r--r--. 1 root root    577388 Feb  3 15:18 moduli
-rw-r--r--. 1 root root      1770 Feb  3 15:18 ssh_config
drwxr-xr-x. 2 root root        28 Feb  3 15:18 ssh_config.d
-rw-------. 1 root polkitd    492 Feb  3 15:11 ssh_host_ecdsa_key
-rw-r--r--. 1 root root       162 Feb  3 15:11 ssh_host_ecdsa_key.pub
-rw-------. 1 root polkitd    387 Feb  3 15:11 ssh_host_ed25519_key
-rw-r--r--. 1 root root        82 Feb  3 15:11 ssh_host_ed25519_key.pub
-rw-------. 1 root polkitd   2578 Feb  3 15:11 ssh_host_rsa_key
-rw-r--r--. 1 root root       554 Feb  3 15:11 ssh_host_rsa_key.pub
-rw-------. 1 root root      4269 Feb  3 15:18 sshd_config

Version-Release number of selected component (if applicable):
python3-osbuild-27.1-1.el8.noarch
osbuild-composer-worker-28.3-1.el8.x86_64
osbuild-27.1-1.el8.noarch
osbuild-composer-core-28.3-1.el8.x86_64
osbuild-composer-28.3-1.el8.x86_64
osbuild-ostree-27.1-1.el8.noarch
osbuild-selinux-27.1-1.el8.noarch

How reproducible:

Steps to Reproduce:
1. Build a container image with "normal" kernel
$ cat container.toml
name = "container"
description = "A base rhel-edge container image"
version = "0.0.1"
modules = []
groups = []
[[packages]]
name = "python36"
version = "*"
[[customizations.user]]
name = "admin"
description = "Administrator account"
password = "$6$GRmb7S0p8vsYmXzH$o0E020S.9JQGaHkszoog4ha4AQVs3sk8q0DvLjSMxoxHBKnB2FBXGQ/OkwZQfW/76ktHd0NX5nls2LPxPuUdl."
key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC61wMCjOSHwbVb4VfVyl5sn497qW4PsdQ7Ty7aD6wDNZ/QjjULkDV/yW5WjDlDQ7UqFH0Sr7vywjqDizUAqK7zM5FsUKsUXWHWwg/ehKg8j9xKcMv11AkFoUoujtfAujnKODkk58XSA9whPr7qcw3vPrmog680pnMSzf9LC7J6kXfs6lkoKfBh9VnlxusCrw2yg0qI1fHAZBLPx7mW6+me71QZsS6sVz8v8KXyrXsKTdnF50FjzHcK9HXDBtSJS5wA3fkcRYymJe0o6WMWNdgSRVpoSiWaHHmFgdMUJaYoCfhXzyl7LtNb3Q+Sveg+tJK7JaRXBLMUllOlJ6ll5Hod root@localhost"
home = "/home/admin/"
groups = ["wheel"]

2. Install it on VM
3. Build an upgrade container image with rt kernel
$ cat container.toml
name = "upgrade"
description = "rhel-edge upgrade image"
version = "0.0.1"
modules = []
groups = []
[[packages]]
name = "python36"
version = "*"
[customizations.kernel]
name = "kernel-rt"
[[customizations.user]]
name = "admin"
description = "Administrator account"
password = "$6$GRmb7S0p8vsYmXzH$o0E020S.9JQGaHkszoog4ha4AQVs3sk8q0DvLjSMxoxHBKnB2FBXGQ/OkwZQfW/76ktHd0NX5nls2LPxPuUdl."
key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC61wMCjOSHwbVb4VfVyl5sn497qW4PsdQ7Ty7aD6wDNZ/QjjULkDV/yW5WjDlDQ7UqFH0Sr7vywjqDizUAqK7zM5FsUKsUXWHWwg/ehKg8j9xKcMv11AkFoUoujtfAujnKODkk58XSA9whPr7qcw3vPrmog680pnMSzf9LC7J6kXfs6lkoKfBh9VnlxusCrw2yg0qI1fHAZBLPx7mW6+me71QZsS6sVz8v8KXyrXsKTdnF50FjzHcK9HXDBtSJS5wA3fkcRYymJe0o6WMWNdgSRVpoSiWaHHmFgdMUJaYoCfhXzyl7LtNb3Q+Sveg+tJK7JaRXBLMUllOlJ6ll5Hod root@localhost"
home = "/home/admin/"
groups = ["wheel"]
4. Run rpm-ostree upgrade on VM
5. SSH to VM

Actual results:
Can't SSH to VM

Expected results:
SSH to VM without any error

Additional info:

Comment 4 Xiaofeng Wang 2021-08-31 15:04:32 UTC

Verified on the following builds:
osbuild-composer-worker-33-1.el8.x86_64
osbuild-35-1.el8.noarch
osbuild-composer-core-33-1.el8.x86_64
osbuild-selinux-35-1.el8.noarch
osbuild-composer-33-1.el8.x86_64
osbuild-ostree-35-1.el8.noarch
python3-osbuild-35-1.el8.noarch

Comment 6 errata-xmlrpc 2021-11-09 18:46:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (osbuild bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:4273

Note You need to log in before you can comment on or make changes to this bug.