Bug 2002492

Summary: util.py[WARNING]: Failed generating key type rsa to file /etc/ssh/ssh_host_rsa_key
Product: Red Hat Enterprise Linux 9 Reporter: Frank Liang <xiliang>
Component: cloud-initAssignee: Emanuele Giuseppe Esposito <eesposit>
Status: CLOSED ERRATA QA Contact: xiachen
Severity: medium Docs Contact:
Priority: medium    
Version: 9.0CC: eesposit, eterrell, huzhao, jgreguske, linl, mrezanin, vkuznets, wshi, xiachen, xiliang, yacao, ymao
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: cloud-init-21.1-10.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-17 12:26:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
os_tests.tests.test_cloud_init.TestCloudInit.test_check_cloudinit_log_traceback.debug none

Description Frank Liang 2021-09-09 01:48:34 UTC
Created attachment 1821670 [details]
os_tests.tests.test_cloud_init.TestCloudInit.test_check_cloudinit_log_traceback.debug

Description of problem:

Upload rhel-guest-image-9.0-20210906.6.aarch64 to the aws and found below warning in 'cloud-init.log'.

2021-09-08 08:42:18,295 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/i-076adaf0b0fd87d2a/sem/config_ssh - wb: [644] 24 bytes
2021-09-08 08:42:18,296 - util.py[DEBUG]: Restoring selinux mode for /var/lib/cloud/instances/i-076adaf0b0fd87d2a/sem/config_ssh (recursive=False)
2021-09-08 08:42:18,297 - util.py[DEBUG]: Restoring selinux mode for /var/lib/cloud/instances/i-076adaf0b0fd87d2a/sem/config_ssh (recursive=False)
2021-09-08 08:42:18,297 - helpers.py[DEBUG]: Running config-ssh using lock (<FileLock using file '/var/lib/cloud/instances/i-076adaf0b0fd87d2a/sem/config_ssh'>)
2021-09-08 08:42:18,298 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key
2021-09-08 08:42:18,298 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key.pub
2021-09-08 08:42:18,298 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key
2021-09-08 08:42:18,298 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key.pub
2021-09-08 08:42:18,298 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'rsa', '-N', '', '-f', '/etc/ssh/ssh_host_rsa_key'] with allowed return codes [0] (shell=False, capture=True)
2021-09-08 08:42:19,551 - util.py[WARNING]: Failed generating key type rsa to file /etc/ssh/ssh_host_rsa_key
2021-09-08 08:42:19,552 - util.py[DEBUG]: Failed generating key type rsa to file /etc/ssh/ssh_host_rsa_key
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/cloudinit/config/cc_ssh.py", line 241, in handle
    out, err = subp.subp(cmd, capture=True, env=lang_c)
  File "/usr/lib/python3.9/site-packages/cloudinit/subp.py", line 293, in subp
    raise ProcessExecutionError(stdout=out, stderr=err,
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['ssh-keygen', '-t', 'rsa', '-N', '', '-f', '/etc/ssh/ssh_host_rsa_key']
Exit code: 1
Reason: -
Stdout: Generating public/private rsa key pair.
        /etc/ssh/ssh_host_rsa_key already exists.
        Overwrite (y/n)?
Stderr:
2021-09-08 08:42:19,666 - util.py[DEBUG]: Restoring selinux mode for /etc/ssh (recursive=True)
2021-09-08 08:42:19,679 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ecdsa', '-N', '', '-f', '/etc/ssh/ssh_host_ecdsa_key'] with allowed return codes [0] (shell=False, capture=True)
2021-09-08 08:42:19,687 - util.py[DEBUG]: Restoring selinux mode for /etc/ssh (recursive=True)
2021-09-08 08:42:19,692 - subp.py[DEBUG]: Running command ['ssh-keygen', '-t', 'ed25519', '-N', '', '-f', '/etc/ssh/ssh_host_ed25519_key'] with allowed return codes [0] (shell=False, capture=True)
2021-09-08 08:42:19,700 - util.py[DEBUG]: Restoring selinux mode for /etc/ssh (recursive=True)
2021-09-08 08:42:19,706 - util.py[DEBUG]: Reading from /etc/ssh/ssh_host_rsa_key.pub (quiet=False)

 #rpm -q cloud-init
cloud-init-21.1-7.el9.noarch

RHEL Version:
RHEL-9.0(5.14.0-1.el9.aarch64)

How reproducible:
100%

Steps to Reproduce:
1. start a rhel9 t4g.small
2. check cloud-init.log

Actual results:
Failed generating key type rsa to file /etc/ssh/ssh_host_rsa_key

Expected results:
Can generate key type rsa to file /etc/ssh/ssh_host_rsa_key

Additional info:
- N/A

Comment 2 xiachen 2021-09-09 03:32:53 UTC
Additional Information from test on OpenStack.

This issue happened on OpenStack env once, not always reproduce on OpenStack.

$ rpm -q cloud-init
cloud-init-21.1-8.el9.noarch

RHEL version
Red Hat Enterprise Linux release 9.0 Beta (Plow)
5.14.0-1.el9.x86_64 

Same error log as bug description. 

From cloud-init.log, we can see there should be two lines about removing ssh_host_rsa_key files but on the instance that went wrong, it does not contain these logs.
- util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key
- util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key.pub

List files in /etc/ssh, we can see the group owner of ssh_host_rsa_key file is 'ssh_keys', not root.
$ ls -al /etc/ssh
total 608
drwxr-xr-x.  4 root root       4096 Sep  5 22:38 .
drwxr-xr-x. 93 root root       8192 Sep  5 22:38 ..
-rw-r--r--.  1 root root     570321 Aug 10 11:35 moduli
-rw-r--r--.  1 root root       1921 Aug 10 11:35 ssh_config
drwxr-xr-x.  2 root root         28 Sep  1 02:40 ssh_config.d
-rw-------.  1 root root        513 Sep  5 22:38 ssh_host_ecdsa_key
-rw-r--r--.  1 root root        186 Sep  5 22:38 ssh_host_ecdsa_key.pub
-rw-------.  1 root root        419 Sep  5 22:38 ssh_host_ed25519_key
-rw-r--r--.  1 root root        106 Sep  5 22:38 ssh_host_ed25519_key.pub
-rw-r-----.  1 root ssh_keys   2578 Sep  5 22:38 ssh_host_rsa_key
-rw-r--r--.  1 root root        554 Sep  5 22:38 ssh_host_rsa_key.pub
-rw-------.  1 root root       3702 Sep  5 22:38 sshd_config
drwx------.  2 root root         28 Sep  1 02:40 sshd_config.d

So the question is, why did it not attempt to remove ssh_host_rsa_key before generate it?

Comment 31 errata-xmlrpc 2022-05-17 12:26:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: cloud-init), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2308