Bug 2185036

Summary:	cloud-init and host fails to generate ssh keys after cloud-init upgrade
Product:	Red Hat Enterprise Linux 8	Reporter:	Chris Patterson <cpatterson>
Component:	cloud-init	Assignee:	Virtualization Maintenance <virt-maint>
Status:	CLOSED NOTABUG	QA Contact:	xiachen
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.7	CC:	anisinha, eterrell, huzhao, jgreguske, xiachen, xiliang, yacao
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-05-04 07:02:03 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Chris Patterson 2023-04-06 15:07:36 UTC

Description of problem:

Updating to newest cloudinit on RHEL8 (from say 18.x) will install a systemd config to disable the old sshd-keygen services:
/etc/systemd/system/sshd-keygen@.service.d/disable-sshd-keygen-if-cloud-init-active.conf

When upgrading cloud-init, yum does not overwrite the existing /etc/cloud/cloud.cfg with the necessary config for ssh_genkeytypes.  It instead writes this config to /etc/cloud/cloud.cfg.rpmnew.

We can see the difference:
$ grep -r ssh_genkey /etc/cloud
/etc/cloud/cloud.cfg.rpmnew:ssh_genkeytypes:  ['rsa', 'ecdsa', 'ed25519']
/etc/cloud/cloud.cfg:ssh_genkeytypes:  ~

The solution for the customer is to update their cloud configs to the new defaults:
sudo mv /etc/cloud/cloud.cfg.rpmnew /etc/cloud/cloud.cfg
sudo mv /etc/cloud/cloud.cfg.d/05_logging.cfg.rpmnew /etc/cloud/cloud.cfg.d/05_logging.cfg

Some customers will create an image which then fails to generate SSH keys without understanding that they need to update the cloud-init config.

Version-Release number of selected component (if applicable):

cloud-init-18.5-1.el8.4 => cloud-init-22.1-6.el8_7.2

How reproducible:
100%

Steps to Reproduce:
1. Create RHEL 8.0 VM on Azure.
2. yum update cloud-init
3. rm -f /etc/ssh/ssh_host-*  [comparable to operation when creating image from VM]
4. cloud-init clean
5. reboot

Actual results:
sshd fails to start, neither host or cloud-init generate keys.

Expected results:
Keys are generated and sshd starts.

Comment 1 xiachen 2023-04-07 07:07:56 UTC

Hi Chris,

Thanks for raising this bug. 
It's a problem left over from history but it could not handled by code, and the solution is updating cloud.cfg manually.
so we created a KCS for the customers to follow, could you share it with customers?
https://access.redhat.com/solutions/6988034

If any more question, please feel free to let me know.

Amy

Comment 2 Ani Sinha 2023-05-02 16:22:02 UTC

@xiachen should we close this bug then?

Comment 3 xiachen 2023-05-04 07:02:03 UTC

This bug is going to closed, if any question, feel free to reopen it.