Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2086856

Summary: [RHEL-9.1] SSH keys with \r\n line breaks are not properly handled on Azure
Product: Red Hat Enterprise Linux 9 Reporter: Chris Patterson <cpatterson>
Component: cloud-initAssignee: Miroslav Rezanina <mrezanin>
Status: CLOSED ERRATA QA Contact: Huijuan Zhao <huzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 9.0CC: anujmaurya, campbell.vertesi, eesposit, eterrell, huzhao, jgreguske, mamccoma, mheslin, mrezanin, toneata, vkuznets, xiachen, xiliang, yacao, yuxisun
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: cloud-init-22.1-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2088027 (view as bug list) Environment:
Last Closed: 2022-11-15 10:13:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2088027    

Description Chris Patterson 2022-05-16 16:58:28 UTC
Description of problem:

SSH keys with \r\n line breaks are not properly handled on Azure.  Discussed in upstream with Launchpad bug ID #1910835:
https://bugs.launchpad.net/cloud-init/+bug/1910835

Some relevant patches:

Fix regression with handling of IMDS ssh keys (#760) · canonical/cloud-init@4f62ae8 (github.com)
https://github.com/canonical/cloud-init/pull/760

Azure: Retrieve username and hostname from IMDS (#865) · canonical/cloud-init@431b368 (github.com)
https://github.com/canonical/cloud-init/pull/865

Add \r\n check for SSH keys in Azure (#889) · canonical/cloud-init@f17f78f (github.com)
https://github.com/canonical/cloud-init/pull/889

Version-Release number of selected component (if applicable):
cloud-init-21.1-19.el9.noarch

How reproducible:
100%

Steps to Reproduce:
1. Use Azure portal with generated SSH key
2. Attempt to SSH to VM with generated key.

Actual results: Fail to login with SSH key.

Expected results: Successful login with SSH key.

Comment 1 Mark Heslin 🎸 2022-05-16 20:15:18 UTC
Chris,

Can you provide detail on how the keys are being generated in Azure? From Azure CLI, Portal GUI? etc,
It seem like if the keys could be generated correctly from the start, that would be a simpler path
versus trying/waiting to get cloud-init updated. For example, generating the keys from a linux
host will not add any extra chars to it. It sounds like the keys are being generated from a Win-base?

Comment 2 Chris Patterson 2022-05-16 20:55:51 UTC
Mark, you are correct.  Keys generated on Linux hosts are OK as it is Windows' line endings that are problematic.  The portal/cli UIs allow the user to request that Azure generate a key-pair for them and have this format.

Comment 3 Eduardo Otubo 2022-05-17 07:59:56 UTC
(In reply to Mark Heslin from comment #1)
> Chris,
> 
> Can you provide detail on how the keys are being generated in Azure? From
> Azure CLI, Portal GUI? etc,
> It seem like if the keys could be generated correctly from the start, that
> would be a simpler path
> versus trying/waiting to get cloud-init updated. For example, generating the
> keys from a linux
> host will not add any extra chars to it. It sounds like the keys are being
> generated from a Win-base?

Based on Chris' comment (#2) would you still believe fixing how the keys are generated is faster than patching cloud-init?

Thanks!

Comment 4 Huijuan Zhao 2022-05-17 09:00:39 UTC
QE can reproduce this issue in rhel-8.6
No such issue in rhel-7.9

Per the https://bugs.launchpad.net/cloud-init/+bug/1910835, this is a regression from cloud-init 20.4, this issue should exist in rhel-8.5/8.6/9.0

Comment 5 anujmaurya 2022-05-17 09:18:57 UTC
yes, the issue exists with RHEL 8.6 and RHEL 9.0 since they are using cloud-init-21.1-15.el8.src.rpm and 21.1-19.el9 respectively.

Since we published RHEL 8.5 with cloud-init version: 21.1-7.el8_5.4, it is working as expected.

Comment 6 anujmaurya 2022-05-17 12:03:02 UTC
does this spec has all the patches required for fixing this ssh issue: https://git.centos.org/rpms/cloud-init/blob/c9/f/SPECS/cloud-init.spec?

For RHEL 8.6, can we downgrade the cloud-init version to 21.1-7.el8_5.5 and publish the image?

On RHEL 9, Since we don't have any other version, Is it possible for Redhat to build a new cloud-init rpm with the patches required for fixing this bug?

Comment 7 Campbell Vertesi 2022-05-17 14:05:21 UTC
For RHEL 9, we need cloud-init version 21.1 PLUS these two patches:

- https://github.com/canonical/cloud-init/commit/431b3683fce309e7d673f97c9cf6cc726463b9e0 - retrieve username and hostname from IMDS 
- https://github.com/canonical/cloud-init/commit/f17f78fa9d28e62793a5f2c7109fc29eeffb0c89 - add \r \n check for Azure

We could use the resulting RPM on both 8.6 and 9.


Separately we are working on fixing how keys are generated in Azure, but it's doubtful that this can happen in time for RHEL 9 release. Seems to me that taking two patches which are already in upstream could be faster.

Comment 8 Mark Heslin 🎸 2022-05-17 14:16:01 UTC
To re-iterate, Chris's original comment, this is a *regression* in the upstream cloud-init code. 

We are *blocked* from releasing RHEL 9 on Azure until this is resolved. Target Azure release date is Tue 5/24.

Comment 9 Miroslav Rezanina 2022-05-17 14:50:04 UTC
(In reply to Campbell Vertesi from comment #7)
> For RHEL 9, we need cloud-init version 21.1 PLUS these two patches:
> 
> -
> https://github.com/canonical/cloud-init/commit/
> 431b3683fce309e7d673f97c9cf6cc726463b9e0 - retrieve username and hostname
> from IMDS 
> -
> https://github.com/canonical/cloud-init/commit/
> f17f78fa9d28e62793a5f2c7109fc29eeffb0c89 - add \r \n check for Azure
> 
> We could use the resulting RPM on both 8.6 and 9.
> 
> 
> Separately we are working on fixing how keys are generated in Azure, but
> it's doubtful that this can happen in time for RHEL 9 release. Seems to me
> that taking two patches which are already in upstream could be faster.

We already have #865 commit in tree (retrieve username and hostname from IMDS), so we just need the second one.

Comment 15 Mark Heslin 🎸 2022-05-18 13:25:29 UTC
Thank you all for the quick turnaround. Anuj @ Microsoft confirms that this works. 

The Microsoft target date to make RHEL 9 available to Azure customers is next Tue 5/24.

   1. Do we know when the RHEL 9 repos will reflect the fix? Approximate ETA?
   2. Do we have any idea when the RHEL 8.6/7 repos will reflect this? Approximate ETA?

Anuj - any other questions?

Comment 16 Miroslav Rezanina 2022-05-18 16:06:13 UTC
Fix available in 9.1.0 through rebase.

Comment 18 anujmaurya 2022-05-19 06:48:31 UTC
hi @miroslav.popov

Comment 19 anujmaurya 2022-05-19 07:27:03 UTC
HI Miroslav,

>>Fix available in 9.1.0 through rebase.
Is the fix released?

We are facing another issue with SSH for which I have filed a different bug https://bugzilla.redhat.com/show_bug.cgi?id=2088298. Can you please check and suggest the config change?

Thanks,
Anuj Maurya

Comment 23 Huijuan Zhao 2022-05-19 10:20:13 UTC
Move to VERIFIED per comment 20

Comment 24 Miroslav Rezanina 2022-05-19 11:05:29 UTC
(In reply to anujmaurya from comment #19)
> HI Miroslav,
> 
> >>Fix available in 9.1.0 through rebase.
> Is the fix released?
> 
> We are facing another issue with SSH for which I have filed a different bug
> https://bugzilla.redhat.com/show_bug.cgi?id=2088298. Can you please check
> and suggest the config change?
> 
> Thanks,
> Anuj Maurya


Regarding fix:
This BZ is tracking fix for 9.1.0 release, 9.0.0 release tracking is done in BZ 2088027


Regarding ssh issue:
I would recommend to not use rsa keys as they are becomming unsupported. I'm afraid I'm not familiar with RHEL 9 openssl so you will need someone from openssl team to help whether it's possible to configure RHEL 9 openssl to accept used format of rsa keys or requirements on used keys has to be updated.

Comment 26 errata-xmlrpc 2022-11-15 10:13:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (cloud-init bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:8056