Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2181466

Summary: [RHEL-8.9][AWS]nm-cloud-setup fail to detect EC2 provider(NetworkManager-cloud-setup)
Product: Red Hat Enterprise Linux 8 Reporter: libhe
Component: NetworkManagerAssignee: Lubomir Rintel <lrintel>
Status: CLOSED ERRATA QA Contact: David Jaša <djasa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.9CC: bgalvani, linl, lrintel, rkhan, sfaye, sukulkar, thaller, till, vbenes, xiliang, ymao
Target Milestone: rcKeywords: CustomerScenariosInitiative, Regression, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: NetworkManager-1.40.16-3.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-14 15:49:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description libhe 2023-03-24 08:07:58 UTC
Description of problem:

nm-cloud-setup service fails to detect ec2 provider. As a result, it fails to get aws metadata.

Mar 24 07:21:40 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: nm-cloud-setup.service: Succeeded.
Mar 24 07:21:40 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Started Automatically configure NetworkManager in cloud.
Mar 24 07:25:59 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Starting Automatically configure NetworkManager in cloud.>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08008] <debug> nm-cloud-setup 1.40.16-2.e>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08022] <debug> start detecting ec2 provid>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <debug> start detecting GCP provid>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <debug> start detecting azure prov>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <debug> start detecting aliyun pro>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <info>  provider GCP not detected:>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <info>  provider azure not detecte>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <info>  provider aliyun not detect>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08024] <debug> http-request[25f12258e7e5a>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08031] <debug> http-request[25f12258e7e5a>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08032] <trace> http-request[25f12258e7e5a>
Mar 24 07:26:01 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1060.08032] <debug> http-request[b4d3361768536>
Mar 24 07:26:01 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1060.08040] <debug> http-request[b4d3361768536>
Mar 24 07:26:01 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1060.08040] <trace> http-request[b4d3361768536>
Mar 24 07:26:02 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1061.08040] <debug> http-request[bb8154ac4c1c6>
Mar 24 07:26:02 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1061.08047] <debug> http-request[bb8154ac4c1c6>
Mar 24 07:26:02 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1061.08048] <trace> http-request[bb8154ac4c1c6>
Mar 24 07:26:03 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1062.08058] <debug> http-request[ed734c9388351>
Mar 24 07:26:03 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1062.08065] <debug> http-request[ed734c9388351>
Mar 24 07:26:03 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1062.08066] <trace> http-request[ed734c9388351>
Mar 24 07:26:04 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1063.08066] <debug> http-request[fca6fe93ca38f>
Mar 24 07:26:04 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1063.08074] <debug> http-request[fca6fe93ca38f>
Mar 24 07:26:04 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1063.08074] <trace> http-request[fca6fe93ca38f>
Mar 24 07:26:05 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1064.08074] <debug> http-request[7b7cd4b5f311b>
Mar 24 07:26:05 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1064.08083] <debug> http-request[7b7cd4b5f311b>
Mar 24 07:26:05 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1064.08083] <trace> http-request[7b7cd4b5f311b>
Mar 24 07:26:06 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1065.08084] <debug> http-request[4d26bb6eb94e9>
Mar 24 07:26:06 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1065.08091] <debug> http-request[4d26bb6eb94e9>
Mar 24 07:26:06 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1065.08092] <trace> http-request[4d26bb6eb94e9>
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  provider ec2 not detected:>
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  no provider detected
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: nm-cloud-setup.service: Succeeded.
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Started Automatically configure NetworkManager in cloud.
Mar 24 07:31:18 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Starting Automatically configure NetworkManager in cloud.>

RHEL Version:
RHEL-8.9(4.18.0-481.el8.aarch64)
NetworkManager-cloud-setup-1.40.16-2.el8.aarch64

How reproducible:
100%

Steps to Reproduce:

1. Launch an aws instance with the latest RHEL-8.9 AMI
2. Enable trace " Environment=NM_CLOUD_SETUP_LOG=TRACE" for nm-cloud-setup service
3. Restart nm-cloud-setup service
4. Check journal log 


Actual results:
1.Journal log indicates that ec2 provider not detected 
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  provider ec2 not detected:>
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  no provider detected
2.As a result, when user add the second IP from console, RHEL cannot get it from metadata.


Expected results:
nm-cloud-setup is able to detect ec2 provider and get EC2 metadata.

Additional info:
- No such issue in RHEL-8.8 with NetworkManager-cloud-setup-1.40.16-1.el8.aarch64
- It seems that it is introduced by the latest fix - cloud-setup/ec2: add IMDSv2 support (rh #2151986)

Comment 1 Thomas Haller 2023-03-27 08:39:13 UTC
bug 2179718 is the same issue for rhel-9.3

Comment 9 David Jaša 2023-05-01 19:37:11 UTC
VERIFIED in NetworkManager-1.40.16-4.el8. As https://bugzilla.redhat.com/show_bug.cgi?id=2151987#c11 shows, PUT method is correctly used for getting the token.

Comment 13 errata-xmlrpc 2023-11-14 15:49:35 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:7118