Bug 2181466

Summary:	[RHEL-8.9][AWS]nm-cloud-setup fail to detect EC2 provider(NetworkManager-cloud-setup)
Product:	Red Hat Enterprise Linux 8	Reporter:	libhe
Component:	NetworkManager	Assignee:	Lubomir Rintel <lrintel>
Status:	VERIFIED ---	QA Contact:	David Jaša <djasa>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	8.9	CC:	bgalvani, linl, lrintel, rkhan, sfaye, sukulkar, thaller, till, vbenes, xiliang, ymao
Target Milestone:	rc	Keywords:	CustomerScenariosInitiative, Regression, Triaged
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:	NetworkManager-1.40.16-3.el8	Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description libhe 2023-03-24 08:07:58 UTC

Description of problem:

nm-cloud-setup service fails to detect ec2 provider. As a result, it fails to get aws metadata.

Mar 24 07:21:40 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: nm-cloud-setup.service: Succeeded.
Mar 24 07:21:40 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Started Automatically configure NetworkManager in cloud.
Mar 24 07:25:59 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Starting Automatically configure NetworkManager in cloud.>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08008] <debug> nm-cloud-setup 1.40.16-2.e>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08022] <debug> start detecting ec2 provid>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <debug> start detecting GCP provid>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <debug> start detecting azure prov>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <debug> start detecting aliyun pro>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <info>  provider GCP not detected:>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <info>  provider azure not detecte>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08023] <info>  provider aliyun not detect>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08024] <debug> http-request[25f12258e7e5a>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08031] <debug> http-request[25f12258e7e5a>
Mar 24 07:26:00 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1059.08032] <trace> http-request[25f12258e7e5a>
Mar 24 07:26:01 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1060.08032] <debug> http-request[b4d3361768536>
Mar 24 07:26:01 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1060.08040] <debug> http-request[b4d3361768536>
Mar 24 07:26:01 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1060.08040] <trace> http-request[b4d3361768536>
Mar 24 07:26:02 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1061.08040] <debug> http-request[bb8154ac4c1c6>
Mar 24 07:26:02 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1061.08047] <debug> http-request[bb8154ac4c1c6>
Mar 24 07:26:02 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1061.08048] <trace> http-request[bb8154ac4c1c6>
Mar 24 07:26:03 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1062.08058] <debug> http-request[ed734c9388351>
Mar 24 07:26:03 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1062.08065] <debug> http-request[ed734c9388351>
Mar 24 07:26:03 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1062.08066] <trace> http-request[ed734c9388351>
Mar 24 07:26:04 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1063.08066] <debug> http-request[fca6fe93ca38f>
Mar 24 07:26:04 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1063.08074] <debug> http-request[fca6fe93ca38f>
Mar 24 07:26:04 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1063.08074] <trace> http-request[fca6fe93ca38f>
Mar 24 07:26:05 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1064.08074] <debug> http-request[7b7cd4b5f311b>
Mar 24 07:26:05 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1064.08083] <debug> http-request[7b7cd4b5f311b>
Mar 24 07:26:05 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1064.08083] <trace> http-request[7b7cd4b5f311b>
Mar 24 07:26:06 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1065.08084] <debug> http-request[4d26bb6eb94e9>
Mar 24 07:26:06 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1065.08091] <debug> http-request[4d26bb6eb94e9>
Mar 24 07:26:06 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1065.08092] <trace> http-request[4d26bb6eb94e9>
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  provider ec2 not detected:>
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  no provider detected
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: nm-cloud-setup.service: Succeeded.
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Started Automatically configure NetworkManager in cloud.
Mar 24 07:31:18 ip-10-0-24-62.us-west-2.compute.internal systemd[1]: Starting Automatically configure NetworkManager in cloud.>

RHEL Version:
RHEL-8.9(4.18.0-481.el8.aarch64)
NetworkManager-cloud-setup-1.40.16-2.el8.aarch64

How reproducible:
100%

Steps to Reproduce:

1. Launch an aws instance with the latest RHEL-8.9 AMI
2. Enable trace " Environment=NM_CLOUD_SETUP_LOG=TRACE" for nm-cloud-setup service
3. Restart nm-cloud-setup service
4. Check journal log 


Actual results:
1.Journal log indicates that ec2 provider not detected 
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  provider ec2 not detected:>
Mar 24 07:26:07 ip-10-0-24-62.us-west-2.compute.internal nm-cloud-setup[7107]: [1066.08042] <info>  no provider detected
2.As a result, when user add the second IP from console, RHEL cannot get it from metadata.


Expected results:
nm-cloud-setup is able to detect ec2 provider and get EC2 metadata.

Additional info:
- No such issue in RHEL-8.8 with NetworkManager-cloud-setup-1.40.16-1.el8.aarch64
- It seems that it is introduced by the latest fix - cloud-setup/ec2: add IMDSv2 support (rh #2151986)

Comment 1 Thomas Haller 2023-03-27 08:39:13 UTC

bug 2179718 is the same issue for rhel-9.3

Comment 2 Thomas Haller 2023-04-06 07:34:32 UTC

fixed on upstream nm-1-40 branch by https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/6abbdaaa64bad00acb65ba7049dcea6bb7ae1084

Comment 9 David Jaša 2023-05-01 19:37:11 UTC

VERIFIED in NetworkManager-1.40.16-4.el8. As https://bugzilla.redhat.com/show_bug.cgi?id=2151987#c11 shows, PUT method is correctly used for getting the token.