RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2079849 - [Aliyun] The primary ip address will be changed after assigning multiple secondary ip addresses from the web console in Alibaba Cloud [rhel-9.1]
Summary: [Aliyun] The primary ip address will be changed after assigning multiple seco...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: NetworkManager
Version: 9.0
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Thomas Haller
QA Contact: Vladimir Benes
Mayur Patil
URL:
Whiteboard:
Depends On:
Blocks: 2082000
TreeView+ depends on / blocked
 
Reported: 2022-04-28 11:32 UTC by YongkuiGuo
Modified: 2022-12-27 03:02 UTC (History)
17 users (show)

Fixed In Version: NetworkManager-1.39.5-1.el9
Doc Type: Bug Fix
Doc Text:
.An instance now retains the primary IP address even after starting the `nm-cloud-setup` service in Alibaba Cloud Previously, after launching an instance in the Alibaba Cloud, the `nm-cloud-setup` service configured the incorrect IP address as the primary IP address in case of multiple IPv4 addresses. Consequently, this affected the selection of the IPv4 source address for outgoing connections. With this update, after configuring secondary IP addresses manually, the `NetworkManager` package fetches the primary IP address from `primary-ip-address` metadata and configures both primary and secondary IP addresses correctly.
Clone Of:
: 2082000 (view as bug list)
Environment:
Last Closed: 2022-11-15 10:49:42 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-120386 0 None None None 2022-04-28 15:27:28 UTC
Red Hat Product Errata RHBA-2022:8265 0 None None None 2022-11-15 10:49:58 UTC
freedesktop.org Gitlab NetworkManager NetworkManager merge_requests 1201 0 None closed [th/cloud-setup-aliyun-primary-ip] cloud-setup: reorder addresses to honor "primary_ip_address" 2023-07-05 12:46:21 UTC

Description YongkuiGuo 2022-04-28 11:32:53 UTC 
Description of problem:
I tried to verify whether nm-cloud-setup service can work as expected in Alibaba Cloud. After assigning multiple secondary ip addresses to the instance and starting the nm-cloud-setup service,  the primary ip address has been changed. One of the secondary ip became the primary ip. 


How reproducible:
100%


Steps to Reproduce:

1. Create an instance with RHEL-9.0.0-20220424.1 guest image in Alibaba Cloud
2. Install NetworkManger-cloud-setup package and edit nm-cloud-setup service
# systemctl edit nm-cloud-setup.service
[Service]
Environment=NM_CLOUD_SETUP_ALIYUN=yes

3. 
# systemctl daemon-reload 
4. Check the primary ip address
# ip addr
...
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:0a:a6:a4 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 172.17.133.234/20 brd 172.17.143.255 scope global dynamic noprefixroute eth0
       valid_lft 315359817sec preferred_lft 315359817sec
    inet6 fe80::216:3eff:fe0a:a6a4/64 scope link
       valid_lft forever preferred_lft forever

The primary ip is '172.17.133.234'.

5. Assign three secondary ip addresses ('172.17.134.0', '172.17.134.1', '172.17.134.2') to the instance from the web console
6. Check the ip addresses from meta-data
# curl http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/private-ipv4s
["172.17.134.2","172.17.133.234","172.17.134.0","172.17.134.1"]

Notes: the first ip address of the above result is '172.17.134.2', which is a secondary ip not primary ip.

7. Start the nm-cloud-setup service
# systemctl start nm-cloud-setup.service

8. Check the ip addresses again
# ip addr
...
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:0a:a6:a4 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 172.17.134.2/20 brd 172.17.143.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.17.133.234/20 brd 172.17.143.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.17.134.0/20 brd 172.17.143.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.17.134.1/20 brd 172.17.143.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
 
The primary ip address has been changed from '172.17.133.234' to '172.17.134.2'. 



Actual results:
As above


Expected results:
The primary ip address remains the same.


Additional info:
1. There is no guarantee that the first ip address of returned list from private-ipv4s meta-data is the primary ip. nm-cloud-setup service can get the primary ip address from the below primary-ip-address meta-data:
# curl http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/primary-ip-address
172.17.133.234
2. The same issue with RHEL-8.6
Comment 1 YongkuiGuo 2022-04-28 11:40:41 UTC 
Part of journalctl output:
...
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08828] <debug> http-request[7721c640778b1bbd, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/private-ipv4s"]: start get ...
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08828] <debug> http-request[ba21600bd2fda341, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/netmask"]: start get ...
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08828] <debug> http-request[2e30688e2dbef205, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/gateway"]: start get ...
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08898] <debug> http-request[e49f34b1406edfcc, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/vpc-cidr-block"]: success getting 13 bytes (response code 200)
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08899] <trace> http-request[e49f34b1406edfcc, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/vpc-cidr-block"]: received 13 bytes: [[172.17.0.0/16]]
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08907] <debug> http-request[ba21600bd2fda341, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/netmask"]: success getting 13 bytes (response code 200)
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08907] <trace> http-request[ba21600bd2fda341, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/netmask"]: received 13 bytes: [[255.255.240.0]]
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08914] <debug> http-request[2e30688e2dbef205, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/gateway"]: success getting 14 bytes (response code 200)
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08914] <trace> http-request[2e30688e2dbef205, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/gateway"]: received 14 bytes: [[172.17.143.253]]
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08931] <debug> http-request[7721c640778b1bbd, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/private-ipv4s"]: success getting 63 bytes (response code 200)
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08932] <trace> http-request[7721c640778b1bbd, "http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:0a:a6:a4/private-ipv4s"]: received 63 bytes: [[["172.17.134.2","172.17.133.234","172.17.134.0","172.17.134.1"]]]
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08932] <debug> get-config: success
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08932] <debug> meta data received
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08932] <debug> config device 00:16:3E:0A:A6:A4: configuring "eth0" (/org/freedesktop/NetworkManager/Devices/2)...
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08943] <debug> config device 00:16:3E:0A:A6:A4: reapply connection "System eth0" (5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03)
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ NetworkManager[729]: <info>  [1651143863.6935] audit: op="device-reapply" interface="eth0" ifindex=2 args="ipv4.addresses,ipv4.routing-rules,ipv4.routes" pid=1507 uid=0 result="success"
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08965] <debug> config device 00:16:3E:0A:A6:A4: connection "System eth0" (5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03) reapplied
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ nm-cloud-setup[1507]: [1389.08966] <info>  some changes were applied for provider aliyun
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ systemd[1]: nm-cloud-setup.service: Deactivated successfully.
Apr 28 07:04:23 iZ2ze6n8cezih0l86auvoxZ systemd[1]: Finished Automatically configure NetworkManager in cloud.
...
Comment 2 Thomas Haller 2022-04-28 15:08:58 UTC 
thanks for the helpful report.
Comment 3 Thomas Haller 2022-04-28 15:13:07 UTC 
would you be able to test a scratch build?

On top of which NM build would you like to get the fix?
Were you testing latest rhel-9.0 package, that is "NetworkManager-1.36.0-4.el9_0"?
Comment 4 YongkuiGuo 2022-04-29 02:09:02 UTC 
(In reply to Thomas Haller from comment #3)
> would you be able to test a scratch build?
> 

Yes.  

> On top of which NM build would you like to get the fix?
> Were you testing latest rhel-9.0 package, that is
> "NetworkManager-1.36.0-4.el9_0"?

Yeah, I tested the latest rhel-9.0 package 'NetworkManager-1.36.0-4.el9_0.x86_64'. I think we can make the fix available in RHEL-9.1 since there is no requirement from Alibaba for the time being. This issue also exists in RHEL8, so could you please clone this bug for RHEL-8.7? Thanks a lot.
Comment 5 Thomas Haller 2022-04-29 13:29:04 UTC 
hi. Would you be able to test this scratch [1]

[1] build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44969253

I'd be interested in complete debug logs.

For nm-cloud-setup.service, you already correctly enabled level <trace>. Please keep doing that.
For NetworkManager, add `level=TRACE` in NetworkManager.conf and restart. See [2].
Then reproduce and provide the complete journal output.


[2] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/main/contrib/fedora/rpm/NetworkManager.conf#L27


Thank you!!
Comment 6 Mayur Patil 2022-05-02 13:11:59 UTC 
Hi Yongkui Guo,

Can you please provide doc text so that I can include Release Note for this known issue for our customers?

https://source.redhat.com/groups/public/ccs/rheldocslandingpage/rhel_documentation_master_page_wiki/how_to_provide_information_for_documentation_through_bugzilla_jira_confluence_or_customer_portal#jive_content_id_How_to_provide_information_for_a_Release_Note_in_Bugzilla_or_JIRA

Thanks!
Comment 8 YongkuiGuo 2022-05-05 05:18:01 UTC 
(In reply to Thomas Haller from comment #5)
> hi. Would you be able to test this scratch [1]
> 
> [1] build:
> https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=44969253
> 
> I'd be interested in complete debug logs.
> 
> For nm-cloud-setup.service, you already correctly enabled level <trace>.
> Please keep doing that.
> For NetworkManager, add `level=TRACE` in NetworkManager.conf and restart.
> See [2].
> Then reproduce and provide the complete journal output.
> 
> 
> [2]
> https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/main/
> contrib/fedora/rpm/NetworkManager.conf#L27
> 
> 

Hi,Thomas

Sorry for the late response, just came back from the May Day holiday. I tried the scratch build, and it works fine. Please check the entire journalctl log in the attachment. Thanks.
Comment 9 Thomas Haller 2022-05-05 06:41:47 UTC 
Thank you for testing (and the very good report in the first place). That is very helpful.

Fixed upstream (https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1201).



Setting as "Tested", due to comment 8 and review of the provided logs (which lgtm).
Comment 10 YongkuiGuo 2022-05-05 07:00:58 UTC 
Hi Thomas,

How about involving the fix in RHEL8? Could you help clone this bug for RHEL-8.7? Thanks in advance.
Comment 21 YongkuiGuo 2022-05-25 03:11:56 UTC 
Verified with NetworkManager-1.39.5-1.el9:


Steps:


1. Create an instance with RHEL-9.1.0-20220524.0 guest image in Alibaba Cloud

2. Install the NetworkManger-cloud-setup package and edit the nm-cloud-setup service
# systemctl edit nm-cloud-setup.service
[Service]
Environment=NM_CLOUD_SETUP_ALIYUN=yes

3. 
# systemctl daemon-reload 

4. Check the primary ip address
# ip addr
...
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:36:64:fe brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 172.17.96.132/20 brd 172.17.111.255 scope global dynamic noprefixroute eth0
       valid_lft 315359357sec preferred_lft 315359357sec
    inet6 fe80::216:3eff:fe36:64fe/64 scope link 
       valid_lft forever preferred_lft forever

The primary ip is '172.17.96.132'.

5. Assign three secondary ip addresses to the instance from the web console and check all the ip addresses from meta-data
# curl http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:36:64:fe/private-ipv4s
["172.17.96.134","172.17.96.133","172.17.96.132","172.17.96.135"]

6. Start the nm-cloud-setup service
# systemctl start nm-cloud-setup.service

7. Check the ip addresses again
# ip addr
...
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:36:64:fe brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 172.17.96.132/20 brd 172.17.111.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.17.96.134/20 brd 172.17.111.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.17.96.133/20 brd 172.17.111.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 172.17.96.135/20 brd 172.17.111.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever

The primary ip is still '172.17.96.132', and all the secondary ip addresses have been configured correctly.

8. Unassign all the secondary ip addresses and check all the ip addresses from meta-data
#  curl http://100.100.100.200/2016-01-01/meta-data/network/interfaces/macs/00:16:3e:36:64:fe/private-ipv4s
["172.17.96.132"]

9.
# systemctl start nm-cloud-setup.service

10.
# ip addr
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:36:64:fe brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 172.17.96.132/20 brd 172.17.111.255 scope global dynamic noprefixroute eth0
       valid_lft 315359998sec preferred_lft 315359998sec

All the configurations of the secondary ip addresses have been removed.

11. Enable and start nm-cloud-setup service and nm-cloud-setup timer
# systemctl enable --now nm-cloud-setup.service
# systemctl enable --now nm-cloud-setup.timer

12. Retest steps 5 and 8, the secondary ip addresses can be configured automatically.
Comment 33 errata-xmlrpc 2022-11-15 10:49:42 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8265
Note You need to log in before you can comment on or make changes to this bug.