Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2132570

Summary: Addresses configured at different order than specified at state
Product: Red Hat Enterprise Linux 8 Reporter: Quique Llorente <ellorent>
Component: nmstateAssignee: Gris Ge <fge>
Status: CLOSED ERRATA QA Contact: Mingyu Shi <mshi>
Severity: urgent Docs Contact:
Priority: high    
Version: 8.6CC: ferferna, fge, jiji, jishi, network-qe, rbobek, sfaye, till
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: nmstate-1.4.0-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2149048 2149049 2149050 (view as bug list) Environment:
Last Closed: 2023-05-16 08:26:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2149048, 2149049, 2149050    

Description Quique Llorente 2022-10-06 07:48:47 UTC 
Description of problem:
When multiple addresses are configured for an interface the order in witch they get added is different than the specified on the state

Doing same operation with "nmcli" and "ip" commands keep the order fine.

Version-Release number of selected component (if applicable): 
nmstate-1.0.2-19.el8_4.noarch


How reproducible: Always


Steps to Reproduce:
1. 

cat <<EOF | nmstatectl apply
interfaces:
  - name: dummy0
    type: dummy
    state: up 
    ipv4:
      enabled: true
      dhcp: false
      address:
        - ip: 10.147.216.2
          prefix-length: 32
        - ip: 10.147.216.11
          prefix-length: 32
EOF

2. ip a show dev dummy0

Actual results:

.11 address is before .2 


Expected results:

.2 address should be before .11


Additional info:
Comment 1 Roman Bobek 2022-11-02 10:31:09 UTC 
Hello @fge 

can we have some update on this one, pls? When can we expect it will be triaged?

Many thanks,


-Roman
Comment 2 Gris Ge 2022-11-02 12:49:25 UTC 
Sorry, it slipped through my TODO list.

Working on it now.
Comment 3 Gris Ge 2022-11-02 13:29:17 UTC 
I have created a scratch build:

https://people.redhat.com/fge/bz_2132570/nmstate-1.0.2-20.bz2132570.el8_4.noarch.rpm
https://people.redhat.com/fge/bz_2132570/python3-libnmstate-1.0.2-20.bz2132570.el8_4.noarch.rpm

Could you give it a try?
Comment 5 Gris Ge 2022-11-02 14:11:59 UTC 
In RHEL 8.4, the NetworkManager-1.30.0-16.el8_4 has bug on preserving IPv6 address order(is already fixed in upstream 1.41). If zstream is requested, NM bug is also required.
Comment 6 Gris Ge 2022-11-02 14:37:21 UTC 
The IPv6 address order issue in NetworkManager in RHEL 8 is by design: https://bugzilla.redhat.com/show_bug.cgi?id=2139443#c1


Let me hide the detail in nmstate to provide consistent way on ordering the IP address among IPv4 and IPv6.
Comment 7 Gris Ge 2022-11-03 00:18:59 UTC 
Patch sent to upstream: https://github.com/nmstate/nmstate/pull/2080


RHEL 8.6 scratch build send to test feedback.
Comment 8 Roman Bobek 2022-11-11 12:18:20 UTC 
Hello @fge 

I'm adding the request to include this fix into RHEL 8.4 zstream as the customer needs to have it in CoreOS under OpenShift 4.10.

Please, let me know in case you need more information from my side.

Best regards,


-Roman
Comment 15 Mingyu Shi 2022-12-07 03:22:21 UTC 
Hi Gris, could you please take a look, the IP addresses are not sort by order in the desired state when changing the IP:

cat << EOF > ip-order.yaml
interfaces:
- name: veth0
  type: veth
  state: up
  veth:
    peer: veth0_p
  ipv4:
    enabled: true
    address:
    - ip: 192.168.200.5
      prefix-length: 24
    - ip: 192.168.19.4
      prefix-length: 24
    - ip: 192.168.199.3
      prefix-length: 24
  ipv6:
    enabled: true
    address:
    - ip: 192:168:200::5
      prefix-length: 64
    - ip: 192:168:19::4
      prefix-length: 64
    - ip: 192:168:199::3
      prefix-length: 64
EOF

nmstatectl apply ip-order.yaml
ip addr show veth0
sed 's/200/15/g' ip-order.yaml | nmstatectl apply
ip addr show veth0


The 1st time order, 200-19-199, which is following the order in the YAML file:
845: veth0@veth0_p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:4b:45:1f:89:1b brd ff:ff:ff:ff:ff:ff
    inet 192.168.200.5/24 brd 192.168.200.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.19.4/24 brd 192.168.19.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.199.3/24 brd 192.168.199.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet6 192:168:200::5/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 192:168:19::4/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 192:168:199::3/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::944b:45ff:fe1f:891b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

The 2nd time order(after replace 200 in YAML file to 15), 19-199-15, which is not sort by order of the desired state(15-19-199)
[11:07:33@dell-per740-66 ~]0# ip addr show veth0
845: veth0@veth0_p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:4b:45:1f:89:1b brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.4/24 brd 192.168.19.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.199.3/24 brd 192.168.199.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.15.5/24 brd 192.168.15.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet6 192:168:15::5/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 192:168:19::4/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 192:168:199::3/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::944b:45ff:fe1f:891b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever


I tested on RHEL 8.7(nmstate-1.3.3-3.el8_7.x86_64) and RHEL 8.6(nmstate-1.2.1-9.el8_6.x86_64), got the same result.
Comment 16 Mingyu Shi 2022-12-07 03:37:07 UTC 
If apply such state instead of "sed 's/200/15/g' ip-order.yaml | nmstatectl apply" step:
interfaces:
- name: veth0
  type: veth
  state: up
  veth:
    peer: veth0_p
  ipv4:
    enabled: true
    address:
    - ip: 192.168.111.5
      prefix-length: 24
    - ip: 192.168.100.5
      prefix-length: 24
    - ip: 192.168.200.5
      prefix-length: 24
    - ip: 192.168.19.4
      prefix-length: 24
    - ip: 192.168.222.4
      prefix-length: 24
    - ip: 192.168.199.3
      prefix-length: 24
  ipv6:
    enabled: true
    address:
    - ip: 192:168:200::5
      prefix-length: 64
    - ip: 192:168:19::4
      prefix-length: 64
    - ip: 192:168:199::3
      prefix-length: 64

The result would be:
[11:27:26@dell-per740-66 ~]0# ip addr show veth0
848: veth0@veth0_p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:2f:76:17:ab:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.19.4/24 brd 192.168.19.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.199.3/24 brd 192.168.199.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.200.5/24 brd 192.168.200.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.111.5/24 brd 192.168.111.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.100.5/24 brd 192.168.100.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet 192.168.222.4/24 brd 192.168.222.255 scope global noprefixroute veth0
       valid_lft forever preferred_lft forever
    inet6 192:168:200::5/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 192:168:19::4/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 192:168:199::3/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::f42f:76ff:fe17:ab00/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

The newer IPs were appended.
Comment 17 Mingyu Shi 2022-12-07 03:39:03 UTC 
As Gris suggested, to deactive and then active "veth0" NM connection got the correct order
Comment 18 Gris Ge 2022-12-07 04:06:50 UTC 
The issue above will be fixed by NetworkManager via https://bugzilla.redhat.com/show_bug.cgi?id=2151426

RHEL 8.4 zstream reqeust also submitted there.
Comment 19 Mingyu Shi 2022-12-15 03:17:59 UTC 
Verified with nmstate-1.4.0-1.el8
http://lab-02.rhts.eng.brq.redhat.com/beaker/logs/results/717631+/717631313/resultoutputfile.log
Comment 20 Gris Ge 2022-12-20 10:05:06 UTC 
The order of IPv4 address on reapply turn out to be not a bug but limitation of kernel(no API for inserting IP address to certain index).

Will fix nmstate to ignore the IPv4 address on verification.
Comment 22 Gris Ge 2022-12-21 06:45:46 UTC 
Hi Mingyu,

No extra patch required for nmstate. For the IPv4 address missorder after reapply(change existing interface) is caused by limitation of kernel API on IPv4. Since the order between subnet does not actually matters in any use case, I would suggest you to test on IP address ordering in the same subnet especially the `secondary` flag is corrected added to IPv4 address. For IPv6 address, they should be strictly ordered.
Comment 24 Mingyu Shi 2022-12-22 02:11:18 UTC 
Used one subnet, all passed.
Comment 26 errata-xmlrpc 2023-05-16 08:26:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:2772