Description of problem: When multiple addresses are configured for an interface the order in witch they get added is different than the specified on the state Doing same operation with "nmcli" and "ip" commands keep the order fine. Version-Release number of selected component (if applicable): nmstate-1.0.2-19.el8_4.noarch How reproducible: Always Steps to Reproduce: 1. cat <<EOF | nmstatectl apply interfaces: - name: dummy0 type: dummy state: up ipv4: enabled: true dhcp: false address: - ip: 10.147.216.2 prefix-length: 32 - ip: 10.147.216.11 prefix-length: 32 EOF 2. ip a show dev dummy0 Actual results: .11 address is before .2 Expected results: .2 address should be before .11 Additional info:
Hello @fge can we have some update on this one, pls? When can we expect it will be triaged? Many thanks, -Roman
Sorry, it slipped through my TODO list. Working on it now.
I have created a scratch build: https://people.redhat.com/fge/bz_2132570/nmstate-1.0.2-20.bz2132570.el8_4.noarch.rpm https://people.redhat.com/fge/bz_2132570/python3-libnmstate-1.0.2-20.bz2132570.el8_4.noarch.rpm Could you give it a try?
In RHEL 8.4, the NetworkManager-1.30.0-16.el8_4 has bug on preserving IPv6 address order(is already fixed in upstream 1.41). If zstream is requested, NM bug is also required.
The IPv6 address order issue in NetworkManager in RHEL 8 is by design: https://bugzilla.redhat.com/show_bug.cgi?id=2139443#c1 Let me hide the detail in nmstate to provide consistent way on ordering the IP address among IPv4 and IPv6.
Patch sent to upstream: https://github.com/nmstate/nmstate/pull/2080 RHEL 8.6 scratch build send to test feedback.
Hello @fge I'm adding the request to include this fix into RHEL 8.4 zstream as the customer needs to have it in CoreOS under OpenShift 4.10. Please, let me know in case you need more information from my side. Best regards, -Roman
Hi Gris, could you please take a look, the IP addresses are not sort by order in the desired state when changing the IP: cat << EOF > ip-order.yaml interfaces: - name: veth0 type: veth state: up veth: peer: veth0_p ipv4: enabled: true address: - ip: 192.168.200.5 prefix-length: 24 - ip: 192.168.19.4 prefix-length: 24 - ip: 192.168.199.3 prefix-length: 24 ipv6: enabled: true address: - ip: 192:168:200::5 prefix-length: 64 - ip: 192:168:19::4 prefix-length: 64 - ip: 192:168:199::3 prefix-length: 64 EOF nmstatectl apply ip-order.yaml ip addr show veth0 sed 's/200/15/g' ip-order.yaml | nmstatectl apply ip addr show veth0 The 1st time order, 200-19-199, which is following the order in the YAML file: 845: veth0@veth0_p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 96:4b:45:1f:89:1b brd ff:ff:ff:ff:ff:ff inet 192.168.200.5/24 brd 192.168.200.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.19.4/24 brd 192.168.19.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.199.3/24 brd 192.168.199.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet6 192:168:200::5/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 192:168:19::4/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 192:168:199::3/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 fe80::944b:45ff:fe1f:891b/64 scope link noprefixroute valid_lft forever preferred_lft forever The 2nd time order(after replace 200 in YAML file to 15), 19-199-15, which is not sort by order of the desired state(15-19-199) [11:07:33@dell-per740-66 ~]0# ip addr show veth0 845: veth0@veth0_p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 96:4b:45:1f:89:1b brd ff:ff:ff:ff:ff:ff inet 192.168.19.4/24 brd 192.168.19.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.199.3/24 brd 192.168.199.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.15.5/24 brd 192.168.15.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet6 192:168:15::5/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 192:168:19::4/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 192:168:199::3/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 fe80::944b:45ff:fe1f:891b/64 scope link noprefixroute valid_lft forever preferred_lft forever I tested on RHEL 8.7(nmstate-1.3.3-3.el8_7.x86_64) and RHEL 8.6(nmstate-1.2.1-9.el8_6.x86_64), got the same result.
If apply such state instead of "sed 's/200/15/g' ip-order.yaml | nmstatectl apply" step: interfaces: - name: veth0 type: veth state: up veth: peer: veth0_p ipv4: enabled: true address: - ip: 192.168.111.5 prefix-length: 24 - ip: 192.168.100.5 prefix-length: 24 - ip: 192.168.200.5 prefix-length: 24 - ip: 192.168.19.4 prefix-length: 24 - ip: 192.168.222.4 prefix-length: 24 - ip: 192.168.199.3 prefix-length: 24 ipv6: enabled: true address: - ip: 192:168:200::5 prefix-length: 64 - ip: 192:168:19::4 prefix-length: 64 - ip: 192:168:199::3 prefix-length: 64 The result would be: [11:27:26@dell-per740-66 ~]0# ip addr show veth0 848: veth0@veth0_p: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether f6:2f:76:17:ab:00 brd ff:ff:ff:ff:ff:ff inet 192.168.19.4/24 brd 192.168.19.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.199.3/24 brd 192.168.199.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.200.5/24 brd 192.168.200.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.111.5/24 brd 192.168.111.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.100.5/24 brd 192.168.100.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet 192.168.222.4/24 brd 192.168.222.255 scope global noprefixroute veth0 valid_lft forever preferred_lft forever inet6 192:168:200::5/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 192:168:19::4/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 192:168:199::3/64 scope global noprefixroute valid_lft forever preferred_lft forever inet6 fe80::f42f:76ff:fe17:ab00/64 scope link noprefixroute valid_lft forever preferred_lft forever The newer IPs were appended.
As Gris suggested, to deactive and then active "veth0" NM connection got the correct order
The issue above will be fixed by NetworkManager via https://bugzilla.redhat.com/show_bug.cgi?id=2151426 RHEL 8.4 zstream reqeust also submitted there.
Verified with nmstate-1.4.0-1.el8 http://lab-02.rhts.eng.brq.redhat.com/beaker/logs/results/717631+/717631313/resultoutputfile.log
The order of IPv4 address on reapply turn out to be not a bug but limitation of kernel(no API for inserting IP address to certain index). Will fix nmstate to ignore the IPv4 address on verification.
Hi Mingyu, No extra patch required for nmstate. For the IPv4 address missorder after reapply(change existing interface) is caused by limitation of kernel API on IPv4. Since the order between subnet does not actually matters in any use case, I would suggest you to test on IP address ordering in the same subnet especially the `secondary` flag is corrected added to IPv4 address. For IPv6 address, they should be strictly ordered.
Used one subnet, all passed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:2772