Bug 2215280

Summary:	Failed to apply DHCP along with static routes
Product:	Red Hat Enterprise Linux 9	Reporter:	Mingyu Shi <mshi>
Component:	nmstate	Assignee:	Gris Ge <fge>
Status:	CLOSED NOTABUG	QA Contact:	Mingyu Shi <mshi>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	9.3	CC:	ferferna, jiji, jishi, network-qe, sfaye, till
Target Milestone:	rc	Keywords:	Regression, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-07-12 07:34:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Mingyu Shi 2023-06-15 10:08:23 UTC

Description of problem:
This looks like a regression of https://bugzilla.redhat.com/show_bug.cgi?id=2169982
Nmstate cannot apply DHCP along with static routes(no matter "auto-routes=false" is set or not), while this works on RHEL 8 backport.

Version-Release number of selected component (if applicable):
nmstate-2.2.12-2.el9.x86_64
nispor-1.2.10-1.el9.x86_64
NetworkManager-1.43.9-1.el9.x86_64
DISTRO=RHEL-9.3.0-updates-20230611.28

How reproducible:
100%

Steps to Reproduce:
echo "
interfaces:
  - name: bond0
    type: bond
    state: up
    mtu: 9000
    mac-address: d4:ee:07:25:42:5a
    ipv4:
      enabled: true
      dhcp: true
    ipv6:
      enabled: true
      dhcp: true
      autoconf: true
    link-aggregation:
      mode: 802.3ad
      options:
        miimon: "150"
      port:
        - veth0
        - veth1
  - name: veth0
    type: veth
    state: up
    veth:
      peer: veth0_p
  - name: veth1
    type: veth
    state: up
    veth:
      peer: veth1_p
dns-resolver:
  config:
    server:
      - 8.8.8.8
      - 2400:3200:baba::1
routes:
  config:
    - destination: 0.0.0.0/0
      next-hop-address: 192.168.199.1
      next-hop-interface: bond0
      table-id: 200
    - destination: ::/0
      next-hop-address: 192:168:199::1
      next-hop-interface: bond0
      table-id: 200
" | nmstatectl apply

Actual results:
[2023-06-15T10:02:19Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/88
[2023-06-15T10:02:20Z WARN  nmstate::nm::query_apply::apply] Cannot store DNS to NetworkManager interface connection: InvalidArgument: Failed to find suitable(IP enabled with DHCP off or auto-dns: false) interface for DNS server ["2400:3200:baba::1"]
[2023-06-15T10:02:20Z WARN  nmstate::nm::query_apply::dns] Storing DNS to NetworkManager via global dns API, this will cause __all__ interface level DNS settings been ignored
[2023-06-15T10:02:20Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth0_p for veth0
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth0 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth0 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth1_p for veth1
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth1 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth1 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("dd9146ed-84dc-4124-a22d-637dd2a98687"), ID Some("bond0"), type Some("bond") name Some("bond0")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("654e6874-14cb-44ff-8b1c-6f3a6ebb5766"), ID Some("veth0"), type Some("veth") name Some("veth0")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("3bfbc04a-631f-47e7-a8e9-7246b6a07ba8"), ID Some("veth0_p"), type Some("veth") name Some("veth0_p")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("1e7c1a37-b9b8-457c-baaa-a95eb0909d2e"), ID Some("veth1"), type Some("veth") name Some("veth1")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("8592169a-9f39-43c4-aff2-0e6b3944e486"), ID Some("veth1_p"), type Some("veth") name Some("veth1_p")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Activating connection dd9146ed-84dc-4124-a22d-637dd2a98687: bond0/bond
[2023-06-15T10:02:20Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: bond0.interface.link-aggregation.port desire '["veth0","veth1"]', current '[]'
[2023-06-15T10:02:21Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:23Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:24Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:26Z WARN  nmstate::nm::query_apply::apply] Cannot store DNS to NetworkManager interface connection: InvalidArgument: Failed to find suitable(IP enabled with DHCP off or auto-dns: false) interface for DNS server ["2400:3200:baba::1"]
[2023-06-15T10:02:26Z WARN  nmstate::nm::query_apply::dns] Storing DNS to NetworkManager via global dns API, this will cause __all__ interface level DNS settings been ignored
[2023-06-15T10:02:26Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth0_p for veth0
[2023-06-15T10:02:26Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth1_p for veth1
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("dd9146ed-84dc-4124-a22d-637dd2a98687"), ID Some("bond0"), type Some("bond") name Some("bond0")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("654e6874-14cb-44ff-8b1c-6f3a6ebb5766"), ID Some("veth0"), type Some("veth") name Some("veth0")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("3bfbc04a-631f-47e7-a8e9-7246b6a07ba8"), ID Some("veth0_p"), type Some("veth") name Some("veth0_p")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("1e7c1a37-b9b8-457c-baaa-a95eb0909d2e"), ID Some("veth1"), type Some("veth") name Some("veth1")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("8592169a-9f39-43c4-aff2-0e6b3944e486"), ID Some("veth1_p"), type Some("veth") name Some("veth1_p")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Activating connection dd9146ed-84dc-4124-a22d-637dd2a98687: bond0/bond
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 654e6874-14cb-44ff-8b1c-6f3a6ebb5766: veth0/veth
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Activating connection 3bfbc04a-631f-47e7-a8e9-7246b6a07ba8: veth0_p/veth
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 1e7c1a37-b9b8-457c-baaa-a95eb0909d2e: veth1/veth
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Activating connection 8592169a-9f39-43c4-aff2-0e6b3944e486: veth1_p/veth
[2023-06-15T10:02:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:27Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:28Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:29Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:31Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/88
NmstateError: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply

Expected results:
No failure

Additional info:

Comment 1 Gris Ge 2023-06-27 13:19:36 UTC

Tried in my VM with 2.2.12 , works well there.

Could you try again and prepare a server for debug?

Thank you!

Comment 4 Gris Ge 2023-07-12 07:01:57 UTC

In RHEL 8, route been applied to interface *before* DHCP finished. In RHEL 9, route only applied to interface *after* DHCP finished.

For this bug, nmstate noticed desired route is never applied to bond0 because there is no DHCP server.

Still waiting NM developer to confirm this as a bug or expected behavior.

Comment 5 Gris Ge 2023-07-12 07:07:45 UTC

Got confirmation from NM developer as this is expected behavior in RHEL 9.


The static route will only been applied to kernel *after* DHCP finished, please check the detail at https://bugzilla.redhat.com/show_bug.cgi?id=2102212

Comment 6 Gris Ge 2023-07-12 07:34:15 UTC

Closing as this is expected behavior and got consent from reporter.

The static route on DHCP interface required a working DHCP server.