Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2215280

Summary: Failed to apply DHCP along with static routes
Product: Red Hat Enterprise Linux 9 Reporter: Mingyu Shi <mshi>
Component: nmstateAssignee: Gris Ge <fge>
Status: CLOSED NOTABUG QA Contact: Mingyu Shi <mshi>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.3CC: ferferna, jiji, jishi, network-qe, sfaye, till
Target Milestone: rcKeywords: Regression, Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-07-12 07:34:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Mingyu Shi 2023-06-15 10:08:23 UTC 
Description of problem:
This looks like a regression of https://bugzilla.redhat.com/show_bug.cgi?id=2169982
Nmstate cannot apply DHCP along with static routes(no matter "auto-routes=false" is set or not), while this works on RHEL 8 backport.

Version-Release number of selected component (if applicable):
nmstate-2.2.12-2.el9.x86_64
nispor-1.2.10-1.el9.x86_64
NetworkManager-1.43.9-1.el9.x86_64
DISTRO=RHEL-9.3.0-updates-20230611.28

How reproducible:
100%

Steps to Reproduce:
echo "
interfaces:
  - name: bond0
    type: bond
    state: up
    mtu: 9000
    mac-address: d4:ee:07:25:42:5a
    ipv4:
      enabled: true
      dhcp: true
    ipv6:
      enabled: true
      dhcp: true
      autoconf: true
    link-aggregation:
      mode: 802.3ad
      options:
        miimon: "150"
      port:
        - veth0
        - veth1
  - name: veth0
    type: veth
    state: up
    veth:
      peer: veth0_p
  - name: veth1
    type: veth
    state: up
    veth:
      peer: veth1_p
dns-resolver:
  config:
    server:
      - 8.8.8.8
      - 2400:3200:baba::1
routes:
  config:
    - destination: 0.0.0.0/0
      next-hop-address: 192.168.199.1
      next-hop-interface: bond0
      table-id: 200
    - destination: ::/0
      next-hop-address: 192:168:199::1
      next-hop-interface: bond0
      table-id: 200
" | nmstatectl apply

Actual results:
[2023-06-15T10:02:19Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/88
[2023-06-15T10:02:20Z WARN  nmstate::nm::query_apply::apply] Cannot store DNS to NetworkManager interface connection: InvalidArgument: Failed to find suitable(IP enabled with DHCP off or auto-dns: false) interface for DNS server ["2400:3200:baba::1"]
[2023-06-15T10:02:20Z WARN  nmstate::nm::query_apply::dns] Storing DNS to NetworkManager via global dns API, this will cause __all__ interface level DNS settings been ignored
[2023-06-15T10:02:20Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth0_p for veth0
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth0 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth0 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth1_p for veth1
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth1 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::profile] Skipping activation of veth1 as its controller bond0 will automatically activate it
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("dd9146ed-84dc-4124-a22d-637dd2a98687"), ID Some("bond0"), type Some("bond") name Some("bond0")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("654e6874-14cb-44ff-8b1c-6f3a6ebb5766"), ID Some("veth0"), type Some("veth") name Some("veth0")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("3bfbc04a-631f-47e7-a8e9-7246b6a07ba8"), ID Some("veth0_p"), type Some("veth") name Some("veth0_p")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("1e7c1a37-b9b8-457c-baaa-a95eb0909d2e"), ID Some("veth1"), type Some("veth") name Some("veth1")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Creating connection UUID Some("8592169a-9f39-43c4-aff2-0e6b3944e486"), ID Some("veth1_p"), type Some("veth") name Some("veth1_p")
[2023-06-15T10:02:20Z INFO  nmstate::nm::query_apply::profile] Activating connection dd9146ed-84dc-4124-a22d-637dd2a98687: bond0/bond
[2023-06-15T10:02:20Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: bond0.interface.link-aggregation.port desire '["veth0","veth1"]', current '[]'
[2023-06-15T10:02:21Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:23Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:24Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: ::/0 next-hop-interface: bond0 next-hop-address: 192:168:199::1 table-id: 200 not found after apply
[2023-06-15T10:02:26Z WARN  nmstate::nm::query_apply::apply] Cannot store DNS to NetworkManager interface connection: InvalidArgument: Failed to find suitable(IP enabled with DHCP off or auto-dns: false) interface for DNS server ["2400:3200:baba::1"]
[2023-06-15T10:02:26Z WARN  nmstate::nm::query_apply::dns] Storing DNS to NetworkManager via global dns API, this will cause __all__ interface level DNS settings been ignored
[2023-06-15T10:02:26Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth0_p for veth0
[2023-06-15T10:02:26Z INFO  nmstate::nm::settings::connection] Creating veth peer profile veth1_p for veth1
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("dd9146ed-84dc-4124-a22d-637dd2a98687"), ID Some("bond0"), type Some("bond") name Some("bond0")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("654e6874-14cb-44ff-8b1c-6f3a6ebb5766"), ID Some("veth0"), type Some("veth") name Some("veth0")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("3bfbc04a-631f-47e7-a8e9-7246b6a07ba8"), ID Some("veth0_p"), type Some("veth") name Some("veth0_p")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("1e7c1a37-b9b8-457c-baaa-a95eb0909d2e"), ID Some("veth1"), type Some("veth") name Some("veth1")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("8592169a-9f39-43c4-aff2-0e6b3944e486"), ID Some("veth1_p"), type Some("veth") name Some("veth1_p")
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Activating connection dd9146ed-84dc-4124-a22d-637dd2a98687: bond0/bond
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 654e6874-14cb-44ff-8b1c-6f3a6ebb5766: veth0/veth
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Activating connection 3bfbc04a-631f-47e7-a8e9-7246b6a07ba8: veth0_p/veth
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 1e7c1a37-b9b8-457c-baaa-a95eb0909d2e: veth1/veth
[2023-06-15T10:02:26Z INFO  nmstate::nm::query_apply::profile] Activating connection 8592169a-9f39-43c4-aff2-0e6b3944e486: veth1_p/veth
[2023-06-15T10:02:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:27Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:28Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:29Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply
[2023-06-15T10:02:31Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/88
NmstateError: VerificationError: Desired route destination: 0.0.0.0/0 next-hop-interface: bond0 next-hop-address: 192.168.199.1 table-id: 200 not found after apply

Expected results:
No failure

Additional info:
Comment 1 Gris Ge 2023-06-27 13:19:36 UTC 
Tried in my VM with 2.2.12 , works well there.

Could you try again and prepare a server for debug?

Thank you!
Comment 4 Gris Ge 2023-07-12 07:01:57 UTC 
In RHEL 8, route been applied to interface *before* DHCP finished. In RHEL 9, route only applied to interface *after* DHCP finished.

For this bug, nmstate noticed desired route is never applied to bond0 because there is no DHCP server.

Still waiting NM developer to confirm this as a bug or expected behavior.
Comment 5 Gris Ge 2023-07-12 07:07:45 UTC 
Got confirmation from NM developer as this is expected behavior in RHEL 9.


The static route will only been applied to kernel *after* DHCP finished, please check the detail at https://bugzilla.redhat.com/show_bug.cgi?id=2102212
Comment 6 Gris Ge 2023-07-12 07:34:15 UTC 
Closing as this is expected behavior and got consent from reporter.

The static route on DHCP interface required a working DHCP server.