Created attachment 1955659 [details] NMtrace(mlx5_core).log Description of problem: Cannot reduce VF counts(maximum - 1), currently it is found on ixgbe and some mlx5_core. There is a similar bug https://bugzilla.redhat.com/show_bug.cgi?id=2038050 but in this case, i40e works well. Version-Release number of selected component (if applicable): nmstate-2.2.9-1.el9.x86_64 nispor-1.2.10-1.el9.x86_64 NetworkManager-1.43.4-1.el9.x86_64 On dell-per740-68.rhts.eng.pek2.redhat.com driver: mlx5_core version: 5.14.0-289.el9.x86_64 firmware-version: 14.28.2006 (MT_2420110004) Mellanox-MT27710_Family On dell-per740-70.rhts.eng.pek2.redhat.com driver: ixgbe version: 5.14.0-289.el9.x86_64 firmware-version: 0x000161ae Intel-82599ES_10Gb How reproducible: 100% Steps to Reproduce: (Use mlx5_core as an example) # set VF to maximum counts echo " interfaces: - name: ens3f0np0 state: up ethernet: sr-iov: total-vfs: 8 " | nmstatectl apply # minus 1 echo " interfaces: - name: ens3f0np0 state: up ethernet: sr-iov: total-vfs: 7 " | nmstatectl apply Actual results: [2023-04-04T06:05:18Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/53 [2023-04-04T06:05:19Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("d27a3653-1ea2-4c54-95a2-690751360e03"), ID Some("ens3f0np0"), type Some("802-3-ethernet") name Some("ens3f0np0") [2023-04-04T06:05:19Z INFO nmstate::nm::query_apply::profile] Reapplying connection d27a3653-1ea2-4c54-95a2-690751360e03: ens3f0np0/802-3-ethernet [2023-04-04T06:05:19Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:20Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:21Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:22Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:23Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:25Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("d27a3653-1ea2-4c54-95a2-690751360e03"), ID Some("ens3f0np0"), type Some("802-3-ethernet") name Some("ens3f0np0") [2023-04-04T06:05:25Z INFO nmstate::nm::query_apply::profile] Reapplying connection d27a3653-1ea2-4c54-95a2-690751360e03: ens3f0np0/802-3-ethernet [2023-04-04T06:05:25Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:26Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:27Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:28Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8' [2023-04-04T06:05:41Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/53 NmstateError: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '0' Expected results: No failure Additional info:
Created attachment 1955660 [details] NMtrace(ixgbe).log This is the log when change max 63 to 62 on an ixgbe card: [2023-04-04T06:28:01Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/136 [2023-04-04T06:28:01Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("2f2161e7-7159-41da-8544-b8b945b3b4c2"), ID Some("ens1f0"), type Some("802-3-ethernet") name Some("ens1f0") [2023-04-04T06:28:01Z INFO nmstate::nm::query_apply::profile] Reapplying connection 2f2161e7-7159-41da-8544-b8b945b3b4c2: ens1f0/802-3-ethernet [2023-04-04T06:28:03Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:07Z WARN nmstate::nm::nm_dbus::error] Unknown DBUS error MethodError("org.freedesktop.DBus.Error.UnknownMethod", Some("No such interface “org.freedesktop.NetworkManager.Device” on object at path /org/freedesktop/NetworkManager/Devices/838"), Msg { type: Error, sender: ":1.1329", reply-serial: 12, body: Signature: [ s (115), ] }) [2023-04-04T06:28:07Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:10Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:15Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:18Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:20Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("2f2161e7-7159-41da-8544-b8b945b3b4c2"), ID Some("ens1f0"), type Some("802-3-ethernet") name Some("ens1f0") [2023-04-04T06:28:20Z INFO nmstate::nm::query_apply::profile] Reapplying connection 2f2161e7-7159-41da-8544-b8b945b3b4c2: ens1f0/802-3-ethernet [2023-04-04T06:28:22Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:24Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:26Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:28:30Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0' [2023-04-04T06:29:13Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/136 NmstateError: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
Corret the title as the problem can also happens on i40e card in another way. Given an i40e card which supports up to 64 VFs: 1. Applied 64 VFs with multiple sr-iov options per VF, then change it to 63 VFs with the options: PASS 2. Applied 64 VFs(it doesn't matter if the VF is with options or not), then apply a state including 63 or 1 VF(s) *WITHOUT* any options -> this will get failed. The following is an example, applied a 64-VFs state without any options, then desired a 63 VFs state: [2023-04-04T09:51:19Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/183 [2023-04-04T09:51:19Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("e95103ae-c312-47c1-9b59-e4c18a786acf"), ID Some("ens2f0"), type Some("802-3-ethernet") name Some("ens2f0") [2023-04-04T09:51:19Z INFO nmstate::nm::query_apply::profile] Reapplying connection e95103ae-c312-47c1-9b59-e4c18a786acf: ens2f0/802-3-ethernet [2023-04-04T09:51:20Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:21Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:22Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:23Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:24Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:26Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("e95103ae-c312-47c1-9b59-e4c18a786acf"), ID Some("ens2f0"), type Some("802-3-ethernet") name Some("ens2f0") [2023-04-04T09:51:26Z INFO nmstate::nm::query_apply::profile] Reapplying connection e95103ae-c312-47c1-9b59-e4c18a786acf: ens2f0/802-3-ethernet [2023-04-04T09:51:26Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:28Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:29Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:30Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0' [2023-04-04T09:51:31Z ERROR nmstate::query_apply::sriov] VerificationError: Failed to find VF 0 interface name of PF ens2f0 [2023-04-04T09:51:41Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/183 NmstateError: VerificationError: Failed to find VF 0 interface name of PF ens2f0
Maybe it has the same root cause with https://bugzilla.redhat.com/show_bug.cgi?id=2212380?
This is fixed by other SR-IOV fix of nmstate. Please try nmstate-2.2.13-1.el9