Description of problem: We have an InfiniBand test case where we create 45 VFs and then move to 60 VFs. NM has to go to 0 and then to 60 which takes time and in the meantime nmstate timeouts. https://desktopqe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/beaker-NetworkManager-dcb-RHEL-9.3/6/artifact/artifacts/FAIL_report_NetworkManager-dcb_Test0036_sriov_nmstate_many_vfs.html Version-Release number of selected component (if applicable): nmstate-2.2.11-1.el9.x86_64 NetworkManager-1.43.9-1.el9.x86_64 kernel-5.14.0-319.el9.x86_64 This wasn't present with 9.2 How reproducible: always Steps to Reproduce: 1. run the reproducer on the machine dell-per720-03.klab.eng.bos.redhat.com 2. 3. Actual results: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/behave/model.py", line 1329, in run match.run(runner.context) File "/usr/local/lib/python3.9/site-packages/behave/matchers.py", line 98, in run self.func(context, *args, **kwargs) File "features/steps/commands.py", line 68, in execute_command context.process.run_stdout( File "/mnt/tests/NetworkManager-ci/nmci/process.py", line 341, in run_stdout return self._run( File "/mnt/tests/NetworkManager-ci/nmci/process.py", line 272, in _run self.raise_results(argv_real, f"exited with {returncode}", results) File "/mnt/tests/NetworkManager-ci/nmci/process.py", line 206, in raise_results raise Exception(msg) Exception: `nmstatectl apply /tmp/many-vfs.yaml` exited with 1 STDERR: [2023-06-02T08:00:30Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/2 [2023-06-02T08:00:30Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("56cdfe28-b6e5-47e7-a12a-1ff335e9051a"), ID Some("p4p1"), type Some("802-3-ethernet") name Some("p4p1") [2023-06-02T08:00:30Z INFO nmstate::nm::query_apply::profile] Reapplying connection 56cdfe28-b6e5-47e7-a12a-1ff335e9051a: p4p1/802-3-ethernet [2023-06-02T08:00:35Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:00:43Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:00:52Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:00:57Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:01:04Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:01:07Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("56cdfe28-b6e5-47e7-a12a-1ff335e9051a"), ID Some("p4p1"), type Some("802-3-ethernet") name Some("p4p1") [2023-06-02T08:01:07Z INFO nmstate::nm::query_apply::profile] Reapplying connection 56cdfe28-b6e5-47e7-a12a-1ff335e9051a: p4p1/802-3-ethernet [2023-06-02T08:01:13Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:01:15Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:01:19Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:01:22Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' [2023-06-02T08:01:47Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/2 NmstateError: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0' Expected results: success Additional info:
Patch sent to upstream: https://github.com/nmstate/nmstate/pull/2366 To reproduce the issue, just set SRIOV VF to 62 and then changed to 63.
Verified with: nmstate-2.2.12-2.el9.x86_64 nispor-1.2.10-1.el9.x86_64 NetworkManager-1.43.10-1.el9.x86_64 Linux dell-per740-70.rhts.eng.pek2.redhat.com 5.14.0-332.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jun 26 12:34:51 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux DISTRO=RHEL-9.3.0-20230628.0 Tested drivers ixgbe and i40e