Bug 2212380

Summary:	having many VFs can lead to incorrect behavior
Product:	Red Hat Enterprise Linux 9	Reporter:	Vladimir Benes <vbenes>
Component:	nmstate	Assignee:	Gris Ge <fge>
Status:	VERIFIED ---	QA Contact:	Mingyu Shi <mshi>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	9.3	CC:	ferferna, jiji, jishi, network-qe, sfaye, till
Target Milestone:	rc	Keywords:	Regression, Triaged
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	nmstate-2.2.12-2.el9	Doc Type:	No Doc Update
Doc Text:	If this bug requires documentation, please select an appropriate Doc Type value.	Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vladimir Benes 2023-06-05 13:29:25 UTC

Description of problem:
We have an InfiniBand test case where we create 45 VFs and then move to 60 VFs. NM has to go to 0 and then to 60 which takes time and in the meantime nmstate timeouts. 

https://desktopqe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/beaker-NetworkManager-dcb-RHEL-9.3/6/artifact/artifacts/FAIL_report_NetworkManager-dcb_Test0036_sriov_nmstate_many_vfs.html

Version-Release number of selected component (if applicable):
nmstate-2.2.11-1.el9.x86_64
NetworkManager-1.43.9-1.el9.x86_64
kernel-5.14.0-319.el9.x86_64

This wasn't present with 9.2

How reproducible:
always

Steps to Reproduce:
1. run the reproducer on the machine dell-per720-03.klab.eng.bos.redhat.com
2.
3.

Actual results:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/behave/model.py", line 1329, in run
    match.run(runner.context)
  File "/usr/local/lib/python3.9/site-packages/behave/matchers.py", line 98, in run
    self.func(context, *args, **kwargs)
  File "features/steps/commands.py", line 68, in execute_command
    context.process.run_stdout(
  File "/mnt/tests/NetworkManager-ci/nmci/process.py", line 341, in run_stdout
    return self._run(
  File "/mnt/tests/NetworkManager-ci/nmci/process.py", line 272, in _run
    self.raise_results(argv_real, f"exited with {returncode}", results)
  File "/mnt/tests/NetworkManager-ci/nmci/process.py", line 206, in raise_results
    raise Exception(msg)
Exception: `nmstatectl apply /tmp/many-vfs.yaml` exited with 1
STDERR:
[2023-06-02T08:00:30Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/2
[2023-06-02T08:00:30Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("56cdfe28-b6e5-47e7-a12a-1ff335e9051a"), ID Some("p4p1"), type Some("802-3-ethernet") name Some("p4p1")
[2023-06-02T08:00:30Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 56cdfe28-b6e5-47e7-a12a-1ff335e9051a: p4p1/802-3-ethernet
[2023-06-02T08:00:35Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:00:43Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:00:52Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:00:57Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:01:04Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:01:07Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("56cdfe28-b6e5-47e7-a12a-1ff335e9051a"), ID Some("p4p1"), type Some("802-3-ethernet") name Some("p4p1")
[2023-06-02T08:01:07Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 56cdfe28-b6e5-47e7-a12a-1ff335e9051a: p4p1/802-3-ethernet
[2023-06-02T08:01:13Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:01:15Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:01:19Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:01:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'
[2023-06-02T08:01:47Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/2
NmstateError: VerificationError: Verification failure: p4p1.interface.ethernet.sr-iov.total-vfs desire '60', current '0'


Expected results:
success

Additional info:

Comment 1 Gris Ge 2023-06-06 10:03:42 UTC

Patch sent to upstream: https://github.com/nmstate/nmstate/pull/2366

To reproduce the issue, just set SRIOV VF to 62 and then changed to 63.

Comment 4 Mingyu Shi 2023-06-29 12:53:05 UTC

Verified with:
nmstate-2.2.12-2.el9.x86_64
nispor-1.2.10-1.el9.x86_64
NetworkManager-1.43.10-1.el9.x86_64
Linux dell-per740-70.rhts.eng.pek2.redhat.com 5.14.0-332.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jun 26 12:34:51 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
DISTRO=RHEL-9.3.0-20230628.0

Tested drivers ixgbe and i40e