Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2184285

Summary: [sr-iov] cannot reduce VF counts, the failures depend on NICs
Product: Red Hat Enterprise Linux 9 Reporter: Mingyu Shi <mshi>
Component: nmstateAssignee: Gris Ge <fge>
Status: CLOSED ERRATA QA Contact: Mingyu Shi <mshi>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.3CC: ferferna, jiji, jishi, network-qe, sfaye, till
Target Milestone: rcKeywords: Triaged
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: nmstate-2.2.13-1.el9 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:24:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
NMtrace(mlx5_core).log
none
NMtrace(ixgbe).log none

Description Mingyu Shi 2023-04-04 06:26:19 UTC 
Created attachment 1955659 [details]
NMtrace(mlx5_core).log

Description of problem:
Cannot reduce VF counts(maximum - 1), currently it is found on ixgbe and some mlx5_core.
There is a similar bug https://bugzilla.redhat.com/show_bug.cgi?id=2038050 but in this case, i40e works well.

Version-Release number of selected component (if applicable):
nmstate-2.2.9-1.el9.x86_64
nispor-1.2.10-1.el9.x86_64
NetworkManager-1.43.4-1.el9.x86_64

On dell-per740-68.rhts.eng.pek2.redhat.com
driver: mlx5_core
version: 5.14.0-289.el9.x86_64
firmware-version: 14.28.2006 (MT_2420110004)
Mellanox-MT27710_Family

On dell-per740-70.rhts.eng.pek2.redhat.com
driver: ixgbe
version: 5.14.0-289.el9.x86_64
firmware-version: 0x000161ae
Intel-82599ES_10Gb

How reproducible:
100%

Steps to Reproduce:
(Use mlx5_core as an example)
# set VF to maximum counts
echo "
interfaces:
  - name: ens3f0np0
    state: up
    ethernet:
      sr-iov:
        total-vfs: 8
" | nmstatectl apply

# minus 1
echo "
interfaces:
  - name: ens3f0np0
    state: up
    ethernet:
      sr-iov:
        total-vfs: 7
" | nmstatectl apply

Actual results:
[2023-04-04T06:05:18Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/53
[2023-04-04T06:05:19Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("d27a3653-1ea2-4c54-95a2-690751360e03"), ID Some("ens3f0np0"), type Some("802-3-ethernet") name Some("ens3f0np0")
[2023-04-04T06:05:19Z INFO  nmstate::nm::query_apply::profile] Reapplying connection d27a3653-1ea2-4c54-95a2-690751360e03: ens3f0np0/802-3-ethernet
[2023-04-04T06:05:19Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:20Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:21Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:23Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:25Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("d27a3653-1ea2-4c54-95a2-690751360e03"), ID Some("ens3f0np0"), type Some("802-3-ethernet") name Some("ens3f0np0")
[2023-04-04T06:05:25Z INFO  nmstate::nm::query_apply::profile] Reapplying connection d27a3653-1ea2-4c54-95a2-690751360e03: ens3f0np0/802-3-ethernet
[2023-04-04T06:05:25Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:27Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:28Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:41Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/53
NmstateError: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '0'

Expected results:
No failure

Additional info:
Comment 1 Mingyu Shi 2023-04-04 06:32:16 UTC 
Created attachment 1955660 [details]
NMtrace(ixgbe).log

This is the log when change max 63 to 62 on an ixgbe card:
[2023-04-04T06:28:01Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/136
[2023-04-04T06:28:01Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("2f2161e7-7159-41da-8544-b8b945b3b4c2"), ID Some("ens1f0"), type Some("802-3-ethernet") name Some("ens1f0")
[2023-04-04T06:28:01Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 2f2161e7-7159-41da-8544-b8b945b3b4c2: ens1f0/802-3-ethernet
[2023-04-04T06:28:03Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:07Z WARN  nmstate::nm::nm_dbus::error] Unknown DBUS error MethodError("org.freedesktop.DBus.Error.UnknownMethod", Some("No such interface “org.freedesktop.NetworkManager.Device” on object at path /org/freedesktop/NetworkManager/Devices/838"), Msg { type: Error, sender: ":1.1329", reply-serial: 12, body: Signature: [
        s (115),
    ] })
[2023-04-04T06:28:07Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:10Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:15Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:18Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:20Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("2f2161e7-7159-41da-8544-b8b945b3b4c2"), ID Some("ens1f0"), type Some("802-3-ethernet") name Some("ens1f0")
[2023-04-04T06:28:20Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 2f2161e7-7159-41da-8544-b8b945b3b4c2: ens1f0/802-3-ethernet
[2023-04-04T06:28:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:24Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:30Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:29:13Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/136
NmstateError: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
Comment 2 Mingyu Shi 2023-04-04 09:55:07 UTC 
Corret the title as the problem can also happens on i40e card in another way. Given an i40e card which supports up to 64 VFs:

1. Applied 64 VFs with multiple sr-iov options per VF, then change it to 63 VFs with the options: PASS
2. Applied 64 VFs(it doesn't matter if the VF is with options or not), then apply a state including 63 or 1 VF(s) *WITHOUT* any options -> this will get failed.

The following is an example, applied a 64-VFs state without any options, then desired a 63 VFs state:
[2023-04-04T09:51:19Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/183
[2023-04-04T09:51:19Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("e95103ae-c312-47c1-9b59-e4c18a786acf"), ID Some("ens2f0"), type Some("802-3-ethernet") name Some("ens2f0")
[2023-04-04T09:51:19Z INFO  nmstate::nm::query_apply::profile] Reapplying connection e95103ae-c312-47c1-9b59-e4c18a786acf: ens2f0/802-3-ethernet
[2023-04-04T09:51:20Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:21Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:23Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:24Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("e95103ae-c312-47c1-9b59-e4c18a786acf"), ID Some("ens2f0"), type Some("802-3-ethernet") name Some("ens2f0")
[2023-04-04T09:51:26Z INFO  nmstate::nm::query_apply::profile] Reapplying connection e95103ae-c312-47c1-9b59-e4c18a786acf: ens2f0/802-3-ethernet
[2023-04-04T09:51:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:28Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:29Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:30Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:31Z ERROR nmstate::query_apply::sriov] VerificationError: Failed to find VF 0 interface name of PF ens2f0
[2023-04-04T09:51:41Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/183
NmstateError: VerificationError: Failed to find VF 0 interface name of PF ens2f0
Comment 3 Mingyu Shi 2023-06-29 12:44:15 UTC 
Maybe it has the same root cause with https://bugzilla.redhat.com/show_bug.cgi?id=2212380?
Comment 5 Gris Ge 2023-07-13 07:44:37 UTC 
This is fixed by other SR-IOV fix of nmstate. Please try nmstate-2.2.13-1.el9
Comment 10 errata-xmlrpc 2023-11-07 08:24:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:6323