Bug 2184285

Summary:

[sr-iov] cannot reduce VF counts, the failures depend on NICs

Product:

Red Hat Enterprise Linux 9

Reporter:

Mingyu Shi <mshi>

Component:

nmstate

Assignee:

Gris Ge <fge>

Status:

VERIFIED ---

QA Contact:

Mingyu Shi <mshi>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

9.3

CC:

ferferna, jiji, jishi, network-qe, sfaye, till

Target Milestone:

Keywords:

Triaged

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

nmstate-2.2.13-1.el9

Doc Type:

No Doc Update

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
NMtrace(mlx5_core).log	none
NMtrace(ixgbe).log	none

Description Mingyu Shi 2023-04-04 06:26:19 UTC

Created attachment 1955659 [details]
NMtrace(mlx5_core).log

Description of problem:
Cannot reduce VF counts(maximum - 1), currently it is found on ixgbe and some mlx5_core.
There is a similar bug https://bugzilla.redhat.com/show_bug.cgi?id=2038050 but in this case, i40e works well.

Version-Release number of selected component (if applicable):
nmstate-2.2.9-1.el9.x86_64
nispor-1.2.10-1.el9.x86_64
NetworkManager-1.43.4-1.el9.x86_64

On dell-per740-68.rhts.eng.pek2.redhat.com
driver: mlx5_core
version: 5.14.0-289.el9.x86_64
firmware-version: 14.28.2006 (MT_2420110004)
Mellanox-MT27710_Family

On dell-per740-70.rhts.eng.pek2.redhat.com
driver: ixgbe
version: 5.14.0-289.el9.x86_64
firmware-version: 0x000161ae
Intel-82599ES_10Gb

How reproducible:
100%

Steps to Reproduce:
(Use mlx5_core as an example)
# set VF to maximum counts
echo "
interfaces:
  - name: ens3f0np0
    state: up
    ethernet:
      sr-iov:
        total-vfs: 8
" | nmstatectl apply

# minus 1
echo "
interfaces:
  - name: ens3f0np0
    state: up
    ethernet:
      sr-iov:
        total-vfs: 7
" | nmstatectl apply

Actual results:
[2023-04-04T06:05:18Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/53
[2023-04-04T06:05:19Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("d27a3653-1ea2-4c54-95a2-690751360e03"), ID Some("ens3f0np0"), type Some("802-3-ethernet") name Some("ens3f0np0")
[2023-04-04T06:05:19Z INFO  nmstate::nm::query_apply::profile] Reapplying connection d27a3653-1ea2-4c54-95a2-690751360e03: ens3f0np0/802-3-ethernet
[2023-04-04T06:05:19Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:20Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:21Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:23Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:25Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("d27a3653-1ea2-4c54-95a2-690751360e03"), ID Some("ens3f0np0"), type Some("802-3-ethernet") name Some("ens3f0np0")
[2023-04-04T06:05:25Z INFO  nmstate::nm::query_apply::profile] Reapplying connection d27a3653-1ea2-4c54-95a2-690751360e03: ens3f0np0/802-3-ethernet
[2023-04-04T06:05:25Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:27Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:28Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '8'
[2023-04-04T06:05:41Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/53
NmstateError: VerificationError: Verification failure: ens3f0np0.interface.ethernet.sr-iov.total-vfs desire '7', current '0'

Expected results:
No failure

Additional info:

Comment 1 Mingyu Shi 2023-04-04 06:32:16 UTC

Created attachment 1955660 [details]
NMtrace(ixgbe).log

This is the log when change max 63 to 62 on an ixgbe card:
[2023-04-04T06:28:01Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/136
[2023-04-04T06:28:01Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("2f2161e7-7159-41da-8544-b8b945b3b4c2"), ID Some("ens1f0"), type Some("802-3-ethernet") name Some("ens1f0")
[2023-04-04T06:28:01Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 2f2161e7-7159-41da-8544-b8b945b3b4c2: ens1f0/802-3-ethernet
[2023-04-04T06:28:03Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:07Z WARN  nmstate::nm::nm_dbus::error] Unknown DBUS error MethodError("org.freedesktop.DBus.Error.UnknownMethod", Some("No such interface “org.freedesktop.NetworkManager.Device” on object at path /org/freedesktop/NetworkManager/Devices/838"), Msg { type: Error, sender: ":1.1329", reply-serial: 12, body: Signature: [
        s (115),
    ] })
[2023-04-04T06:28:07Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:10Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:15Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:18Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:20Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("2f2161e7-7159-41da-8544-b8b945b3b4c2"), ID Some("ens1f0"), type Some("802-3-ethernet") name Some("ens1f0")
[2023-04-04T06:28:20Z INFO  nmstate::nm::query_apply::profile] Reapplying connection 2f2161e7-7159-41da-8544-b8b945b3b4c2: ens1f0/802-3-ethernet
[2023-04-04T06:28:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:24Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:28:30Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'
[2023-04-04T06:29:13Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/136
NmstateError: VerificationError: Verification failure: ens1f0.interface.ethernet.sr-iov.total-vfs desire '62', current '0'

Comment 2 Mingyu Shi 2023-04-04 09:55:07 UTC

Corret the title as the problem can also happens on i40e card in another way. Given an i40e card which supports up to 64 VFs:

1. Applied 64 VFs with multiple sr-iov options per VF, then change it to 63 VFs with the options: PASS
2. Applied 64 VFs(it doesn't matter if the VF is with options or not), then apply a state including 63 or 1 VF(s) *WITHOUT* any options -> this will get failed.

The following is an example, applied a 64-VFs state without any options, then desired a 63 VFs state:
[2023-04-04T09:51:19Z INFO  nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/183
[2023-04-04T09:51:19Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("e95103ae-c312-47c1-9b59-e4c18a786acf"), ID Some("ens2f0"), type Some("802-3-ethernet") name Some("ens2f0")
[2023-04-04T09:51:19Z INFO  nmstate::nm::query_apply::profile] Reapplying connection e95103ae-c312-47c1-9b59-e4c18a786acf: ens2f0/802-3-ethernet
[2023-04-04T09:51:20Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:21Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:22Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:23Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:24Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:26Z INFO  nmstate::nm::query_apply::profile] Modifying connection UUID Some("e95103ae-c312-47c1-9b59-e4c18a786acf"), ID Some("ens2f0"), type Some("802-3-ethernet") name Some("ens2f0")
[2023-04-04T09:51:26Z INFO  nmstate::nm::query_apply::profile] Reapplying connection e95103ae-c312-47c1-9b59-e4c18a786acf: ens2f0/802-3-ethernet
[2023-04-04T09:51:26Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:28Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:29Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:30Z INFO  nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ens2f0.interface.ethernet.sr-iov.total-vfs desire '63', current '0'
[2023-04-04T09:51:31Z ERROR nmstate::query_apply::sriov] VerificationError: Failed to find VF 0 interface name of PF ens2f0
[2023-04-04T09:51:41Z INFO  nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/183
NmstateError: VerificationError: Failed to find VF 0 interface name of PF ens2f0

Comment 3 Mingyu Shi 2023-06-29 12:44:15 UTC

Maybe it has the same root cause with https://bugzilla.redhat.com/show_bug.cgi?id=2212380?

Comment 5 Gris Ge 2023-07-13 07:44:37 UTC

This is fixed by other SR-IOV fix of nmstate. Please try nmstate-2.2.13-1.el9