Bug 1819588

Summary: Got NM_ACTIVE_CONNECTION_STATE_REASON_DEVICE_DISCONNECTED when activating SRIOV with min-tx-rate=1 on IGB NIC
Product: Red Hat Enterprise Linux 8 Reporter: Gris Ge <fge>
Component: nmstateAssignee: Fernando F. Mancera <ferferna>
Status: CLOSED ERRATA QA Contact: Mingyu Shi <mshi>
Severity: medium Docs Contact:
Priority: medium    
Version: 8.2CC: acardace, atragler, bgalvani, ferferna, jiji, jishi, lrintel, mshi, network-qe, rkhan, sukulkar, thaller, till
Target Milestone: rc   
Target Release: 8.3   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: nmstate-0.3.4-1.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 03:08:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
bug.yml none

Description Gris Ge 2020-04-01 06:54:53 UTC 
Created attachment 1675305 [details]
bug.yml

Description of problem:

With the same reproduce in bug 1819587, nmstate got failure:

NM_ACTIVE_CONNECTION_STATE_REASON_DEVICE_DISCONNECTED when activating the profile.

Version-Release number of selected component (if applicable):
NetworkManager-1.22.8-4.el8.x86_64
nmstate-0.2.9-1.el8.noarch

How reproducible:
100%

Steps to Reproduce:
1. Find a server with igb SRIOV ready NIC. Assuming that is ens2f0.
2. sudo nmstatectl set bug.yml

Actual results:

Activation failed.

Expected results:

Activation pass.
If min-tx-rate=1 is not supported by NIC driver, 
I would expect NM to use something like
NM_ACTIVE_CONNECTION_STATE_REASON_SRIOV_VF_ATTRIBUTE_NO_SUPPORT

Additional info:
Comment 2 Beniamino Galvani 2020-04-01 08:29:03 UTC 
Hi Gris,

I tried nmstate with the yaml attached and indeed it fails with
"device disconnected". Using the same configuration, nmcli gives the
correct error message:

 # nmcli connection add type ethernet ifname eno2 autoconnect no sriov.total-vfs 1 sriov.vfs "0 min-tx-rate=1"
 Connection 'ethernet-eno2' (4c079460-ded7-4779-bc51-0a1c0685beec) successfully added.

 # nmcli connection up ethernet-eno2
 Error: Connection activation failed: Failed to configure SR-IOV parameters

That's because nmcli does [1], i.e. it takes the reason from the
device the the AC reason is "device disconnected". I think nmstate
should do the same.

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/1.22.8/clients/common/nm-client-utils.c#L392
Comment 3 Gris Ge 2020-04-09 03:23:06 UTC 
Hi Beniamino,

Thanks for the information.
Changing component of this bug to nmstate and I will fix it from nmstate part.
Comment 4 Fernando F. Mancera 2020-05-03 16:40:16 UTC 
This is the upstream patch: https://github.com/nmstate/nmstate/pull/1059
Comment 7 Mingyu Shi 2020-07-20 12:40:18 UTC 
Hi Fernando, looks like it didn't raise a expected exception, can you have a look:

[20:18:40@hp-dl388g8-04 ~]0# cat sriov.yaml ; nmstatectl set sriov.yaml 
---
interfaces:
- name: ens2f0
  state: up
  ethernet:
    sr-iov:
      total-vfs: 1
      vfs:
      - id: 0
        mac-address: d4:Ee:00:25:42:5A
        max-tx-rate: 1000
        min-tx-rate: 1
        spoof-check: true
        trust: true
2020-07-20 20:18:47,312 root         DEBUG    Async action: Create checkpoint started
2020-07-20 20:18:47,315 root         DEBUG    Checkpoint None created for all devices
2020-07-20 20:18:47,315 root         DEBUG    Async action: Create checkpoint finished
2020-07-20 20:18:47,318 root         DEBUG    Async action: Add profile: ens2f0 started
2020-07-20 20:18:47,323 root         DEBUG    Async action: Add profile: ens2f0 finished
2020-07-20 20:18:47,324 root         DEBUG    Async action: Activate profile: ens2f0 started
2020-07-20 20:18:47,326 root         DEBUG    Connection activation initiated: dev=ens2f0, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATING of type NM.ActiveConnectionState>
2020-07-20 20:18:47,964 root         DEBUG    Async action: Rollback to checkpoint /org/freedesktop/NetworkManager/Checkpoint/20 started
2020-07-20 20:18:47,969 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/20 rollback executed
2020-07-20 20:18:47,969 root         DEBUG    Interface ens2f1 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface ens2f0 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface ens6f1 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface ens6f3 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface ens6f0 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface eno1 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface ens6f2 rollback succeeded
2020-07-20 20:18:47,969 root         DEBUG    Interface eno2 rollback succeeded
2020-07-20 20:18:47,970 root         DEBUG    Interface lo rollback succeeded
2020-07-20 20:18:47,970 root         DEBUG    Interface eno3 rollback succeeded
2020-07-20 20:18:47,970 root         DEBUG    Interface eno4 rollback succeeded
2020-07-20 20:18:47,970 root         DEBUG    Async action: Rollback to checkpoint /org/freedesktop/NetworkManager/Checkpoint/20 finished
Traceback (most recent call last):
  File "/usr/bin/nmstatectl", line 11, in <module>
    load_entry_point('nmstate==0.3.3', 'console_scripts', 'nmstatectl')()
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 67, in main
    return args.func(args)
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 256, in apply
    args.save_to_disk,
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 289, in apply_state
    save_to_disk=save_to_disk,
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 71, in apply
    _apply_ifaces_state(plugins, net_state, verify_change, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 104, in _apply_ifaces_state
    plugin.apply_changes(net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/plugin.py", line 178, in apply_changes
    nm_applier.apply_changes(self.context, net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 131, in apply_changes
    _set_ifaces_admin_state(context, ifaces_desired_state, con_profiles)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 306, in _set_ifaces_admin_state
    context.wait_all_finish()
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/context.py", line 216, in wait_all_finish
    raise tmp_error
libnmstate.error.NmstateLibnmError: Activate profile: ens2f0 failed: reason=<enum NM_ACTIVE_CONNECTION_STATE_REASON_DEVICE_DISCONNECTED of type NM.ActiveConnectionStateReason> <enum NM_DEVICE_STATE_REASON_SRIOV_CONFIGURATION_FAILED of type NM.DeviceStateReason>

Versions:
nmstate-0.3.3-2.el8.noarch
NetworkManager-1.26.0-1.el8.x86_64
DISTRO=RHEL-8.3.0-20200716.n.0
Linux hp-dl388g8-04.rhts.eng.pek2.redhat.com 4.18.0-226.el8.x86_64 #1 SMP Wed Jul 15 07:40:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Comment 8 Fernando F. Mancera 2020-07-20 14:10:55 UTC 
Sure, let me look into it. Thanks!
Comment 9 Mingyu Shi 2020-07-30 22:31:00 UTC 
Verified with versions:
nmstate-0.3.4-1.el8.noarch
NetworkManager-1.26.0-2.el8.x86_64
DISTRO=RHEL-8.3.0-20200728.n.0
Linux hp-dl388g8-04.rhts.eng.pek2.redhat.com 4.18.0-228.el8.x86_64 #1 SMP Mon Jul 27 06:05:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

It reports not-supported error now:
# cat sriov.yaml 
---
interfaces:
- name: ens2f0
  state: up
  ethernet:
    sr-iov:
      total-vfs: 1
      vfs:
      - id: 0
        mac-address: d4:Ee:00:25:42:5A
        max-tx-rate: 1000
        min-tx-rate: 1
        spoof-check: true
        trust: true

[06:26:50@hp-dl388g8-04 ~]0# nmstatectl set sriov.yaml 
2020-07-31 06:26:59,091 root         DEBUG    Async action: Create checkpoint started
2020-07-31 06:26:59,094 root         DEBUG    Checkpoint None created for all devices
2020-07-31 06:26:59,095 root         DEBUG    Async action: Create checkpoint finished
2020-07-31 06:26:59,097 root         DEBUG    Async action: Delete profile: id:ens2f0, uuid:5f42f2c7-155b-46b7-991a-e42165ded673 started
2020-07-31 06:26:59,097 root         DEBUG    Async action: Add profile: ens2f0 started
2020-07-31 06:26:59,099 root         DEBUG    Async action: Delete profile: id:ens2f0, uuid:5f42f2c7-155b-46b7-991a-e42165ded673 finished
2020-07-31 06:26:59,102 root         DEBUG    Async action: Add profile: ens2f0 finished
2020-07-31 06:26:59,104 root         DEBUG    Async action: Activate profile: ens2f0 started
2020-07-31 06:26:59,105 root         DEBUG    Connection activation initiated: dev=ens2f0, con-state=<enum NM_ACTIVE_CONNECTION_STATE_ACTIVATING of type NM.ActiveConnectionState>
2020-07-31 06:26:59,695 root         DEBUG    Async action: Rollback to checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 started
2020-07-31 06:26:59,701 root         DEBUG    Checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 rollback executed
2020-07-31 06:26:59,701 root         DEBUG    Interface ens6f0 rollback succeeded
2020-07-31 06:26:59,701 root         DEBUG    Interface ens6f3 rollback succeeded
2020-07-31 06:26:59,701 root         DEBUG    Interface eno3 rollback succeeded
2020-07-31 06:26:59,701 root         DEBUG    Interface eno1 rollback succeeded
2020-07-31 06:26:59,701 root         DEBUG    Interface eno4 rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Interface ens6f1 rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Interface ens2f1 rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Interface ens2f0 rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Interface ens6f2 rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Interface eno2 rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Interface lo rollback succeeded
2020-07-31 06:26:59,702 root         DEBUG    Async action: Rollback to checkpoint /org/freedesktop/NetworkManager/Checkpoint/1 finished
Traceback (most recent call last):
  File "/usr/bin/nmstatectl", line 11, in <module>
    load_entry_point('nmstate==0.3.4', 'console_scripts', 'nmstatectl')()
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 67, in main
    return args.func(args)
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 256, in apply
    args.save_to_disk,
  File "/usr/lib/python3.6/site-packages/nmstatectl/nmstatectl.py", line 289, in apply_state
    save_to_disk=save_to_disk,
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 71, in apply
    _apply_ifaces_state(plugins, net_state, verify_change, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 104, in _apply_ifaces_state
    plugin.apply_changes(net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/plugin.py", line 178, in apply_changes
    nm_applier.apply_changes(self.context, net_state, save_to_disk)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 130, in apply_changes
    _set_ifaces_admin_state(context, ifaces_desired_state, con_profiles)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 316, in _set_ifaces_admin_state
    context.wait_all_finish()
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/context.py", line 216, in wait_all_finish
    raise tmp_error
libnmstate.error.NmstateValueError: Activate profile: ens2f0 failed: reason=The device=ens2f0 does not support one or more of the SR-IOV parameters set.
[06:28:10@hp-dl388g8-04 ~]0# ethtool -i ens2f0
driver: igb
version: 5.6.0-k
firmware-version: 0.93, 0x800005fd
expansion-rom-version: 
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Comment 12 errata-xmlrpc 2020-11-04 03:08:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4696