Bug 2052984

Summary: SNO 4.9: NO-CARRIER on pod interface using VF on intel E810-C NIC; IAVF_ERR_ADMIN_QUEUE_ERROR [rhel-8.4.0.z]
Product: Red Hat Enterprise Linux 8 Reporter: RHEL Program Management Team <pgm-rhel-tools>
Component: kernelAssignee: Patrick Talbert <ptalbert>
kernel sub component: NIC Drivers QA Contact: Ma Yuying <yuma>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: anbhat, apanatto, bnemeth, brault, bzhai, cback, cgoncalves, eglottma, fdangelo, joboyer, jtoppins, kzhang, mhou, mmethot, mschmidt, network-qe, nmurray, openshift-bugs-escalate, ptalbert, sscheink
Version: 8.4Keywords: ZStream
Target Milestone: rcFlags: ptalbert: needinfo-
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: kernel-4.18.0-305.40.1.el8_4 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 2049237 Environment:
Last Closed: 2022-03-08 17:53:02 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2049237    
Bug Blocks: 2042305    

Comment 11 Ma Yuying 2022-02-16 09:54:32 UTC 
Verified with the MR:

[root@dell-per740-49 ~]# dpdk-testpmd -a 0000:3b:01.0 -a 0000:3b:01.1 -n 4 --iova-mode=va --log-level="*:debug" -- -i --nb-cores=5 --forward-mode=mac --port-topology=loop --no-mlockall
EAL: Detected lcore 0 as core 24 on socket 0
EAL: Detected lcore 1 as core 1 on socket 1
EAL: Detected lcore 2 as core 3 on socket 0
EAL: Detected lcore 3 as core 4 on socket 1
EAL: Detected lcore 4 as core 17 on socket 0
EAL: Detected lcore 5 as core 3 on socket 1
EAL: Detected lcore 6 as core 20 on socket 0
EAL: Detected lcore 7 as core 11 on socket 1
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 8 on socket 1
EAL: Detected lcore 10 as core 27 on socket 0
EAL: Detected lcore 11 as core 17 on socket 1
EAL: Detected lcore 12 as core 25 on socket 0
EAL: Detected lcore 13 as core 25 on socket 1
EAL: Detected lcore 14 as core 26 on socket 0
EAL: Detected lcore 15 as core 27 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 16 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.21.0
EAL: Registered [vmbus] bus.
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.21.0
rte_vmbus_register(): Registered driver net_netvsc
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.21.0
EAL: Registered [vdpa] device class.
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: VFIO PCI modules not loaded
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is supported
EAL:   IOMMU type 7 (sPAPR) is not supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 262144
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:1 hugepage_sz:1073741824
EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x980000000, size 800000000
EAL: Creating 2 segment lists: n_segs:32 socket_id:1 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x1000)
EAL: Memseg list allocated at socket 1, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x11c0000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x1000)
EAL: Memseg list allocated at socket 1, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x1a00000000, size 800000000
EAL: TSC frequency is ~3300000 KHz
EAL: Main lcore 0 is ready (tid=7f7bf7041900;cpuset=[0])
EAL: lcore 1 is ready (tid=7f7beda8c700;cpuset=[1])
EAL: lcore 3 is ready (tid=7f7bed28b700;cpuset=[3])
EAL: lcore 5 is ready (tid=7f7beca8a700;cpuset=[5])
EAL: lcore 7 is ready (tid=7f7bdffff700;cpuset=[7])
EAL: lcore 9 is ready (tid=7f7bdf7fe700;cpuset=[9])
EAL: lcore 13 is ready (tid=7f7bde7fc700;cpuset=[13])
EAL: lcore 15 is ready (tid=7f7bddffb700;cpuset=[15])
EAL: lcore 11 is ready (tid=7f7bdeffd700;cpuset=[11])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:3b:01.0 on NUMA socket 0
EAL:   probe driver: 8086:1889 net_iavf
EAL:   using IOMMU type 1 (Type 1)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
EAL: VFIO reports MSI-X BAR as mappable
EAL:   PCI memory mapped at 0x2200000000
EAL:   PCI memory mapped at 0x2200020000
EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:3b:01.0 (socket 0)
iavf_dev_init():  >>
iavf_set_mac_type

iavf_set_mac_type found mac: 2, returns: 0
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0
iavf_check_api_version(): Peer is supported PF host
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0
EAL: PCI device 0000:3b:01.1 on NUMA socket 0
EAL:   probe driver: 8086:1889 net_iavf
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
EAL: VFIO reports MSI-X BAR as mappable
EAL:   PCI memory mapped at 0x2200024000
EAL:   PCI memory mapped at 0x2200044000
EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:3b:01.1 (socket 0)
iavf_dev_init():  >>
iavf_set_mac_type

iavf_set_mac_type found mac: 2, returns: 0
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0
iavf_check_api_version(): Peer is supported PF host
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0
EAL: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set mac packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_0>: n=211456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 1: link state change event
testpmd: create a new mbuf pool <mb_pool_1>: n=211456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 1
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: Heap on socket 1 was expanded by 1024MB
Configuring Port 0 (socket 0)
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 28
iavf_init_rss(): RSS is enabled by PF by default
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24
iavf_dev_tx_queue_setup():  >>
check_tx_vec_allow(): Vector tx can be enabled on this txq.
iavf_dev_rx_queue_setup():  >>
iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
check_rx_vec_allow(): Vector Rx can be enabled on this rxq.
iavf_dev_start():  >>
iavf_set_rx_function(): Using avx2 Vector Rx (port 0).
iavf_set_tx_function(): Using avx2 Vector Tx (port 0).
iavf_configure_queues(): RXDID[1] is not supported, request default RXDID[1] in Queue[0]
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7
iavf_add_del_all_mac_addr(): add/rm mac:6a:49:7d:cd:c3:5a
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10
iavf_dev_tx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 0: link state change event
iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57
iavf_dev_rx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 0: link state change event
iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 0: 6A:49:7D:CD:C3:5A
Configuring Port 1 (socket 0)
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 28
iavf_init_rss(): RSS is enabled by PF by default
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24
iavf_dev_tx_queue_setup():  >>
check_tx_vec_allow(): Vector tx can be enabled on this txq.
iavf_dev_rx_queue_setup():  >>
iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
check_rx_vec_allow(): Vector Rx can be enabled on this rxq.
iavf_dev_start():  >>
iavf_set_rx_function(): Using avx2 Vector Rx (port 1).
iavf_set_tx_function(): Using avx2 Vector Tx (port 1).
iavf_configure_queues(): RXDID[1] is not supported, request default RXDID[1] in Queue[0]
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7
iavf_add_del_all_mac_addr(): add/rm mac:36:88:8a:ee:1d:e2
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10
iavf_dev_tx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 1: link state change event
iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57
iavf_dev_rx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 1: link state change event
iavf_handle_virtchnl_msg(): Failed to read msg from AdminQ,ret: -57
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 1: 36:88:8A:EE:1D:E2
Checking link statuses...
Done
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
testpmd> start
mac packet forwarding - ports=2 - cores=2 - streams=2 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 1) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00
Logical Core 3 (socket 1) forwards packets on 1 streams:
  RX P=1/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01

  mac packet forwarding packets/burst=32
  nb forwarding cores=5 - nb forwarding ports=2
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=512 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=512 - TX free threshold=32
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
  port 1: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=512 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=512 - TX free threshold=32
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.
testpmd> quit

Stopping port 0...
Stopping ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 9
iavf_add_del_all_mac_addr(): add/rm mac:6a:49:7d:cd:c3:5a
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 11
Done

Stopping port 1...
Stopping ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 9
iavf_add_del_all_mac_addr(): add/rm mac:36:88:8a:ee:1d:e2
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 11
Done

Shutting down port 0...
Closing ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 1 is closed
Done

Bye...
[root@dell-per740-49 ~]# ip link show 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether dc:f4:01:e7:99:54 brd ff:ff:ff:ff:ff:ff
3: eno2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether dc:f4:01:e7:99:55 brd ff:ff:ff:ff:ff:ff
4: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether dc:f4:01:e7:99:56 brd ff:ff:ff:ff:ff:ff
5: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether dc:f4:01:e7:99:57 brd ff:ff:ff:ff:ff:ff
6: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:19:03:58 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 6a:49:7d:cd:c3:5a brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off      ------------------> the mac is NOT ZERO
    vf 1     link/ether 36:88:8a:ee:1d:e2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether c2:25:c0:69:ab:40 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether ce:52:52:00:a2:f4 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 46:c0:23:e2:2a:35 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether 16:5d:39:3a:7f:2d brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether 52:fb:40:9c:63:5b brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 3a:c4:f6:a3:a8:9a brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
7: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 40:a6:b7:19:03:59 brd ff:ff:ff:ff:ff:ff
8: ens1f0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether ce:52:52:00:a2:f4 brd ff:ff:ff:ff:ff:ff
9: ens1f0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether c2:25:c0:69:ab:40 brd ff:ff:ff:ff:ff:ff
11: ens1f0v4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 46:c0:23:e2:2a:35 brd ff:ff:ff:ff:ff:ff
12: ens1f0v5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 16:5d:39:3a:7f:2d brd ff:ff:ff:ff:ff:ff
13: ens1f0v6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 52:fb:40:9c:63:5b brd ff:ff:ff:ff:ff:ff
15: ens1f0v7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3a:c4:f6:a3:a8:9a brd ff:ff:ff:ff:ff:ff
[root@dell-per740-49 ~]# uname -a
Linux dell-per740-49.rhts.eng.pek2.redhat.com 4.18.0-305.37.1.el8_4.mr2205_220211_1019.x86_64 #1 SMP Fri Feb 11 10:28:53 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@dell-per740-49 ~]# ethtool -i ens1f0 
driver: ice
version: 4.18.0-305.37.1.el8_4.mr2205_22
firmware-version: 3.00 0x80008256 1.2992.0
expansion-rom-version: 
bus-info: 0000:3b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
Comment 15 mhou 2022-02-23 14:18:57 UTC 
Test on 4.18.0-305.39.1.rt7.111.el8_4.bz2021326.ms2.x86_64 kernel, this issue can't reproduce. But when I reboot the system, the system is still in reboot progress. I don't think this change suit for RT kernel.

Here is test steps:

# ethtool -i ens4f0
driver: ice
version: 4.18.0-305.39.1.rt7.111.el8_4.b
firmware-version: 3.00 0x80008256 1.2992.0
expansion-rom-version: 
bus-info: 0000:b1:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes


1. configure 1G hugepage and CPU tunning.
# cat /sys/class/net/ens4f0/device/numa_node 
1
# lscpi
...
NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
...
# echo "isolated_cores=1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63" > /etc/tuned/realtime-virtual-host-variables.conf
# echo "isolate_managed_irq=Y" >> /etc/tuned/realtime-virtual-host-variables.conf
# grubby --args="default_hugepagesz=1G hugepagesz=1G hugepages=24" --update-kernel=`/usr/sbin/grubby --default-kernel`
# reboot

2. wait for server come back and start test.
# modprobe vfio-pci
# modprobe vfio
# echo 8 > /sys/class/net/ens4f0/device/sriov_numvfs
# ip link show ens4f0 
8: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:ac:23:80 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether de:a4:43:19:61:57 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether e2:3f:fa:ab:9f:14 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 9e:66:93:80:80:d3 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether e6:6e:77:72:de:8e brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 7e:03:48:49:d5:d8 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether c6:45:25:37:f4:0d brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether ca:eb:3a:26:37:d2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 92:b8:a2:53:fb:32 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

# ip link set ens4f0v0 down
# ip link set ens4f0v1 down
# dpdk-devbind.py -u 0000:b1:01.0
# dpdk-devbind.py -u 0000:b1:01.1
# dpdk-devbind.py -b vfio-pci 0000:b1:01.0
# dpdk-devbind.py -b vfio-pci 0000:b1:01.1
# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:b1:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
0000:b1:01.1 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8303 drv=tg3 unused=vfio-pci *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8403 drv=tg3 unused=vfio-pci 
0000:32:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12399 drv=tg3 unused=vfio-pci 
0000:32:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12409 drv=tg3 unused=vfio-pci 
0000:33:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12419 drv=tg3 unused=vfio-pci 
0000:33:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12429 drv=tg3 unused=vfio-pci 
0000:b1:00.0 'Ethernet Controller E810-C for QSFP 1592' if=ens4f0 drv=ice unused=vfio-pci 
0000:b1:00.1 'Ethernet Controller E810-C for QSFP 1592' if=ens4f1 drv=ice unused=vfio-pci 
0000:b1:01.2 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v2 drv=iavf unused=vfio-pci *Active*
0000:b1:01.3 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v3 drv=iavf unused=vfio-pci *Active*
0000:b1:01.4 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v4 drv=iavf unused=vfio-pci *Active*
0000:b1:01.5 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v5 drv=iavf unused=vfio-pci *Active*
0000:b1:01.6 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v6 drv=iavf unused=vfio-pci *Active*
0000:b1:01.7 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v7 drv=iavf unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

# dpdk-testpmd -a 0000:b1:01.0 -a 0000:b1:01.1 -n 4 --iova-mode=va -- -i --nb-cores=5 --forward-mode=mac --port-topology=loop --no-mlockall
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 1
EAL: Detected lcore 2 as core 8 on socket 0
EAL: Detected lcore 3 as core 8 on socket 1
EAL: Detected lcore 4 as core 1 on socket 0
EAL: Detected lcore 5 as core 1 on socket 1
EAL: Detected lcore 6 as core 9 on socket 0
EAL: Detected lcore 7 as core 9 on socket 1
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 2 on socket 1
EAL: Detected lcore 10 as core 10 on socket 0
EAL: Detected lcore 11 as core 10 on socket 1
EAL: Detected lcore 12 as core 3 on socket 0
EAL: Detected lcore 13 as core 3 on socket 1
EAL: Detected lcore 14 as core 11 on socket 0
EAL: Detected lcore 15 as core 11 on socket 1
EAL: Detected lcore 16 as core 4 on socket 0
EAL: Detected lcore 17 as core 4 on socket 1
EAL: Detected lcore 18 as core 12 on socket 0
EAL: Detected lcore 19 as core 12 on socket 1
EAL: Detected lcore 20 as core 5 on socket 0
EAL: Detected lcore 21 as core 5 on socket 1
EAL: Detected lcore 22 as core 13 on socket 0
EAL: Detected lcore 23 as core 13 on socket 1
EAL: Detected lcore 24 as core 6 on socket 0
EAL: Detected lcore 25 as core 6 on socket 1
EAL: Detected lcore 26 as core 14 on socket 0
EAL: Detected lcore 27 as core 14 on socket 1
EAL: Detected lcore 28 as core 7 on socket 0
EAL: Detected lcore 29 as core 7 on socket 1
EAL: Detected lcore 30 as core 15 on socket 0
EAL: Detected lcore 31 as core 15 on socket 1
EAL: Detected lcore 32 as core 0 on socket 0
EAL: Detected lcore 33 as core 0 on socket 1
EAL: Detected lcore 34 as core 8 on socket 0
EAL: Detected lcore 35 as core 8 on socket 1
EAL: Detected lcore 36 as core 1 on socket 0
EAL: Detected lcore 37 as core 1 on socket 1
EAL: Detected lcore 38 as core 9 on socket 0
EAL: Detected lcore 39 as core 9 on socket 1
EAL: Detected lcore 40 as core 2 on socket 0
EAL: Detected lcore 41 as core 2 on socket 1
EAL: Detected lcore 42 as core 10 on socket 0
EAL: Detected lcore 43 as core 10 on socket 1
EAL: Detected lcore 44 as core 3 on socket 0
EAL: Detected lcore 45 as core 3 on socket 1
EAL: Detected lcore 46 as core 11 on socket 0
EAL: Detected lcore 47 as core 11 on socket 1
EAL: Detected lcore 48 as core 4 on socket 0
EAL: Detected lcore 49 as core 4 on socket 1
EAL: Detected lcore 50 as core 12 on socket 0
EAL: Detected lcore 51 as core 12 on socket 1
EAL: Detected lcore 52 as core 5 on socket 0
EAL: Detected lcore 53 as core 5 on socket 1
EAL: Detected lcore 54 as core 13 on socket 0
EAL: Detected lcore 55 as core 13 on socket 1
EAL: Detected lcore 56 as core 6 on socket 0
EAL: Detected lcore 57 as core 6 on socket 1
EAL: Detected lcore 58 as core 14 on socket 0
EAL: Detected lcore 59 as core 14 on socket 1
EAL: Detected lcore 60 as core 7 on socket 0
EAL: Detected lcore 61 as core 7 on socket 1
EAL: Detected lcore 62 as core 15 on socket 0
EAL: Detected lcore 63 as core 15 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_tap.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_virtio.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_pci.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vdev.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_bus_vmbus.so.21.0
EAL: Registered [vmbus] bus.
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_iavf.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_common_mlx5.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_mempool_ring.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_bnxt.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_e1000.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_enic.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_failsafe.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_i40e.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_iavf.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ice.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ixgbe.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx4.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_mlx5.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_netvsc.so.21.0
rte_vmbus_register(): Registered driver net_netvsc
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_nfp.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_qede.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_ring.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vdev_netvsc.so.21.0
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_net_vhost.so.21.0
EAL: Registered [vdpa] device class.
EAL: Ask a virtual area of 0x7000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x7000)
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: VFIO PCI modules not loaded
EAL: Selected IOVA mode 'VA'
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is supported
EAL:   IOMMU type 7 (sPAPR) is not supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Ask a virtual area of 0x2e000 bytes
EAL: Virtual area found at 0x100007000 (size = 0x2e000)
EAL: Setting up physically contiguous memory...
EAL: Setting maximum number of open files to 262144
EAL: Detected memory type: socket_id:0 hugepage_sz:1073741824
EAL: Detected memory type: socket_id:1 hugepage_sz:1073741824
EAL: Creating 2 segment lists: n_segs:32 socket_id:0 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x100035000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x140000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x140000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x940000000 (size = 0x1000)
EAL: Memseg list allocated at socket 0, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x980000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x980000000, size 800000000
EAL: Creating 2 segment lists: n_segs:32 socket_id:1 hugepage_sz:1073741824
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x1180000000 (size = 0x1000)
EAL: Memseg list allocated at socket 1, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x11c0000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x11c0000000, size 800000000
EAL: Ask a virtual area of 0x1000 bytes
EAL: Virtual area found at 0x19c0000000 (size = 0x1000)
EAL: Memseg list allocated at socket 1, page size 0x100000kB
EAL: Ask a virtual area of 0x800000000 bytes
EAL: Virtual area found at 0x1a00000000 (size = 0x800000000)
EAL: VA reserved for memseg list at 0x1a00000000, size 800000000
EAL: TSC frequency is ~2900000 KHz
EAL: Main lcore 0 is ready (tid=7f046f11c900;cpuset=[0])
EAL: lcore 2 is ready (tid=7f0465b68700;cpuset=[2])
EAL: lcore 4 is ready (tid=7f0465367700;cpuset=[4])
EAL: lcore 6 is ready (tid=7f0464b66700;cpuset=[6])
EAL: lcore 8 is ready (tid=7f0457fff700;cpuset=[8])
EAL: lcore 12 is ready (tid=7f0456ffd700;cpuset=[12])
EAL: lcore 14 is ready (tid=7f04567fc700;cpuset=[14])
EAL: lcore 10 is ready (tid=7f04577fe700;cpuset=[10])
EAL: lcore 18 is ready (tid=7f04557fa700;cpuset=[18])
EAL: lcore 20 is ready (tid=7f0454ff9700;cpuset=[20])
EAL: lcore 22 is ready (tid=7f043ffff700;cpuset=[22])
EAL: lcore 24 is ready (tid=7f043f7fe700;cpuset=[24])
EAL: lcore 26 is ready (tid=7f043effd700;cpuset=[26])
EAL: lcore 16 is ready (tid=7f0455ffb700;cpuset=[16])
EAL: lcore 28 is ready (tid=7f043e7fc700;cpuset=[28])
EAL: lcore 30 is ready (tid=7f043dffb700;cpuset=[30])
EAL: lcore 32 is ready (tid=7f043d7fa700;cpuset=[32])
EAL: lcore 34 is ready (tid=7f043cff9700;cpuset=[34])
EAL: lcore 36 is ready (tid=7f0417fff700;cpuset=[36])
EAL: lcore 38 is ready (tid=7f04177fe700;cpuset=[38])
EAL: lcore 40 is ready (tid=7f0416ffd700;cpuset=[40])
EAL: lcore 42 is ready (tid=7f04167fc700;cpuset=[42])
EAL: lcore 46 is ready (tid=7f04157fa700;cpuset=[46])
EAL: lcore 48 is ready (tid=7f0414ff9700;cpuset=[48])
EAL: lcore 50 is ready (tid=7f03fffff700;cpuset=[50])
EAL: lcore 52 is ready (tid=7f03ff7fe700;cpuset=[52])
EAL: lcore 54 is ready (tid=7f03feffd700;cpuset=[54])
EAL: lcore 56 is ready (tid=7f03fe7fc700;cpuset=[56])
EAL: lcore 58 is ready (tid=7f03fdffb700;cpuset=[58])
EAL: lcore 60 is ready (tid=7f03fd7fa700;cpuset=[60])
EAL: lcore 62 is ready (tid=7f03fcff9700;cpuset=[62])
EAL: lcore 44 is ready (tid=7f0415ffb700;cpuset=[44])
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 0
EAL: Restoring previous memory policy: 0
EAL: request: mp_malloc_sync
EAL: Heap on socket 0 was expanded by 1024MB
EAL: PCI device 0000:b1:01.0 on NUMA socket 1
EAL:   probe driver: 8086:1889 net_iavf
EAL:   using IOMMU type 1 (Type 1)
EAL: Mem event callback 'vfio_mem_event_clb:(nil)' registered
EAL: Installed memory event callback for VFIO
EAL: VFIO reports MSI-X BAR as mappable
EAL:   PCI memory mapped at 0x2200000000
EAL:   PCI memory mapped at 0x2200020000
EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:b1:01.0 (socket 1)
EAL: Trying to obtain current memory policy.
EAL: Setting policy MPOL_PREFERRED for socket 1
EAL: Restoring previous memory policy: 0
EAL: Calling mem event callback 'vfio_mem_event_clb:(nil)'
EAL: request: mp_malloc_sync
EAL: Heap on socket 1 was expanded by 1024MB
iavf_dev_init():  >>
iavf_set_mac_type

iavf_set_mac_type found mac: 2, returns: 0
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0
iavf_check_api_version(): Peer is supported PF host
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
EAL: PCI device 0000:b1:01.1 on NUMA socket 1
EAL:   probe driver: 8086:1889 net_iavf
EAL: VFIO reports MSI-X BAR as mappable
EAL:   PCI memory mapped at 0x2200024000
EAL:   PCI memory mapped at 0x2200044000
EAL: Probe PCI driver: net_iavf (8086:1889) device: 0000:b1:01.1 (socket 1)
iavf_dev_init():  >>
iavf_set_mac_type

iavf_set_mac_type found mac: 2, returns: 0
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 1, retval 0
iavf_check_api_version(): Peer is supported PF host
iavf_read_msg_from_pf(): Can't read msg from AQ
iavf_read_msg_from_pf(): AQ from pf carries opcode 3, retval 0
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event
EAL: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set mac packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_0>: n=395456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=395456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 1)
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 28
iavf_init_rss(): RSS is enabled by PF by default
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24
iavf_dev_tx_queue_setup():  >>
check_tx_vec_allow(): Vector tx can be enabled on this txq.
iavf_dev_rx_queue_setup():  >>
iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=0, queue=0.
check_rx_vec_allow(): Vector Rx can be enabled on this rxq.
iavf_dev_start():  >>
iavf_set_rx_function(): Using avx2 Vector Rx (port 0).
iavf_set_tx_function(): Using avx2 Vector Tx (port 0).
iavf_configure_queues(): RXDID[1] is not supported, request default RXDID[1] in Queue[0]
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7
iavf_add_del_all_mac_addr(): add/rm mac:de:a4:43:19:61:57
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10
iavf_dev_tx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 0: link state change event
iavf_dev_rx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 0: link state change event
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 0: DE:A4:43:19:61:57
Configuring Port 1 (socket 1)
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 28
iavf_init_rss(): RSS is enabled by PF by default
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 24
iavf_dev_tx_queue_setup():  >>
check_tx_vec_allow(): Vector tx can be enabled on this txq.
iavf_dev_rx_queue_setup():  >>
iavf_dev_rx_queue_setup(): Rx Burst Bulk Alloc Preconditions are satisfied. Rx Burst Bulk Alloc function will be used on port=1, queue=0.
check_rx_vec_allow(): Vector Rx can be enabled on this rxq.
iavf_dev_start():  >>
iavf_set_rx_function(): Using avx2 Vector Rx (port 1).
iavf_set_tx_function(): Using avx2 Vector Tx (port 1).
iavf_configure_queues(): RXDID[1] is not supported, request default RXDID[1] in Queue[0]
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 6
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 7
iavf_add_del_all_mac_addr(): add/rm mac:e2:3f:fa:ab:9f:14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 10
iavf_dev_tx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 1: link state change event
iavf_dev_rx_queue_start():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 8
iavf_handle_pf_event_msg(): VIRTCHNL_EVENT_LINK_CHANGE event

Port 1: link state change event
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 1: E2:3F:FA:AB:9F:14
Checking link statuses...
Done
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
testpmd> show port stats all
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 15

testpmd> show port info all

********************* Infos for port 0  *********************
MAC address: DE:A4:43:19:61:57
Device name: 0000:b1:01.0
Driver name: net_iavf
Firmware-version: not available
Devargs: 
Connect to socket: 1
memory allocation on the socket: 1
Link status: up
Link speed: 100 Gbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 64
Maximum number of MAC addresses of hash filtering: 0
VLAN offload: 
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 52
Redirection table size: 64
Supported RSS offload flow types:
  ipv4-frag
  ipv4-tcp
  ipv4-udp
  ipv4-sctp
  ipv4-other
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 9728
Maximum configurable size of LRO aggregated packet: 0
Current number of RX queues: 1
Max possible RX queues: 256
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 64
RXDs number alignment: 32
Current number of TX queues: 1
Max possible TX queues: 256
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 64
TXDs number alignment: 32
Max segment number per packet: 0
Max segment number per MTU/TSO: 0

********************* Infos for port 1  *********************
MAC address: E2:3F:FA:AB:9F:14
Device name: 0000:b1:01.1
Driver name: net_iavf
Firmware-version: not available
Devargs: 
Connect to socket: 1
memory allocation on the socket: 1
Link status: up
Link speed: 100 Gbps
Link duplex: full-duplex
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 64
Maximum number of MAC addresses of hash filtering: 0
VLAN offload: 
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 52
Redirection table size: 64
Supported RSS offload flow types:
  ipv4-frag
  ipv4-tcp
  ipv4-udp
  ipv4-sctp
  ipv4-other
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 9728
Maximum configurable size of LRO aggregated packet: 0
Current number of RX queues: 1
Max possible RX queues: 256
Max possible number of RXDs per queue: 4096
Min possible number of RXDs per queue: 64
RXDs number alignment: 32
Current number of TX queues: 1
Max possible TX queues: 256
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 64
TXDs number alignment: 32
Max segment number per packet: 0
Max segment number per MTU/TSO: 0
testpmd> quit

Stopping port 0...
Stopping ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 9
iavf_add_del_all_mac_addr(): add/rm mac:de:a4:43:19:61:57
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 11
Done

Stopping port 1...
Stopping ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 9
iavf_add_del_all_mac_addr(): add/rm mac:e2:3f:fa:ab:9f:14
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 11
Done

Shutting down port 0...
Closing ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
iavf_dev_stop():  >>
iavf_handle_virtchnl_msg(): adminq response is received, opcode = 14
Port 1 is closed
Done

Bye...

# ip link show ens4f0
8: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:ac:23:80 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether de:a4:43:19:61:57 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether e2:3f:fa:ab:9f:14 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 9e:66:93:80:80:d3 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether e6:6e:77:72:de:8e brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 7e:03:48:49:d5:d8 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether c6:45:25:37:f4:0d brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether ca:eb:3a:26:37:d2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 92:b8:a2:53:fb:32 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:b1:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
0000:b1:01.1 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8303 drv=tg3 unused=vfio-pci *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8403 drv=tg3 unused=vfio-pci 
0000:32:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12399 drv=tg3 unused=vfio-pci 
0000:32:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12409 drv=tg3 unused=vfio-pci 
0000:33:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12419 drv=tg3 unused=vfio-pci 
0000:33:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12429 drv=tg3 unused=vfio-pci 
0000:b1:00.0 'Ethernet Controller E810-C for QSFP 1592' if=ens4f0 drv=ice unused=vfio-pci 
0000:b1:00.1 'Ethernet Controller E810-C for QSFP 1592' if=ens4f1 drv=ice unused=vfio-pci 
0000:b1:01.2 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v2 drv=iavf unused=vfio-pci *Active*
0000:b1:01.3 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v3 drv=iavf unused=vfio-pci *Active*
0000:b1:01.4 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v4 drv=iavf unused=vfio-pci *Active*
0000:b1:01.5 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v5 drv=iavf unused=vfio-pci *Active*
0000:b1:01.6 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v6 drv=iavf unused=vfio-pci *Active*
0000:b1:01.7 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v7 drv=iavf unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================

3. keep vf load vfio-pci driver and reboot server. The system still hung the reboot progress.
# reboot
         Starting Reboot...
[  702.975038] printk: systemd-shutdow: 37 output lines suppressed due to ratelimiting
[  703.069241] systemd-shutdown[1]: Syncing filesystems and block devices.
[  703.086178] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[  703.128191] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  703.149376] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0
[  703.156560] systemd-shutdown[1]: Unmounting file systems.
[  703.162739] [4199]: Remounting '/' read-only in with options 'seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota'.
[  703.249830] systemd-shutdown[1]: All filesystems unmounted.
[  703.255400] systemd-shutdown[1]: Deactivating swaps.
[  703.260400] systemd-shutdown[1]: All swaps deactivated.
[  703.265626] systemd-shutdown[1]: Detaching loop devices.
[  703.271027] systemd-shutdown[1]: All loop devices detached.
[  703.330233] watchdog: watchdog0: watchdog did not stop!
[  703.335458] printk: shutdown: 9 output lines suppressed due to ratelimiting
[  703.370213] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[  703.422742] XFS (dm-0): Unmounting Filesystem
[  703.433495] dracut Warning: Unmounted /oldroot.
[  703.456259] dracut: Disassembling device-mapper devices
[  703.495460] kvm: exiting hardware virtualization
Rebooting.
[  704.976395] pci 0000:b1:01.0: Removing from iommu group 253
[  704.982162] pci 0000:b1:01.1: Removing from iommu group 254

I do more tests to find why the reboot will hang. If I re-load vf from vfio-pci to iavf and restart the test system, the system still got huge status. check with normal reboot process, I found system hang on "ice 0000:b1:00.0: Removed PTP clock"

Test1: follow the test step with comment41 and do more test steps as below, after check vf mac. 
# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:b1:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
0000:b1:01.1 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8303 drv=tg3 unused=vfio-pci *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8403 drv=tg3 unused=vfio-pci 
0000:32:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12399 drv=tg3 unused=vfio-pci 
0000:32:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12409 drv=tg3 unused=vfio-pci 
0000:33:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12419 drv=tg3 unused=vfio-pci 
0000:33:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12429 drv=tg3 unused=vfio-pci 
0000:b1:00.0 'Ethernet Controller E810-C for QSFP 1592' if=ens4f0 drv=ice unused=vfio-pci 
0000:b1:00.1 'Ethernet Controller E810-C for QSFP 1592' if=ens4f1 drv=ice unused=vfio-pci 
0000:b1:01.2 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v2 drv=iavf unused=vfio-pci *Active*
0000:b1:01.3 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v3 drv=iavf unused=vfio-pci *Active*
0000:b1:01.4 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v4 drv=iavf unused=vfio-pci *Active*
0000:b1:01.5 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v5 drv=iavf unused=vfio-pci *Active*
0000:b1:01.6 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v6 drv=iavf unused=vfio-pci *Active*
0000:b1:01.7 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v7 drv=iavf unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================
# dpdk-devbind.py -b iavf 0000:b1:01.0
# dpdk-devbind.py -b iavf 0000:b1:01.1
# ip link show ens4f0
8: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether b4:96:91:ac:23:80 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether de:a4:43:19:61:57 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether e2:3f:fa:ab:9f:14 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 9e:66:93:80:80:d3 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether e6:6e:77:72:de:8e brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 7e:03:48:49:d5:d8 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether c6:45:25:37:f4:0d brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether ca:eb:3a:26:37:d2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 92:b8:a2:53:fb:32 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
# reboot
         Starting Reboot...
[  188.583796] systemd-shutdown[1]: Syncing filesystems and block devices.
[  188.602720] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[  188.639318] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  188.663081] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0
[  188.670251] systemd-shutdown[1]: Unmounting file systems.
[  188.676501] [4128]: Remounting '/' read-only in with options 'seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota'.
[  188.764105] systemd-shutdown[1]: All filesystems unmounted.
[  188.769678] systemd-shutdown[1]: Deactivating swaps.
[  188.774667] systemd-shutdown[1]: All swaps deactivated.
[  188.779894] systemd-shutdown[1]: Detaching loop devices.
[  188.785283] systemd-shutdown[1]: All loop devices detached.
[  188.848507] watchdog: watchdog0: watchdog did not stop!
[  188.853736] printk: shutdown: 9 output lines suppressed due to ratelimiting
[  188.888620] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[  188.944151] XFS (dm-0): Unmounting Filesystem
[  188.954336] dracut Warning: Unmounted /oldroot.
[  188.976896] dracut: Disassembling device-mapper devices
Rebooting.[  189.016018] kvm: exiting hardware virtualization

Test2: follow the test step with comment41 and do more test steps as below, after check vf mac. 
# dpdk-devbind.py --status

Network devices using DPDK-compatible driver
============================================
0000:b1:01.0 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf
0000:b1:01.1 'Ethernet Adaptive Virtual Function 1889' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:04:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8303 drv=tg3 unused=vfio-pci *Active*
0000:04:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno8403 drv=tg3 unused=vfio-pci 
0000:32:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12399 drv=tg3 unused=vfio-pci 
0000:32:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12409 drv=tg3 unused=vfio-pci 
0000:33:00.0 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12419 drv=tg3 unused=vfio-pci 
0000:33:00.1 'NetXtreme BCM5720 2-port Gigabit Ethernet PCIe 165f' if=eno12429 drv=tg3 unused=vfio-pci 
0000:b1:00.0 'Ethernet Controller E810-C for QSFP 1592' if=ens4f0 drv=ice unused=vfio-pci 
0000:b1:00.1 'Ethernet Controller E810-C for QSFP 1592' if=ens4f1 drv=ice unused=vfio-pci 
0000:b1:01.2 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v2 drv=iavf unused=vfio-pci *Active*
0000:b1:01.3 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v3 drv=iavf unused=vfio-pci *Active*
0000:b1:01.4 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v4 drv=iavf unused=vfio-pci *Active*
0000:b1:01.5 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v5 drv=iavf unused=vfio-pci *Active*
0000:b1:01.6 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v6 drv=iavf unused=vfio-pci *Active*
0000:b1:01.7 'Ethernet Adaptive Virtual Function 1889' if=ens4f0v7 drv=iavf unused=vfio-pci *Active*

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================

No 'Regex' devices detected
===========================
# dpdk-devbind.py -b iavf 0000:b1:01.0
# dpdk-devbind.py -b iavf 0000:b1:01.1
# echo 0 > /sys/class/net/ens4f0/device/sriov_numvfs
# systemctl status libvirtd
Unit libvirtd.service could not be found.
# reboot
 Starting Reboot...
[ 1014.375119] systemd-shutdown[1]: Syncing filesystems and block devices.
[ 1014.398563] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[ 1014.434460] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[ 1014.454409] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0
[ 1014.461455] systemd-shutdown[1]: Unmounting file systems.
[ 1014.467575] [4575]: Remounting '/' read-only in with options 'seclabel,attr2,inode64,logbufs=8,logbsize=64k,sunit=128,swidth=256,noquota'.
[ 1014.571806] systemd-shutdown[1]: All filesystems unmounted.
[ 1014.577379] systemd-shutdown[1]: Deactivating swaps.
[ 1014.582369] systemd-shutdown[1]: All swaps deactivated.
[ 1014.587595] systemd-shutdown[1]: Detaching loop devices.
[ 1014.592987] systemd-shutdown[1]: All loop devices detached.
[ 1014.655498] watchdog: watchdog0: watchdog did not stop!
[ 1014.660729] printk: shutdown: 9 output lines suppressed due to ratelimiting
[ 1014.695739] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[ 1014.754206] XFS (dm-0): Unmounting Filesystem
[ 1014.765143] dracut Warning: Unmounted /oldroot.
[ 1014.787985] dracut: Disassembling device-mapper devices
[ 1014.829837] kvm: exiting hardware virtualization
Rebooting.
[ 1016.655627] ice 0000:b1:00.0: Removed PTP clock
[ 1018.187247] megaraid_sas 0000:31:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[ 1018.512786] reboot: Restarting system
[ 1018.516450] reboot: machine restart
Comment 24 errata-xmlrpc 2022-03-08 17:53:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0777