Bug 2186080

Summary: i40e card: the vf mac address will be changed when run testpmd command on rhel9.2
Product: Red Hat Enterprise Linux Fast Datapath Reporter: liting <tli>
Component: DPDKAssignee: David Marchand <dmarchan>
DPDK sub component: sriov QA Contact: liting <tli>
Status: NEW --- Docs Contact:
Severity: unspecified    
Priority: unspecified CC: ctrautma, dmarchan, jhsiao, sassmann
Version: FDP 23.C   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liting 2023-04-12 03:52:32 UTC
Description of problem:
i40e card: the vf mac address will be changed when run testpmd command on rhel9.2

Version-Release number of selected component (if applicable):
kernel 5.14.0-284.8.1.el9_2.x86_64
dpdk-21.11-1.el9_0.x86_64

How reproducible:


Steps to Reproduce:
1. create 2 vfs for pf
echo 2 >/sys/devices/pci0000:00/0000:00:03.2/0000:07:00.0/sriov_numvfs
2. set vf 0 to trust on
ip link set enp7s0f0 vf 0 trust on
3. set vf 1 to trust off
ip link set enp7s0f0 vf 1 trust off

[root@dell-per730-52 ~]# ip link show
7: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:ad:bc:e8 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 4a:a8:3f:c4:bc:06 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
    vf 1     link/ether 6a:7a:9a:10:b0:04 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

4. bind vf0 to dpdk
driverctl -v set-override  0000:07:02.0 vfio-pci

5. start testpmd with vf0
[root@dell-per730-52 ~]# dpdk-testpmd -c 0xf -n 4 -a 0000:07:02.0 -- -i --forward-mode=mac --auto-start
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 2048 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:07:02.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set mac packet forwarding mode
Auto-start selected
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=171456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
iavf_configure_queues(): RXDID[1] is not supported, request default RXDID[1] in Queue[0]

Port 0: link state change event

Port 0: link state change event
Port 0: 1E:5D:D0:6E:8C:97
Checking link statuses...
Done
Start automatic packet forwarding
mac packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 1) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  mac packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=1
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=512 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=512 - TX free threshold=32
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
testpmd> 

6. check the vf mac
7: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:ad:bc:e8 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 1e:5d:d0:6e:8c:97 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
    vf 1     link/ether 6a:7a:9a:10:b0:04 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

7. bind vf0 to dpdk
driverctl -v set-override  0000:07:02.1 vfio-pci

8. start testpmd with vf1
[root@dell-per730-52 ~]# dpdk-testpmd -c 0xf -n 4 -a 0000:07:02.1 -- -i --forward-mode=mac --auto-start
EAL: Detected CPU lcores: 56
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: No available 2048 kB hugepages reported
EAL: VFIO support initialized
EAL: Using IOMMU type 1 (Type 1)
EAL: Probe PCI driver: net_iavf (8086:154c) device: 0000:07:02.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
Set mac packet forwarding mode
Auto-start selected
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=171456, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
iavf_configure_queues(): RXDID[1] is not supported, request default RXDID[1] in Queue[0]

Port 0: link state change event

Port 0: link state change event
Port 0: F2:82:17:D6:EA:BA
Checking link statuses...
Done
Start automatic packet forwarding
mac packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 1 (socket 1) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  mac packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=1
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=512 - RX free threshold=32
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=512 - TX free threshold=32
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=32
testpmd> 

9. check the vf mac
7: enp7s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:fd:fe:ad:bc:e8 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 1e:5d:d0:6e:8c:97 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
    vf 1     link/ether f2:82:17:d6:ea:ba brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off

Actual results:
when the vf is trust on, the mac address will be changed when start testpmd.

Expected results:
when the vf no matter is trust on or trust off, the mac address should not be changed when start testpmd command.
rhel9.2 has this issue, rhel8.6 has no this issue.

Additional info:

Comment 1 David Marchand 2023-04-13 09:30:24 UTC
(In reply to liting from comment #0)
> Actual results:
> when the vf is trust on, the mac address will be changed when start testpmd.
> 
> Expected results:
> when the vf no matter is trust on or trust off, the mac address should not
> be changed when start testpmd command.

The VF mac address change is handled by the PF (kernel) driver, so it is best our kernel driver guys have a look at this bz (Stefan?).

At least, from my perspective, allowing to change the mac address with trust on is expected.


> rhel9.2 has this issue, rhel8.6 has no this issue.

Which versions of dpdk and kernel respectively are you referring to in both cases?

Comment 2 liting 2023-04-14 01:48:17 UTC
(In reply to David Marchand from comment #1)
> (In reply to liting from comment #0)
> > Actual results:
> > when the vf is trust on, the mac address will be changed when start testpmd.
> > 
> > Expected results:
> > when the vf no matter is trust on or trust off, the mac address should not
> > be changed when start testpmd command.
> 
> The VF mac address change is handled by the PF (kernel) driver, so it is
> best our kernel driver guys have a look at this bz (Stefan?).
> 
> At least, from my perspective, allowing to change the mac address with trust
> on is expected.
> 
> 
> > rhel9.2 has this issue, rhel8.6 has no this issue.
> 
> Which versions of dpdk and kernel respectively are you referring to in both
> cases?

For rhel9.2 test version:
kernel 5.14.0-284.8.1.el9_2
dpdk-21.11-1.el9_0
For rhel8.6 test version:
4.18.0-372.40.1.el8_6
dpdk-21.11-1.el8

Comment 3 Stefan Assmann 2023-04-14 13:16:12 UTC
It looks like this is the result of i40e upstream commit
fed0d9f13266 i40e: Fix VF's MAC Address change on VM

    Clear VF MAC from parent PF and remove VF filter from VSI when both
    conditions are true:
    -VIRTCHNL_VF_OFFLOAD_USO is not used
    -VM MAC was not set from PF level

    It affects older version of IAVF and it allow them to change MAC
    Address on VM, newer IAVF won't change their behaviour.

    Previously it wasn't possible to change VF's MAC Address on VM
    because there is flag on IAVF driver that won't allow to
    change MAC Address if this address is given from PF driver.

I assume iavf DPDK does not advertise VIRTCHNL_VF_OFFLOAD_USO. Then with this patch the old MAC address gets set back to zero and iavf generates a new random one. We don't see this in RHEL as iavf advertises VIRTCHNL_VF_OFFLOAD_USO. I'm not quite sure what's the reason behind this change to be honest.