Bug 1733402 - i40e nic: spoofchk off does not take effect when vf add to ovs bridge
Summary: i40e nic: spoofchk off does not take effect when vf add to ovs bridge
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Enterprise Linux Fast Datapath
Classification: Red Hat
Component: openvswitch2.11
Version: FDP 19.C
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Eelco Chaudron
QA Contact: qding
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-26 02:32 UTC by liting
Modified: 2020-02-07 07:41 UTC (History)
7 users (show)

Fixed In Version: openvswitch2.11-2.11.0-36.el7fdn
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1796586 (view as bug list)
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description liting 2019-07-26 02:32:58 UTC
Description of problem:
spoofchk off does not take effect when vf add to ovs bridge

Version-Release number of selected component (if applicable):
[root@dell-per730-52 ~]# uname -a
Linux dell-per730-52.rhts.eng.pek2.redhat.com 4.18.0-80.el8.x86_64 #1 SMP Wed Mar 13 12:02:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[root@dell-per730-52 ~]# rpm -qa|grep openv
openvswitch2.11-2.11.0-12.el8fdp.x86_64
openvswitch-selinux-extra-policy-1.0-12.el8fdp.noarch

[root@dell-per730-52 ~]# rpm -qa|grep dpdk
dpdk-18.11-8.el8.x86_64
dpdk-tools-18.11-8.el8.x86_64

[root@dell-per730-52 ~]# ethtool -i enp7s0f0
driver: i40e
version: 2.3.2-k
firmware-version: 6.01 0x800034a2 1.1747.0
expansion-rom-version: 
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[root@dell-per730-52 ~]# lspci|grep XXV710
07:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
07:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)

[root@dell-per730-53 ~]# ethtool -i enp4s0f0
driver: i40e
version: 2.3.2-k
firmware-version: 6.01 0x80003554 1.1747.0
expansion-rom-version: 
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[root@dell-per730-53 ~]#  lspci|grep XXV710
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)
04:00.1 Ethernet controller: Intel Corporation Ethernet Controller XXV710 for 25GbE SFP28 (rev 02)


How reproducible:


Steps to Reproduce:
Two servers, dell52 and dell53, they connect back to back.
1. On both servers, create vf for physic port. 
   bind the vf to dpdk. 
   add the dpdk0 to the ovs bridge.
        /usr/share/dpdk/usertools/dpdk-devbind.py -b vfio-pci 0000:04:02.1
        systemctl restart openvswitch
        ovs-vsctl set Open_vSwitch . other_config={}
        ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true
        ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024,1024"
        ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x50000005000000
        systemctl restart openvswitch
        ovs-vsctl add-br ovsbr1 -- set bridge ovsbr1 datapath_type=netdev
        config_vf_mac_tunnel
        ovs-vsctl add-port ovsbr1 dpdk0 -- set Interface dpdk0 type=dpdk type=dpdk options:dpdk-devargs=0000:04:02.1
        ovs-vsctl add-port ovsbr0 dpdkvhostuserclient0 -- set Interface dpdkvhostuserclient0 type=dpdkvhostuserclient -- set Interface dpdkvhostuserclient0 options:vhost-server-path=/tmp/dpdkvhostuserclient0

[root@dell-per730-53 ~]# ovs-vsctl show
b0e055d0-3b5f-4112-97a0-019275a3a74e
    Bridge "ovsbr0"
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:04:02.1"}
        Port "dpdkvhostuserclient0"
            Interface "dpdkvhostuserclient0"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/tmp/dpdkvhostuserclient0"}
    ovs_version: "2.11.0"


2.  Configure the vf mac same with the guest's eth0 mac
ip link set enp4s0f0 vf 1 mac 52:54:00:11:8f:e9
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:11:8f:e9 brd ff:ff:ff:ff:ff:ff
    inet 20.0.0.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:5c0:9168::2/24 scope global 
       valid_lft forever preferred_lft forever

3.  Start a guest, the guest xml as following
[root@dell-per730-53 ~]# virsh dumpxml master3
<domain type='kvm' id='1'>
  <name>master3</name>
  <uuid>37425e76-af6a-44a6-aba0-73434afe34c0</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>5242880</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>3</vcpu>
  <cputune>
    <vcpupin vcpu='2' cpuset='0'/>
    <emulatorpin cpuset='2'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <feature policy='require' name='tsc-deadline'/>
    <numa>
      <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/>
    </numa>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/master3.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='vhostuser'>
      <mac address='52:54:00:11:8f:e9'/>
      <source type='unix' path='/tmp/dpdkvhostuserclient0' mode='server'/>
      <target dev='dpdkvhostuserclient0'/>
      <model type='virtio'/>
      <driver name='vhost'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-master3/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c102,c689</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c102,c689</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+1001</label>
    <imagelabel>+107:+1001</imagelabel>
  </seclabel>
</domain>

4. Change spoofchk to off, run the ping test inside guest. 
Inside guest of dell52, configure the ip 20.0.0.1/24
Inside guest of dell53, configure the ip 20.0.0.2/24 
Inside guest of dell53, ping 20.0.0.1.

5. On dell53, configure the vf 1 mac different from the guest's eth0 mac and restart ovs service. Run the ping test.
ip link set p5p1 vf 1 mac 52:54:00:11:8f:e0
systemctl restart openvswitch
ping 20.0.0.1

Actual results:
Step 4 ping successfully: configure spoofchk off, when vf mac same with guest's eth0 mac, ping successfully
Step 5 ping failed, configure spoofchk off, when vf mac different with guest's eth0 mac, ping failed

Expected results:
Step 5 should ping successfully, configure spoofchk off, when vf mac different with guest's eth0 mac, ping should successfully. 
When run ping test, the packet src mac is the guest's eth0 mac. when vf configure spoofchk off, it should not check the guest's eth0 mac whether same with vf's mac, and ping should successfully.


Additional info:

Comment 1 Eelco Chaudron 2019-08-05 14:36:37 UTC
This looks like a duplicate to me for https://bugzilla.redhat.com/show_bug.cgi?id=1719644, can you verify if you get any of the odd DPDK messages like this: https://mails.dpdk.org/archives/dev/2019-June/135593.html

Comment 2 liting 2019-08-13 08:22:02 UTC
(In reply to Eelco Chaudron from comment #1)
> This looks like a duplicate to me for
> https://bugzilla.redhat.com/show_bug.cgi?id=1719644, can you verify if you
> get any of the odd DPDK messages like this:
> https://mails.dpdk.org/archives/dev/2019-June/135593.html

No, it is not duplicate with bug1719644. After restart openvswitch service, ping still failed when spoofchk is off(vf mac different with guest's eth0 mac). There is no the dpdk messages like following log. The same issue also exist on ixgbe nic.
2019-06-24T15:16:55.052Z|00001|dpdk|ERR|i40evf_handle_aq_msg(): Request 0 is not supported yet 

thanks,
Li Ting

Comment 3 Eelco Chaudron 2019-08-20 12:25:48 UTC
When experimenting with a fix for BZ1719644 I was able to replicate the issue and confirm it's not solved by the fix fixing BZ1719644.

I've asked Intel for any hints to why disabling spoofchk and changing the MAC to a different mac is still not passing traffic. Odd is that even after restarting OVS the issue persists. Making me think this is kernel driver related and not DPDK.

Comment 4 Eelco Chaudron 2019-09-05 08:52:01 UTC
Did some more research and the spoof check works as designed, however, it's for packet ingressing into the VF from the host. In the reverse direction packets, no matching the configured MAC are dropped. To also enable those packets to pass you have to enable trust mode, i.e. "ip link set dev eth0 vf 1 trust on".

Having said this (and tested) this still does not work on the fly, once you enable trusted mode you still need to restart OVS. I can see with the patch for BZ1719644 the reset function is called, but it does not have the desired effect.

I'll do further research, and touch base with Intel.

Comment 5 Eelco Chaudron 2019-09-17 07:49:26 UTC
Sent fix to DPDK mailing list as it's an internal state issue:

http://mails.dpdk.org/archives/dev/2019-September/143683.html

With this patch and the patch for BZ1719644 this will fix the problem.

Comment 6 Eelco Chaudron 2019-10-31 12:48:01 UTC
(In reply to Eelco Chaudron from comment #5)
> Sent fix to DPDK mailing list as it's an internal state issue:
> 
> http://mails.dpdk.org/archives/dev/2019-September/143683.html

Still, some discussion is going on around this patch...

Comment 7 Eelco Chaudron 2019-11-19 13:50:08 UTC
Sent v2 patch:

http://mails.dpdk.org/archives/dev/2019-November/151687.html


Note You need to log in before you can comment on or make changes to this bug.