Bug 2231081

Does it work with a recent OVS 3.1?
Thanks
fbl

Looks like this is caused by the following commit:

 * 5d11c47d3 userspace: Enable IP checksum offloading by default.

Needs a follow up why.

David figured out where the problem was in DPDK, and submitted the following patch:

https://patchwork.dpdk.org/project/dpdk/patch/20230818090351.2402519-1-david.marchand@redhat.com/

I tried it on @liting's setup, and it solved the problem.

We need to wait for the patch to make it to 22.11.x

The net/iavf driver in DPDK is faulty, in that, it was evaluating some fields in the mbuf because of an (unneeded) flag present in ol_flags.
For this, a fix has been sent in DPDK and merged in next-net-intel, as it could affect other DPDK applications.
https://patchwork.dpdk.org/project/dpdk/patch/20230823062911.2483926-1-david.marchand@redhat.com/


But, out of safety (in case other DPDK drivers are showing the same bug), a OVS change upstream now clears the (unneeded) flag before calling the DPDK drivers.
This change has been backported to OVS 3.2 upstream branch, so hopefully, it will get picked up with our downstream robot, soon.
https://github.com/openvswitch/ovs/commit/9b7e1a75378f

The fix has been integrated in openvswitch3.2-3.2.0-10.el9fdp, can you please verify.

(In reply to Eelco Chaudron from comment #11)
> The fix has been integrated in openvswitch3.2-3.2.0-10.el9fdp, can you
> please verify.

I run the case with openvswitch3.2-3.2.0-10.el9fdp, it still not work.
https://beaker.engineering.redhat.com/jobs/8347458

This is odd, David verified it on his setup with E810 cards, and it works fine there. Can you make your setup available to me, so I can take another look?

(In reply to Eelco Chaudron from comment #13)
> This is odd, David verified it on his setup with E810 cards, and it works
> fine there. Can you make your setup available to me, so I can take another
> look?

I run case and it also not work well on e810 card.
e810 card job.
https://beaker.engineering.redhat.com/jobs/8403091
I have the test env of i40e card. You can access to them have a look. i40e card of dell730-52 <--directly connect---> i40e card of dell730-53. 
dell-per730-52.rhts.eng.pek2.redhat.com(root/redhat)
dell-per730-53.rhts.eng.pek2.redhat.com(root/redhat)

I was able to replicate the same issue on my local system with an Intel E810 card, where I looped back the two ports with a cable, so all I needed was a single machine. Here are my instructions to replicate it:


Setup NIC partitioning, and the create VFs:
===========================================

  lshw -c network -businfo
  Bus info          Device     Class          Description
  =======================================================
  pci@0000:31:00.0  eno12399   network        Ethernet Controller E810-XXV for SFP
  pci@0000:31:00.1  eno12409   network        Ethernet Controller E810-XXV for SFP

  echo 0 > /sys/class/net/eno12399/device/sriov_numvfs
  echo 2 > /sys/class/net/eno12399/device/sriov_numvfs


  lshw -c network -businfo
  Bus info          Device      Class          Description
  ========================================================
  pci@0000:31:00.0  eno12399    network        Ethernet Controller E810-XXV for SFP
  pci@0000:31:00.1  eno12409    network        Ethernet Controller E810-XXV for SFP
  pci@0000:31:01.0  eno12399v0  network        Ethernet Adaptive Virtual Function
  pci@0000:31:01.1  eno12399v1  network        Ethernet Adaptive Virtual Function


Setup the second vf (vf 1):
===========================

  ip link show dev eno12399
  4: eno12399: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
      link/ether b4:83:51:01:ce:52 brd ff:ff:ff:ff:ff:ff
      vf 0     link/ether aa:c4:23:98:85:0a brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
      vf 1     link/ether c2:9c:1c:e8:dc:f2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
      altname enp49s0f0

  ip link set dev eno12399 vf 1 spoofchk on trust on

  ip link show dev eno12399
  4: eno12399: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
      link/ether b4:83:51:01:ce:52 brd ff:ff:ff:ff:ff:ff
      vf 0     link/ether aa:c4:23:98:85:0a brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
      vf 1     link/ether c2:9c:1c:e8:dc:f2 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust on
      altname enp49s0f0


Setup openvswitch + driverctl:
==============================

  driverctl -v set-override 0000:31:01.1 vfio-pci

  ovs-vsctl set Open_vSwitch . other_config:dpdk-socket-mem=2048
  ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x4000000040000000
  ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true

  ovs-vsctl del-br ovs_pvp_br0

  ovs-vsctl add-br ovs_pvp_br0 -- \
            set bridge ovs_pvp_br0 datapath_type=netdev

  ovs-vsctl add-port ovs_pvp_br0 dpdk0p0 -- \
            set Interface dpdk0p0 type=dpdk -- \
            set Interface dpdk0p0 options:dpdk-devargs=0000:31:01.1 -- \
            set interface dpdk0p0 options:n_rxq=2

  ovs-vsctl add-port ovs_pvp_br0 vhost0 -- \
            set Interface vhost0 type=dpdkvhostuserclient -- \
            set Interface vhost0 options:vhost-server-path='/tmp/vhost-sock0' -- \
            set Interface vhost0 options:n_rxq=2


Start VM and get it's MAC address:
==================================

  virsh start rhel9_loopback
  virsh console rhel9_loopback

  driverctl unset-override 0000:01:00.0

  ip a show dev eth0
  4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
      link/ether 52:54:00:9e:e6:3d brd ff:ff:ff:ff:ff:ff
      altname enp1s0
      inet 1.1.1.1/24 brd 1.1.1.255 scope global noprefixroute eth0
         valid_lft forever preferred_lft forever


Set the VFs MAC to the VM one:
==============================

  ip link set eno12399 vf 1 mac 52:54:00:9e:e6:3d



Add IP to OVS bridge to make sure the client can ping it:
=========================================================

  ip a a 1.1.1.10/24 dev ovs_pvp_br0

  ip link set ovs_pvp_br0 up

  ping 1.1.1.1
  PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
  64 bytes from 1.1.1.1: icmp_seq=1 ttl=64 time=0.196 ms
  64 bytes from 1.1.1.1: icmp_seq=2 ttl=64 time=0.060 ms
  64 bytes from 1.1.1.1: icmp_seq=3 ttl=64 time=0.043 ms
  ^C
  --- 1.1.1.1 ping statistics ---
  3 packets transmitted, 3 received, 0% packet loss, time 2047ms
  rtt min/avg/max/mdev = 0.043/0.099/0.196/0.068 ms

  ovs-appctl dpctl/dump-flows
  flow-dump from the main thread:
  recirc_id(0),in_port(2),packet_type(ns=0,id=0),eth(src=a6:53:f9:9f:44:4e,dst=52:54:00:9e:e6:3d),eth_type(0x0800),ipv4(frag=no), packets:3, bytes:294, used:1.530s, actions:4
  flow-dump from pmd on cpu core: 62
  recirc_id(0),in_port(4),packet_type(ns=0,id=0),eth(src=52:54:00:9e:e6:3d,dst=a6:53:f9:9f:44:4e),eth_type(0x0800),ipv4(frag=no), packets:3, bytes:294, used:1.530s, actions:2

  [VM] ping 1.1.1.10
       PING 1.1.1.10 (1.1.1.10) 56(84) bytes of data.
       64 bytes from 1.1.1.10: icmp_seq=1 ttl=64 time=0.137 ms
       64 bytes from 1.1.1.10: icmp_seq=2 ttl=64 time=0.061 ms
       64 bytes from 1.1.1.10: icmp_seq=3 ttl=64 time=0.052 ms

       --- 1.1.1.10 ping statistics ---
       3 packets transmitted, 3 received, 0% packet loss, time 2063ms
       rtt min/avg/max/mdev = 0.052/0.083/0.137/0.038 ms


We looped port 1 and 2 of the NIC, so setup net_ns to ping the other port:
==========================================================================
  ip netns add ns_remote

  ip link set eno12409 netns ns_remote

  ip -n ns_remote link set dev lo up
  ip -n ns_remote link set dev eno12409 up
  ip -n ns_remote address add 1.1.1.100/24 dev eno12409


Now we should be able to ping from the VM to the ns_remote IP:
==============================================================

  However, this is not working due to the issue at hand :(

  [root@loopback ~]# ping 1.1.1.100
  PING 1.1.1.100 (1.1.1.100) 56(84) bytes of data.

  ^C
  --- 1.1.1.100 ping statistics ---
  6763 packets transmitted, 0 received, 100% packet loss, time 6924274ms

So it looks like there was a problem with the internal build process, which is now fixed. The openvswitch3.2-3.2.0-17.el9fdp.x86_64.rpm version should work. I did verify this on my test setup with E810 cards. Please verify this on your setup.

(In reply to Eelco Chaudron from comment #18)
> So it looks like there was a problem with the internal build process, which
> is now fixed. The openvswitch3.2-3.2.0-17.el9fdp.x86_64.rpm version should
> work. I did verify this on my test setup with E810 cards. Please verify this
> on your setup.

The ipv4 ping successfully on openvswitch3.2-3.2.0-17.el9fdp.x86_64. Following is the job. 
https://beaker.engineering.redhat.com/jobs/8440484
But the ipv6 still ping failed. It has following known issue. So this bug should be verify pass.
https://bugzilla.redhat.com/show_bug.cgi?id=1921462

Closing this BZ as the issue is solved.