Bug 1637893

Hi Karthik,

As BZ 1628811 mentioned, NIC partitioning is needed. But in X710 & X520 nic card, I did not see the NPAR settings. Do you mean NPAR? Or sr-iov? 

Jiying Qiu

Hi

We use the SR-IOV feature of these cards. We enable SR-IOV on these cards. What we mean NIC partitioning is, each VF is configured for different networks in host. Some VF's could be bound with DPDK drivers, while other VF's could be added as slaves to a linux bond.

Hi Timothy,

Using Rhel7.6 ,openvswitch2.10-2.10.0-28.el7fdp.x86_64 , X520 ixgbe card. And specific configuration is as following.

Ⅰ
echo 2 > /sys/bus/pci/devices/0000\:05\:00.1/sriov_numvfs
p7p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:ab:36:c6 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC f0:8e:38:c4:69:11, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC f0:8e:38:c4:69:01, spoof checking off, link-state auto, trust on, query_rss off

Ⅱ
# ovs-vsctl get Open_vSwitch . other_config
{dpdk-init="true", dpdk-lcore-mask="0x4", dpdk-socket-mem="1024", pmd-cpu-mask="0xff003fc00", vhost-iommu-support="true"}

Ⅲ
p7p2 vf 1 pci 0000:05:10.3 bond to dpdk,and add this vf port to openvswitch

Ⅳ
# ovs-vsctl show 
5c9bd021-0bf6-45df-8458-dea91cf5c3b8
    Bridge "ovsbr0"
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:05:10.3"}
        Port "vhost0"
            Interface "vhost0"
                type: dpdkvhostuserclient
                options: {n_rxq="1", vhost-server-path="/tmp/vhost0"}
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    ovs_version: "2.10.0"

V
create a vm. And the vhost0 is seting as the xml shows.
   <interface type='vhostuser'>
      <mac address='f0:8e:38:c4:69:01'/>
      <source type='unix' path='/tmp/vhost0' mode='server'/>
      <target dev='vhost0'/>
      <model type='virtio'/>
      <driver name='vhost' iommu='on' ats='on'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>

Ⅵ
in the vm, add a vlan port. and add a ipv4 address to vlan 
eth1.10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:8e:38:c4:69:01 brd ff:ff:ff:ff:ff:ff
    inet 10.167.100.2/24 scope global eth1.10
       valid_lft forever preferred_lft forever
    inet6 fe80::f28e:38ff:fec4:6901/64 scope link 
       valid_lft forever preferred_lft forever

Ⅶ
In the remote server ,just add a vlan port and add same network ipv4 address.
p7p2.10@p7p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a0:36:9f:ab:36:16 brd ff:ff:ff:ff:ff:ff
    inet 10.167.100.12/24 scope global p7p2.10
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:feab:3616/64 scope link 
       valid_lft forever preferred_lft forever

Result:
In vm , ping remote is fail.
tcpdump -i dpdk0 can not capture arp reply packet.

If there is some wrong setting or this problem is not this bug,please let me know,I will correct it and make a new bug.If not, I think this bug is still exist.

Thanks,
Jiying Qiu

Hi,

this bugzilla is about backporting a patch to fix VLAN offloading in i40e, but in your setup you are using ixgbe.

Karthik, do you have some advice?

Thank you

Hi,

I am using this patch with X710 card (i40e). On X520/x540 cards this path is not meaningful. I hope it clarifies

Regards
Karthik S

Hi ,

Have tried X710 card. Settings as C#6, In vm,ping remote is success,but it has almost 50% loss.

[root@localhost ~]# ping 10.167.100.12
PING 10.167.100.12 (10.167.100.12) 56(84) bytes of data.
64 bytes from 10.167.100.12: icmp_seq=2 ttl=64 time=0.518 ms
64 bytes from 10.167.100.12: icmp_seq=4 ttl=64 time=0.553 ms
64 bytes from 10.167.100.12: icmp_seq=6 ttl=64 time=0.506 ms
64 bytes from 10.167.100.12: icmp_seq=9 ttl=64 time=0.561 ms

--- 10.167.100.12 ping statistics ---
9 packets transmitted, 4 received, 55% packet loss, time 8003ms
rtt min/avg/max/mdev = 0.506/0.534/0.561/0.032 ms

And tcpdump at vhost0 , have captured two direction packets.

[root@dell-per730-51 ~]# ovs-tcpdump -i vhost0 -ne icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mivhost0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:41:35.630275 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 1, length 64
01:41:35.630367 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 1, length 64
01:41:36.630358 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 2, length 64
01:41:36.630510 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 2, length 64
01:41:37.631401 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 3, length 64
01:41:37.631481 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 3, length 64
01:41:38.631485 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 4, length 64
01:41:38.631559 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 4, length 64
01:41:39.632303 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 5, length 64
01:41:39.632383 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 5, length 64
01:41:40.632474 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 6, length 64
01:41:40.632549 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 6, length 64
01:41:41.633331 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 7, length 64
01:41:41.633408 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 7, length 64
01:41:42.633348 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 8, length 64
01:41:42.633424 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 8, length 64
^C16 packets captured
16 packets received by filter
0 packets dropped by kernel

BTW ,qemu verion :qemu-kvm-rhev-2.12.0-19.el7.x86_64
do you have any idea?

Thanks
Jiying Qiu

Created attachment 1510070 [details]
captured tag icmp packets in vhost0

Please refer to the attachment ,packets captured in vhost0 with tag.

Hi ,

We QE didn't write this doc before.Please refer to http://mails.dpdk.org/archives/stable/2017-December/003856.html for details.

Thanks,
Jiying Qiu

Hi Timothy,

Could you write the doc as C#11 asked?

Thanks.
Jiying Qiu

Since X710 card support vf backport vlan traffic, so set verified.

Sorry ,we will do research of this bug again. C#9 is not the reproducer.

Hi Karthik,

We QE cannot reproduce the issue even with reference of bz#1628811. Can you please give us a more detailed reproducer? especially the topology for the test is expected. We don't have similar test yet to verify the issue. Please help.

Thanks
Qijun

Created attachment 1516769 [details]
log for failedQA

Test with openvswitch2.10-2.10.0-28.el7fdp.1.x86_64.rpm, but failed. Please see the attached log for failedQA

Hi Karthik,

We use the steps below to do the verification, but after doing "ethtool -K p4p1 rxvlan off" ping still fail. Please help check if the steps can verify the issue. Please note that reload pf driver is important of the first several steps to clean the old config. Thanks.

Two hosts are connected back-to-back

on dell-per730-51.rhts.eng.pek2.redhat.com

ovs-vsctl del-br br0
systemctl stop openvswitch
ip link set p4p1 vf 0 vlan 0
driverctl unset-override 0000:82:02.0
driverctl unset-override 0000:82:02.1
modprobe -r i40e
modprobe  i40e

echo 2 > /sys/bus/pci/devices/0000:82:00.0/sriov_numvfs
ip link show
ip link set p4p1 vf 0 mac 00:00:00:00:00:01 spoofchk off
ip link show p4p1
driverctl set-override 0000:82:02.0 vfio-pci
driverctl set-override 0000:82:02.1 vfio-pci
dpdk-devbind -s
ip link show p4p1
systemctl start openvswitch
ovs-vsctl get Open_vSwitch . other_config
ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:82:02.0 ofport_request=10
ovs-vsctl show
ifconfig br0 up
ip add add 192.168.124.1/24 dev br0
ping 192.168.124.2
ip link set p4p1 vf 0 vlan 10
ping 192.168.124.2
ethtool -k p4p1 | grep vlan
ethtool -K p4p1 rxvlan off
ping 192.168.124.2


The remote host:

[root@dell-per730-55 rpmbuild]# ifconfig p6p1.10
p6p1.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.124.2  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::faf2:1eff:fe02:c4a0  prefixlen 64  scopeid 0x20<link>
        ether f8:f2:1e:02:c4:a0  txqueuelen 1000  (Ethernet)
        RX packets 495  bytes 36564 (35.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 437  bytes 38498 (37.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@dell-per730-55 rpmbuild]# 


Thanks
Qijun

(In reply to qding from comment #19)
> Hi Karthik,
> 
> We use the steps below to do the verification, but after doing "ethtool -K
> p4p1 rxvlan off" ping still fail. Please help check if the steps can verify
> the issue. Please note that reload pf driver is important of the first
> several steps to clean the old config. Thanks.
> 
> Two hosts are connected back-to-back
> 
> on dell-per730-51.rhts.eng.pek2.redhat.com
> 
> ovs-vsctl del-br br0
> systemctl stop openvswitch
> ip link set p4p1 vf 0 vlan 0
> driverctl unset-override 0000:82:02.0
> driverctl unset-override 0000:82:02.1
> modprobe -r i40e
> modprobe  i40e
> 
> echo 2 > /sys/bus/pci/devices/0000:82:00.0/sriov_numvfs
> ip link show
> ip link set p4p1 vf 0 mac 00:00:00:00:00:01 spoofchk off
> ip link show p4p1
> driverctl set-override 0000:82:02.0 vfio-pci
> driverctl set-override 0000:82:02.1 vfio-pci
> dpdk-devbind -s
> ip link show p4p1
> systemctl start openvswitch
> ovs-vsctl get Open_vSwitch . other_config
> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
> ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0000:82:02.0 ofport_request=10
> ovs-vsctl show
> ifconfig br0 up
> ip add add 192.168.124.1/24 dev br0
> ping 192.168.124.2
> ip link set p4p1 vf 0 vlan 10
> ping 192.168.124.2
> ethtool -k p4p1 | grep vlan
> ethtool -K p4p1 rxvlan off
> ping 192.168.124.2
> 
> 
> The remote host:
> 
> [root@dell-per730-55 rpmbuild]# ifconfig p6p1.10
> p6p1.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 192.168.124.2  netmask 255.255.255.0  broadcast 0.0.0.0
>         inet6 fe80::faf2:1eff:fe02:c4a0  prefixlen 64  scopeid 0x20<link>
>         ether f8:f2:1e:02:c4:a0  txqueuelen 1000  (Ethernet)
>         RX packets 495  bytes 36564 (35.7 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 437  bytes 38498 (37.5 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> [root@dell-per730-55 rpmbuild]# 
> 
> 
> Thanks
> Qijun

Can you please check if the command " ip link set p4p1 vf 0 vlan 10" is done before adding the port dpdk0 to br0

(In reply to Karthik Sundaravel from comment #20)
> Can you please check if the command " ip link set p4p1 vf 0 vlan 10" is done
> before adding the port dpdk0 to br0

Please ignore my comment#21. Sorry I looked the wrong line.
we exactly do the test in the steps described in comment#19.

"ip link set p4p1 vf 0 vlan 10" is AFTER adding the dpdk0

New bug bz1663373 is created for the issue in comment#19
This bug is set to VERIFIED.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0029