1637893 – Backport "VF Tx error issue" fix to OvS 2.10

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1637893 - Backport "VF Tx error issue" fix to OvS 2.10

Summary: Backport "VF Tx error issue" fix to OvS 2.10

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	openvswitch2.10
Sub Component:
Version:	7.7
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	pre-dev-freeze
Target Release:	---
Assignee:	Timothy Redaelli
QA Contact:	Jiying Qiu
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-10-10 09:51 UTC by Karthik Sundaravel
Modified:	2019-01-07 08:51 UTC (History)
CC List:	14 users (show)
Fixed In Version:	openvswitch2.10-2.10.0-19.el7fdn
Doc Type:	Bug Fix
Doc Text:	Cause: When using kernel PF + DPDK VF, if setting VLAN strip on or off in VF side after setting VLAN for VF with ethtool in PF side. Consequence: VF Tx error is generated Fix: This patch fixes the issue by check VLAN offload capability when setting VLAN offload.
Clone Of:
Environment:
Last Closed:	2019-01-07 08:51:12 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
captured tag icmp packets in vhost0 (6.33 KB, application/octet-stream) 2018-11-30 07:01 UTC, Jiying Qiu	no flags	Details
log for failedQA (17.78 KB, text/plain) 2018-12-26 08:30 UTC, qding	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2019:0029	0	None	None	None	2019-01-07 08:51:23 UTC

Description Karthik Sundaravel 2018-10-10 09:51:03 UTC

Need backport of DPDK patch http://mails.dpdk.org/archives/stable/2017-December/003856.html in OvS 2.10

The actual issue is discussed in BZ 1628811

Comment 4 Jiying Qiu 2018-11-16 06:17:11 UTC

Hi Karthik,

As BZ 1628811 mentioned, NIC partitioning is needed. But in X710 & X520 nic card, I did not see the NPAR settings. Do you mean NPAR? Or sr-iov? 

Jiying Qiu

Comment 5 Karthik Sundaravel 2018-11-16 06:38:19 UTC

Hi

We use the SR-IOV feature of these cards. We enable SR-IOV on these cards. What we mean NIC partitioning is, each VF is configured for different networks in host. Some VF's could be bound with DPDK drivers, while other VF's could be added as slaves to a linux bond.

Comment 6 Jiying Qiu 2018-11-29 06:09:06 UTC

Hi Timothy,

Using Rhel7.6 ,openvswitch2.10-2.10.0-28.el7fdp.x86_64 , X520 ixgbe card. And specific configuration is as following.

Ⅰ
echo 2 > /sys/bus/pci/devices/0000\:05\:00.1/sriov_numvfs
p7p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether a0:36:9f:ab:36:c6 brd ff:ff:ff:ff:ff:ff
    vf 0 MAC f0:8e:38:c4:69:11, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 MAC f0:8e:38:c4:69:01, spoof checking off, link-state auto, trust on, query_rss off

Ⅱ
# ovs-vsctl get Open_vSwitch . other_config
{dpdk-init="true", dpdk-lcore-mask="0x4", dpdk-socket-mem="1024", pmd-cpu-mask="0xff003fc00", vhost-iommu-support="true"}

Ⅲ
p7p2 vf 1 pci 0000:05:10.3 bond to dpdk,and add this vf port to openvswitch

Ⅳ
# ovs-vsctl show 
5c9bd021-0bf6-45df-8458-dea91cf5c3b8
    Bridge "ovsbr0"
        Port "dpdk0"
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:05:10.3"}
        Port "vhost0"
            Interface "vhost0"
                type: dpdkvhostuserclient
                options: {n_rxq="1", vhost-server-path="/tmp/vhost0"}
        Port "ovsbr0"
            Interface "ovsbr0"
                type: internal
    ovs_version: "2.10.0"

V
create a vm. And the vhost0 is seting as the xml shows.
   <interface type='vhostuser'>
      <mac address='f0:8e:38:c4:69:01'/>
      <source type='unix' path='/tmp/vhost0' mode='server'/>
      <target dev='vhost0'/>
      <model type='virtio'/>
      <driver name='vhost' iommu='on' ats='on'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>

Ⅵ
in the vm, add a vlan port. and add a ipv4 address to vlan 
eth1.10@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f0:8e:38:c4:69:01 brd ff:ff:ff:ff:ff:ff
    inet 10.167.100.2/24 scope global eth1.10
       valid_lft forever preferred_lft forever
    inet6 fe80::f28e:38ff:fec4:6901/64 scope link 
       valid_lft forever preferred_lft forever

Ⅶ
In the remote server ,just add a vlan port and add same network ipv4 address.
p7p2.10@p7p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a0:36:9f:ab:36:16 brd ff:ff:ff:ff:ff:ff
    inet 10.167.100.12/24 scope global p7p2.10
       valid_lft forever preferred_lft forever
    inet6 fe80::a236:9fff:feab:3616/64 scope link 
       valid_lft forever preferred_lft forever

Result:
In vm , ping remote is fail.
tcpdump -i dpdk0 can not capture arp reply packet.

If there is some wrong setting or this problem is not this bug,please let me know,I will correct it and make a new bug.If not, I think this bug is still exist.

Thanks,
Jiying Qiu

Comment 7 Timothy Redaelli 2018-11-29 09:35:16 UTC

Hi,

this bugzilla is about backporting a patch to fix VLAN offloading in i40e, but in your setup you are using ixgbe.

Karthik, do you have some advice?

Thank you

Comment 8 Karthik Sundaravel 2018-11-29 09:54:17 UTC

Hi,

I am using this patch with X710 card (i40e). On X520/x540 cards this path is not meaningful. I hope it clarifies

Regards
Karthik S

Comment 9 Jiying Qiu 2018-11-30 06:45:42 UTC

Hi ,

Have tried X710 card. Settings as C#6, In vm,ping remote is success,but it has almost 50% loss.

[root@localhost ~]# ping 10.167.100.12
PING 10.167.100.12 (10.167.100.12) 56(84) bytes of data.
64 bytes from 10.167.100.12: icmp_seq=2 ttl=64 time=0.518 ms
64 bytes from 10.167.100.12: icmp_seq=4 ttl=64 time=0.553 ms
64 bytes from 10.167.100.12: icmp_seq=6 ttl=64 time=0.506 ms
64 bytes from 10.167.100.12: icmp_seq=9 ttl=64 time=0.561 ms

--- 10.167.100.12 ping statistics ---
9 packets transmitted, 4 received, 55% packet loss, time 8003ms
rtt min/avg/max/mdev = 0.506/0.534/0.561/0.032 ms

And tcpdump at vhost0 , have captured two direction packets.

[root@dell-per730-51 ~]# ovs-tcpdump -i vhost0 -ne icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on mivhost0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:41:35.630275 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 1, length 64
01:41:35.630367 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 1, length 64
01:41:36.630358 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 2, length 64
01:41:36.630510 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 2, length 64
01:41:37.631401 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 3, length 64
01:41:37.631481 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 3, length 64
01:41:38.631485 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 4, length 64
01:41:38.631559 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 4, length 64
01:41:39.632303 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 5, length 64
01:41:39.632383 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 5, length 64
01:41:40.632474 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 6, length 64
01:41:40.632549 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 6, length 64
01:41:41.633331 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 7, length 64
01:41:41.633408 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 7, length 64
01:41:42.633348 00:00:00:01:04:01 > f8:f2:1e:02:c4:a0, ethertype IPv4 (0x0800), length 98: 10.167.100.2 > 10.167.100.12: ICMP echo request, id 12355, seq 8, length 64
01:41:42.633424 f8:f2:1e:02:c4:a0 > 00:00:00:01:04:01, ethertype IPv4 (0x0800), length 98: 10.167.100.12 > 10.167.100.2: ICMP echo reply, id 12355, seq 8, length 64
^C16 packets captured
16 packets received by filter
0 packets dropped by kernel

BTW ,qemu verion :qemu-kvm-rhev-2.12.0-19.el7.x86_64
do you have any idea?

Thanks
Jiying Qiu

Comment 10 Jiying Qiu 2018-11-30 07:01:42 UTC

Created attachment 1510070 [details]
captured tag icmp packets in vhost0

Please refer to the attachment ,packets captured in vhost0 with tag.

Comment 12 Jiying Qiu 2018-12-10 02:05:34 UTC

Hi ,

We QE didn't write this doc before.Please refer to http://mails.dpdk.org/archives/stable/2017-December/003856.html for details.

Thanks,
Jiying Qiu

Comment 13 Jiying Qiu 2018-12-10 02:09:42 UTC

Hi Timothy,

Could you write the doc as C#11 asked?

Thanks.
Jiying Qiu

Comment 15 Jiying Qiu 2018-12-20 05:45:10 UTC

Since X710 card support vf backport vlan traffic, so set verified.

Comment 16 Jiying Qiu 2018-12-20 10:43:59 UTC

Sorry ,we will do research of this bug again. C#9 is not the reproducer.

Comment 17 qding 2018-12-21 09:44:24 UTC

Hi Karthik,

We QE cannot reproduce the issue even with reference of bz#1628811. Can you please give us a more detailed reproducer? especially the topology for the test is expected. We don't have similar test yet to verify the issue. Please help.

Thanks
Qijun

Comment 18 qding 2018-12-26 08:30:17 UTC

Created attachment 1516769 [details]
log for failedQA

Test with openvswitch2.10-2.10.0-28.el7fdp.1.x86_64.rpm, but failed. Please see the attached log for failedQA

Comment 19 qding 2018-12-26 08:39:36 UTC

Hi Karthik,

We use the steps below to do the verification, but after doing "ethtool -K p4p1 rxvlan off" ping still fail. Please help check if the steps can verify the issue. Please note that reload pf driver is important of the first several steps to clean the old config. Thanks.

Two hosts are connected back-to-back

on dell-per730-51.rhts.eng.pek2.redhat.com

ovs-vsctl del-br br0
systemctl stop openvswitch
ip link set p4p1 vf 0 vlan 0
driverctl unset-override 0000:82:02.0
driverctl unset-override 0000:82:02.1
modprobe -r i40e
modprobe  i40e

echo 2 > /sys/bus/pci/devices/0000:82:00.0/sriov_numvfs
ip link show
ip link set p4p1 vf 0 mac 00:00:00:00:00:01 spoofchk off
ip link show p4p1
driverctl set-override 0000:82:02.0 vfio-pci
driverctl set-override 0000:82:02.1 vfio-pci
dpdk-devbind -s
ip link show p4p1
systemctl start openvswitch
ovs-vsctl get Open_vSwitch . other_config
ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:82:02.0 ofport_request=10
ovs-vsctl show
ifconfig br0 up
ip add add 192.168.124.1/24 dev br0
ping 192.168.124.2
ip link set p4p1 vf 0 vlan 10
ping 192.168.124.2
ethtool -k p4p1 | grep vlan
ethtool -K p4p1 rxvlan off
ping 192.168.124.2


The remote host:

[root@dell-per730-55 rpmbuild]# ifconfig p6p1.10
p6p1.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.124.2  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::faf2:1eff:fe02:c4a0  prefixlen 64  scopeid 0x20<link>
        ether f8:f2:1e:02:c4:a0  txqueuelen 1000  (Ethernet)
        RX packets 495  bytes 36564 (35.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 437  bytes 38498 (37.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@dell-per730-55 rpmbuild]# 


Thanks
Qijun

Comment 20 Karthik Sundaravel 2018-12-26 09:50:59 UTC

(In reply to qding from comment #19)
> Hi Karthik,
> 
> We use the steps below to do the verification, but after doing "ethtool -K
> p4p1 rxvlan off" ping still fail. Please help check if the steps can verify
> the issue. Please note that reload pf driver is important of the first
> several steps to clean the old config. Thanks.
> 
> Two hosts are connected back-to-back
> 
> on dell-per730-51.rhts.eng.pek2.redhat.com
> 
> ovs-vsctl del-br br0
> systemctl stop openvswitch
> ip link set p4p1 vf 0 vlan 0
> driverctl unset-override 0000:82:02.0
> driverctl unset-override 0000:82:02.1
> modprobe -r i40e
> modprobe  i40e
> 
> echo 2 > /sys/bus/pci/devices/0000:82:00.0/sriov_numvfs
> ip link show
> ip link set p4p1 vf 0 mac 00:00:00:00:00:01 spoofchk off
> ip link show p4p1
> driverctl set-override 0000:82:02.0 vfio-pci
> driverctl set-override 0000:82:02.1 vfio-pci
> dpdk-devbind -s
> ip link show p4p1
> systemctl start openvswitch
> ovs-vsctl get Open_vSwitch . other_config
> ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev
> ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk
> options:dpdk-devargs=0000:82:02.0 ofport_request=10
> ovs-vsctl show
> ifconfig br0 up
> ip add add 192.168.124.1/24 dev br0
> ping 192.168.124.2
> ip link set p4p1 vf 0 vlan 10
> ping 192.168.124.2
> ethtool -k p4p1 | grep vlan
> ethtool -K p4p1 rxvlan off
> ping 192.168.124.2
> 
> 
> The remote host:
> 
> [root@dell-per730-55 rpmbuild]# ifconfig p6p1.10
> p6p1.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 192.168.124.2  netmask 255.255.255.0  broadcast 0.0.0.0
>         inet6 fe80::faf2:1eff:fe02:c4a0  prefixlen 64  scopeid 0x20<link>
>         ether f8:f2:1e:02:c4:a0  txqueuelen 1000  (Ethernet)
>         RX packets 495  bytes 36564 (35.7 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 437  bytes 38498 (37.5 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> 
> [root@dell-per730-55 rpmbuild]# 
> 
> 
> Thanks
> Qijun

Can you please check if the command " ip link set p4p1 vf 0 vlan 10" is done before adding the port dpdk0 to br0

Comment 22 qding 2018-12-26 10:19:53 UTC

(In reply to Karthik Sundaravel from comment #20)
> Can you please check if the command " ip link set p4p1 vf 0 vlan 10" is done
> before adding the port dpdk0 to br0

Please ignore my comment#21. Sorry I looked the wrong line.
we exactly do the test in the steps described in comment#19.

"ip link set p4p1 vf 0 vlan 10" is AFTER adding the dpdk0

Comment 23 qding 2019-01-04 06:16:31 UTC

New bug bz1663373 is created for the issue in comment#19
This bug is set to VERIFIED.

Comment 25 errata-xmlrpc 2019-01-07 08:51:12 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0029

Note You need to log in before you can comment on or make changes to this bug.