Bug 1297954

Summary: Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way
Product: Red Hat Enterprise Linux 7 Reporter: Jean-Tsung Hsiao <jhsiao>
Component: openvswitch-dpdkAssignee: Kevin Traynor <ktraynor>
Status: CLOSED CURRENTRELEASE QA Contact: Jean-Tsung Hsiao <jhsiao>
Severity: high Docs Contact:
Priority: high    
Version: 7.2CC: aloughla, atragler, fbaudin, fleitner, jhladky, jhsiao, kzhang, mleitner, osabart, rcain, rkhan, yrachman
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-05 14:36:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jean-Tsung Hsiao 2016-01-12 20:35:54 UTC
Description of problem: Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way

The original issue happened in a OSP8-dpdk environment. When two OSP8-dpdk vhostusers were sending 64 bytes packets to each other, the throughput rate achieved 5.4 Mpps with port rate set at 35%. But, when one port got stooped, the throughput rate dropped to about .6 Mpps in about 6 minutes.

Similar behaviour happened to a OVS-dpdk environment. When running with two way pktgen 64 bytes traffic, each direction reached to a 4.6 Mpps rate. But, after stopping traffic for one of the two directions, the rate of the other direction went down to about 1.6 Mpps in about 6 minutes.

Version-Release number of selected component (if applicable):
[root@netqe5 jhsiao]# uname -a
Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@netqe5 jhsiao]# rpm -qa | grep dpdk
pktgen-dpdk-2.9.2-2.el7.centos.x86_64
openvswitch-dpdk-2.4.0-0.10346.git97bab959.2.el7.x86_64
dpdk-tools-2.1.0-5.el7.x86_64
dpdk-2.1.0-5.el7.x86_64


How reproducible: Reproducible


Steps to Reproduce: 
Two hosts are needed.
1. Configure a vhostuser for each host.
2. Have 4 CPU's pinned for each vhostuser.
3. Run pktgen 64 bytes tests between them with two way traffic
4. Rate on each direction, if tuned correctly, is expected to reach 4.6 Mpps for OVS-dpdk env, or 5.4 Mpps for OSP8-dpdk env. These are the numbers obtained in our test-beds.
5. Stop traffic for one of the two directions; wait for about 6 minutes; the traffic rate for the other direction would reduced significantly as described above.

Actual results:

See the description above.

Expected results:

The one way throughput rate should not degrade.

Additional info:

Comment 2 Flavio Leitner 2016-06-08 13:33:13 UTC
Hi Jean,

Could you check if this still happens with 2.5?  A lot has changed regarding to DPDK and this problem might be fixed already.

Thanks,
fbl

Comment 3 Jean-Tsung Hsiao 2016-07-07 19:59:28 UTC
(In reply to Flavio Leitner from comment #2)
> Hi Jean,
> 
> Could you check if this still happens with 2.5?  A lot has changed regarding
> to DPDK and this problem might be fixed already.
> 
> Thanks,
> fbl

Just installed Panu's pktgen and dpdk:

dpdk-16.04-1.el7.centos.x86_64
pktgen-dpdk-2.9.16-1.el7.centos.x86_64

But, my pktgen script failed:

Copyright (c) <2010-2016>, Intel Corporation. All rights reserved. Powered by IntelĀ® DPDK
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL:   no supported IOMMU extensions found!
EAL: VFIO support could not be initialized
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1000000 bytes
EAL: Virtual area found at 0x7f98f4e00000 (size = 0x1000000)
EAL: Ask a virtual area of 0x48800000 bytes
EAL: Virtual area found at 0x7f98ac400000 (size = 0x48800000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f98ac000000 (size = 0x200000)
EAL: Ask a virtual area of 0x36000000 bytes
EAL: Virtual area found at 0x7f9875e00000 (size = 0x36000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875600000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875200000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~3399997 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ixgbe.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_e1000.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_null.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_fm10k.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vhost.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_enic.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_pcap.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_i40e.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_cxgbe.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vmxnet3_uio.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_af_packet.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ring.so.2
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bond.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_virtio.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ena.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bnx2x.so.1
EAL: Master lcore 0 is ready (tid=fc9f88c0;cpuset=[0])
EAL: lcore 2 is ready (tid=f3dfd700;cpuset=[2])
EAL: lcore 1 is ready (tid=f45fe700;cpuset=[1])
EAL: lcore 3 is ready (tid=f35fc700;cpuset=[3])
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: Error - exiting with code: 1


Please advise!

Thanks!

Jean

Comment 4 Yariv 2016-07-13 12:34:59 UTC
Hi
It is reproducible with DPDK 2.2 with OVS+DPDK 2.4 also
You could check NFV-QE perf machines.

Eyal

Comment 5 Flavio Leitner 2016-09-28 14:52:22 UTC
After chatting with Jean, he is now able to run pktgen-dpdk.
Please verify.

Comment 6 Jean-Tsung Hsiao 2016-10-03 17:07:09 UTC
(In reply to Flavio Leitner from comment #5)
> After chatting with Jean, he is now able to run pktgen-dpdk.
> Please verify.

Yes.

Comment 7 Flavio Leitner 2016-10-04 14:43:15 UTC
Jean,

Since you can run pktgen-dpdk now, could you check 2.5? See comment#2.
Thanks,
fbl

Comment 8 Jean-Tsung Hsiao 2016-10-05 12:18:39 UTC
Hi Flavio,

With 2.5 the issue is gone.

Using port 0 from each side got a 3.4 Mpps steady state rate for each direction.

Stopped one side traffic, got 3.7 Mpps steady state rate for single direction.

I would suggest closing this issue.

Thanks!

Jean

Comment 9 Flavio Leitner 2016-10-05 14:36:19 UTC
Thanks Jean! Appreciated it.
I am closing this based on comment#8.

Comment 10 Jean-Tsung Hsiao 2016-10-05 18:16:50 UTC
(In reply to Flavio Leitner from comment #9)
> Thanks Jean! Appreciated it.
> I am closing this based on comment#8.

NOTE: The 3.4 Mpps rate is a 26% drop from 4.6 Mpps as reported mentioned in the description. So, we have a significant degradation here.