Bug 1297954 - Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way [NEEDINFO]
Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0...
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch-dpdk (Show other bugs)
7.2
x86_64 Linux
high Severity high
: rc
: ---
Assigned To: Kevin Traynor
Jean-Tsung Hsiao
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-01-12 15:35 EST by Jean-Tsung Hsiao
Modified: 2016-11-01 02:46 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-10-05 10:36:19 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
kzhang: needinfo? (jhsiao)


Attachments (Terms of Use)

  None (edit)
Description Jean-Tsung Hsiao 2016-01-12 15:35:54 EST
Description of problem: Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way

The original issue happened in a OSP8-dpdk environment. When two OSP8-dpdk vhostusers were sending 64 bytes packets to each other, the throughput rate achieved 5.4 Mpps with port rate set at 35%. But, when one port got stooped, the throughput rate dropped to about .6 Mpps in about 6 minutes.

Similar behaviour happened to a OVS-dpdk environment. When running with two way pktgen 64 bytes traffic, each direction reached to a 4.6 Mpps rate. But, after stopping traffic for one of the two directions, the rate of the other direction went down to about 1.6 Mpps in about 6 minutes.

Version-Release number of selected component (if applicable):
[root@netqe5 jhsiao]# uname -a
Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@netqe5 jhsiao]# rpm -qa | grep dpdk
pktgen-dpdk-2.9.2-2.el7.centos.x86_64
openvswitch-dpdk-2.4.0-0.10346.git97bab959.2.el7.x86_64
dpdk-tools-2.1.0-5.el7.x86_64
dpdk-2.1.0-5.el7.x86_64


How reproducible: Reproducible


Steps to Reproduce: 
Two hosts are needed.
1. Configure a vhostuser for each host.
2. Have 4 CPU's pinned for each vhostuser.
3. Run pktgen 64 bytes tests between them with two way traffic
4. Rate on each direction, if tuned correctly, is expected to reach 4.6 Mpps for OVS-dpdk env, or 5.4 Mpps for OSP8-dpdk env. These are the numbers obtained in our test-beds.
5. Stop traffic for one of the two directions; wait for about 6 minutes; the traffic rate for the other direction would reduced significantly as described above.

Actual results:

See the description above.

Expected results:

The one way throughput rate should not degrade.

Additional info:
Comment 2 Flavio Leitner 2016-06-08 09:33:13 EDT
Hi Jean,

Could you check if this still happens with 2.5?  A lot has changed regarding to DPDK and this problem might be fixed already.

Thanks,
fbl
Comment 3 Jean-Tsung Hsiao 2016-07-07 15:59:28 EDT
(In reply to Flavio Leitner from comment #2)
> Hi Jean,
> 
> Could you check if this still happens with 2.5?  A lot has changed regarding
> to DPDK and this problem might be fixed already.
> 
> Thanks,
> fbl

Just installed Panu's pktgen and dpdk:

dpdk-16.04-1.el7.centos.x86_64
pktgen-dpdk-2.9.16-1.el7.centos.x86_64

But, my pktgen script failed:

Copyright (c) <2010-2016>, Intel Corporation. All rights reserved. Powered by Intel® DPDK
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL:   no supported IOMMU extensions found!
EAL: VFIO support could not be initialized
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1000000 bytes
EAL: Virtual area found at 0x7f98f4e00000 (size = 0x1000000)
EAL: Ask a virtual area of 0x48800000 bytes
EAL: Virtual area found at 0x7f98ac400000 (size = 0x48800000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f98ac000000 (size = 0x200000)
EAL: Ask a virtual area of 0x36000000 bytes
EAL: Virtual area found at 0x7f9875e00000 (size = 0x36000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875600000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875200000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~3399997 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ixgbe.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_e1000.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_null.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_fm10k.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vhost.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_enic.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_pcap.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_i40e.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_cxgbe.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vmxnet3_uio.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_af_packet.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ring.so.2
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bond.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_virtio.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ena.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bnx2x.so.1
EAL: Master lcore 0 is ready (tid=fc9f88c0;cpuset=[0])
EAL: lcore 2 is ready (tid=f3dfd700;cpuset=[2])
EAL: lcore 1 is ready (tid=f45fe700;cpuset=[1])
EAL: lcore 3 is ready (tid=f35fc700;cpuset=[3])
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: Error - exiting with code: 1


Please advise!

Thanks!

Jean
Comment 4 Yariv 2016-07-13 08:34:59 EDT
Hi
It is reproducible with DPDK 2.2 with OVS+DPDK 2.4 also
You could check NFV-QE perf machines.

Eyal
Comment 5 Flavio Leitner 2016-09-28 10:52:22 EDT
After chatting with Jean, he is now able to run pktgen-dpdk.
Please verify.
Comment 6 Jean-Tsung Hsiao 2016-10-03 13:07:09 EDT
(In reply to Flavio Leitner from comment #5)
> After chatting with Jean, he is now able to run pktgen-dpdk.
> Please verify.

Yes.
Comment 7 Flavio Leitner 2016-10-04 10:43:15 EDT
Jean,

Since you can run pktgen-dpdk now, could you check 2.5? See comment#2.
Thanks,
fbl
Comment 8 Jean-Tsung Hsiao 2016-10-05 08:18:39 EDT
Hi Flavio,

With 2.5 the issue is gone.

Using port 0 from each side got a 3.4 Mpps steady state rate for each direction.

Stopped one side traffic, got 3.7 Mpps steady state rate for single direction.

I would suggest closing this issue.

Thanks!

Jean
Comment 9 Flavio Leitner 2016-10-05 10:36:19 EDT
Thanks Jean! Appreciated it.
I am closing this based on comment#8.
Comment 10 Jean-Tsung Hsiao 2016-10-05 14:16:50 EDT
(In reply to Flavio Leitner from comment #9)
> Thanks Jean! Appreciated it.
> I am closing this based on comment#8.

NOTE: The 3.4 Mpps rate is a 26% drop from 4.6 Mpps as reported mentioned in the description. So, we have a significant degradation here.

Note You need to log in before you can comment on or make changes to this bug.