RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1297954 - Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way
Summary: Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch-dpdk
Version: 7.2
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Kevin Traynor
QA Contact: Jean-Tsung Hsiao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-01-12 20:35 UTC by Jean-Tsung Hsiao
Modified: 2022-10-31 14:20 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-05 14:36:19 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Jean-Tsung Hsiao 2016-01-12 20:35:54 UTC 
Description of problem: Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way

The original issue happened in a OSP8-dpdk environment. When two OSP8-dpdk vhostusers were sending 64 bytes packets to each other, the throughput rate achieved 5.4 Mpps with port rate set at 35%. But, when one port got stooped, the throughput rate dropped to about .6 Mpps in about 6 minutes.

Similar behaviour happened to a OVS-dpdk environment. When running with two way pktgen 64 bytes traffic, each direction reached to a 4.6 Mpps rate. But, after stopping traffic for one of the two directions, the rate of the other direction went down to about 1.6 Mpps in about 6 minutes.

Version-Release number of selected component (if applicable):
[root@netqe5 jhsiao]# uname -a
Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

[root@netqe5 jhsiao]# rpm -qa | grep dpdk
pktgen-dpdk-2.9.2-2.el7.centos.x86_64
openvswitch-dpdk-2.4.0-0.10346.git97bab959.2.el7.x86_64
dpdk-tools-2.1.0-5.el7.x86_64
dpdk-2.1.0-5.el7.x86_64


How reproducible: Reproducible


Steps to Reproduce: 
Two hosts are needed.
1. Configure a vhostuser for each host.
2. Have 4 CPU's pinned for each vhostuser.
3. Run pktgen 64 bytes tests between them with two way traffic
4. Rate on each direction, if tuned correctly, is expected to reach 4.6 Mpps for OVS-dpdk env, or 5.4 Mpps for OSP8-dpdk env. These are the numbers obtained in our test-beds.
5. Stop traffic for one of the two directions; wait for about 6 minutes; the traffic rate for the other direction would reduced significantly as described above.

Actual results:

See the description above.

Expected results:

The one way throughput rate should not degrade.

Additional info:
Comment 2 Flavio Leitner 2016-06-08 13:33:13 UTC 
Hi Jean,

Could you check if this still happens with 2.5?  A lot has changed regarding to DPDK and this problem might be fixed already.

Thanks,
fbl
Comment 3 Jean-Tsung Hsiao 2016-07-07 19:59:28 UTC 
(In reply to Flavio Leitner from comment #2)
> Hi Jean,
> 
> Could you check if this still happens with 2.5?  A lot has changed regarding
> to DPDK and this problem might be fixed already.
> 
> Thanks,
> fbl

Just installed Panu's pktgen and dpdk:

dpdk-16.04-1.el7.centos.x86_64
pktgen-dpdk-2.9.16-1.el7.centos.x86_64

But, my pktgen script failed:

Copyright (c) <2010-2016>, Intel Corporation. All rights reserved. Powered by Intel® DPDK
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Detected lcore 2 as core 0 on socket 0
EAL: Detected lcore 3 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL:   IOMMU type 1 (Type 1) is not supported
EAL:   IOMMU type 8 (No-IOMMU) is not supported
EAL:   no supported IOMMU extensions found!
EAL: VFIO support could not be initialized
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x1000000 bytes
EAL: Virtual area found at 0x7f98f4e00000 (size = 0x1000000)
EAL: Ask a virtual area of 0x48800000 bytes
EAL: Virtual area found at 0x7f98ac400000 (size = 0x48800000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f98ac000000 (size = 0x200000)
EAL: Ask a virtual area of 0x36000000 bytes
EAL: Virtual area found at 0x7f9875e00000 (size = 0x36000000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875600000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f9875200000 (size = 0x200000)
EAL: Requesting 512 pages of size 2MB from socket 0
EAL: TSC frequency is ~3399997 KHz
EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles !
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ixgbe.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_e1000.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_null.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_fm10k.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vhost.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_enic.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_pcap.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_i40e.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_cxgbe.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vmxnet3_uio.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_af_packet.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ring.so.2
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bond.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_virtio.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ena.so.1
EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bnx2x.so.1
EAL: Master lcore 0 is ready (tid=fc9f88c0;cpuset=[0])
EAL: lcore 2 is ready (tid=f3dfd700;cpuset=[2])
EAL: lcore 1 is ready (tid=f45fe700;cpuset=[1])
EAL: lcore 3 is ready (tid=f35fc700;cpuset=[3])
EAL: PCI device 0000:00:03.0 on NUMA socket -1
EAL:   probe driver: 1af4:1000 rte_virtio_pmd
EAL:   Not managed by a supported kernel driver, skipped
EAL: Error - exiting with code: 1


Please advise!

Thanks!

Jean
Comment 4 Yariv 2016-07-13 12:34:59 UTC 
Hi
It is reproducible with DPDK 2.2 with OVS+DPDK 2.4 also
You could check NFV-QE perf machines.

Eyal
Comment 5 Flavio Leitner 2016-09-28 14:52:22 UTC 
After chatting with Jean, he is now able to run pktgen-dpdk.
Please verify.
Comment 6 Jean-Tsung Hsiao 2016-10-03 17:07:09 UTC 
(In reply to Flavio Leitner from comment #5)
> After chatting with Jean, he is now able to run pktgen-dpdk.
> Please verify.

Yes.
Comment 7 Flavio Leitner 2016-10-04 14:43:15 UTC 
Jean,

Since you can run pktgen-dpdk now, could you check 2.5? See comment#2.
Thanks,
fbl
Comment 8 Jean-Tsung Hsiao 2016-10-05 12:18:39 UTC 
Hi Flavio,

With 2.5 the issue is gone.

Using port 0 from each side got a 3.4 Mpps steady state rate for each direction.

Stopped one side traffic, got 3.7 Mpps steady state rate for single direction.

I would suggest closing this issue.

Thanks!

Jean
Comment 9 Flavio Leitner 2016-10-05 14:36:19 UTC 
Thanks Jean! Appreciated it.
I am closing this based on comment#8.
Comment 10 Jean-Tsung Hsiao 2016-10-05 18:16:50 UTC 
(In reply to Flavio Leitner from comment #9)
> Thanks Jean! Appreciated it.
> I am closing this based on comment#8.

NOTE: The 3.4 Mpps rate is a 26% drop from 4.6 Mpps as reported mentioned in the description. So, we have a significant degradation here.
Note You need to log in before you can comment on or make changes to this bug.