Hide Forgot
Description of problem: Throughput rate of pktgen 64 bytes tests between two VHU's dropped to below 0.6 Mpps from 5+ Mpps when two way traffic reduced to one way The original issue happened in a OSP8-dpdk environment. When two OSP8-dpdk vhostusers were sending 64 bytes packets to each other, the throughput rate achieved 5.4 Mpps with port rate set at 35%. But, when one port got stooped, the throughput rate dropped to about .6 Mpps in about 6 minutes. Similar behaviour happened to a OVS-dpdk environment. When running with two way pktgen 64 bytes traffic, each direction reached to a 4.6 Mpps rate. But, after stopping traffic for one of the two directions, the rate of the other direction went down to about 1.6 Mpps in about 6 minutes. Version-Release number of selected component (if applicable): [root@netqe5 jhsiao]# uname -a Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux [root@netqe5 jhsiao]# rpm -qa | grep dpdk pktgen-dpdk-2.9.2-2.el7.centos.x86_64 openvswitch-dpdk-2.4.0-0.10346.git97bab959.2.el7.x86_64 dpdk-tools-2.1.0-5.el7.x86_64 dpdk-2.1.0-5.el7.x86_64 How reproducible: Reproducible Steps to Reproduce: Two hosts are needed. 1. Configure a vhostuser for each host. 2. Have 4 CPU's pinned for each vhostuser. 3. Run pktgen 64 bytes tests between them with two way traffic 4. Rate on each direction, if tuned correctly, is expected to reach 4.6 Mpps for OVS-dpdk env, or 5.4 Mpps for OSP8-dpdk env. These are the numbers obtained in our test-beds. 5. Stop traffic for one of the two directions; wait for about 6 minutes; the traffic rate for the other direction would reduced significantly as described above. Actual results: See the description above. Expected results: The one way throughput rate should not degrade. Additional info:
Hi Jean, Could you check if this still happens with 2.5? A lot has changed regarding to DPDK and this problem might be fixed already. Thanks, fbl
(In reply to Flavio Leitner from comment #2) > Hi Jean, > > Could you check if this still happens with 2.5? A lot has changed regarding > to DPDK and this problem might be fixed already. > > Thanks, > fbl Just installed Panu's pktgen and dpdk: dpdk-16.04-1.el7.centos.x86_64 pktgen-dpdk-2.9.16-1.el7.centos.x86_64 But, my pktgen script failed: Copyright (c) <2010-2016>, Intel Corporation. All rights reserved. Powered by Intel® DPDK EAL: Detected lcore 0 as core 0 on socket 0 EAL: Detected lcore 1 as core 0 on socket 0 EAL: Detected lcore 2 as core 0 on socket 0 EAL: Detected lcore 3 as core 0 on socket 0 EAL: Support maximum 128 logical core(s) by configuration. EAL: Detected 4 lcore(s) EAL: Probing VFIO support... EAL: IOMMU type 1 (Type 1) is not supported EAL: IOMMU type 8 (No-IOMMU) is not supported EAL: no supported IOMMU extensions found! EAL: VFIO support could not be initialized EAL: Setting up physically contiguous memory... EAL: Ask a virtual area of 0x1000000 bytes EAL: Virtual area found at 0x7f98f4e00000 (size = 0x1000000) EAL: Ask a virtual area of 0x48800000 bytes EAL: Virtual area found at 0x7f98ac400000 (size = 0x48800000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f98ac000000 (size = 0x200000) EAL: Ask a virtual area of 0x36000000 bytes EAL: Virtual area found at 0x7f9875e00000 (size = 0x36000000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f9875a00000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f9875600000 (size = 0x200000) EAL: Ask a virtual area of 0x200000 bytes EAL: Virtual area found at 0x7f9875200000 (size = 0x200000) EAL: Requesting 512 pages of size 2MB from socket 0 EAL: TSC frequency is ~3399997 KHz EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using unreliable clock cycles ! EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ixgbe.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_e1000.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_null.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_fm10k.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vhost.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_enic.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_pcap.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_i40e.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_cxgbe.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_vmxnet3_uio.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_af_packet.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ring.so.2 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bond.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_virtio.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_ena.so.1 EAL: open shared lib /usr/lib64/dpdk-pmds/librte_pmd_bnx2x.so.1 EAL: Master lcore 0 is ready (tid=fc9f88c0;cpuset=[0]) EAL: lcore 2 is ready (tid=f3dfd700;cpuset=[2]) EAL: lcore 1 is ready (tid=f45fe700;cpuset=[1]) EAL: lcore 3 is ready (tid=f35fc700;cpuset=[3]) EAL: PCI device 0000:00:03.0 on NUMA socket -1 EAL: probe driver: 1af4:1000 rte_virtio_pmd EAL: Not managed by a supported kernel driver, skipped EAL: Error - exiting with code: 1 Please advise! Thanks! Jean
Hi It is reproducible with DPDK 2.2 with OVS+DPDK 2.4 also You could check NFV-QE perf machines. Eyal
After chatting with Jean, he is now able to run pktgen-dpdk. Please verify.
(In reply to Flavio Leitner from comment #5) > After chatting with Jean, he is now able to run pktgen-dpdk. > Please verify. Yes.
Jean, Since you can run pktgen-dpdk now, could you check 2.5? See comment#2. Thanks, fbl
Hi Flavio, With 2.5 the issue is gone. Using port 0 from each side got a 3.4 Mpps steady state rate for each direction. Stopped one side traffic, got 3.7 Mpps steady state rate for single direction. I would suggest closing this issue. Thanks! Jean
Thanks Jean! Appreciated it. I am closing this based on comment#8.
(In reply to Flavio Leitner from comment #9) > Thanks Jean! Appreciated it. > I am closing this based on comment#8. NOTE: The 3.4 Mpps rate is a 26% drop from 4.6 Mpps as reported mentioned in the description. So, we have a significant degradation here.