Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1358010

Summary: Xena traffic dropped 10.1% at the ingress dpdk port while running Xena to OVS-dpdk 4-Q vhostuser/testpmd 64 bytes test
Product: Red Hat Enterprise Linux 7 Reporter: Jean-Tsung Hsiao <jhsiao>
Component: openvswitch-dpdkAssignee: Flavio Leitner <fleitner>
Status: CLOSED DUPLICATE QA Contact: Jean-Tsung Hsiao <jhsiao>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: aloughla, atragler, fleitner, jhsiao, kzhang, rcain
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-12-21 15:24:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jean-Tsung Hsiao 2016-07-19 20:05:42 UTC 
Description of problem: 

Primary issue --- Xena traffic dropped 10.1% at the ingress dpdk port while running Xena to OVS-dpdk 4-Q vhostuser/testpmd 64 bytes test

Due this issue the maximum throughput rate is capped at about 13.5 Mpps.

Question: What could cause the drop? Is there a way to reduce the dropped to a minimum ?

Secondary issue --- Under RHEL7.3 kernel-447 the drop rate is 50% at 10.1%, 50% at 2.8% --- out of four tests. As a result the throughput rate is either about 13.5 Mpps or about 14.4 Mpps. Among four tests 

Question: Why two distinct drop rates ? If we can get to the bottom of this issue, we might be able to find a way to solve the drop issue.


Version-Release number of selected component (if applicable):

dpdk-tools-2.2.0-3.el7.x86_64
openvswitch-dpdk-2.5.0-4.el7.x86_64

kernel-447

How reproducible: Reproducible


Steps to Reproduce:
1. Xena 10 G traffic generator to OVS-dpdk 4-Q vhostuser testpmd over ixgbe dpdk0 and dpdk1.
2. Run Flavio's multiple_streams 64 bytes 32/64 streams, 60 seconds duration.
3.

Actual results:


Expected results:


Additional info:
Comment 2 Flavio Leitner 2016-07-28 18:25:12 UTC 
Hi Jean,

The fluctuation is most probably because something is running on a CPU that is busy processing packets.  That can be the OVS PMD thread or the qemu thread in the host that is being preempted or the vCPU in the guest that is being preempted too.

Can you double check?

fbl
Comment 3 Jean-Tsung Hsiao 2016-08-10 02:05:18 UTC 
How to check the preemtion?

Please advise!

Thanks!

Jean
Comment 4 Jean-Tsung Hsiao 2016-08-11 01:58:21 UTC 
(In reply to Jean-Tsung Hsiao from comment #3)
> How to check the preemtion?
> 
> Please advise!
> 
> Thanks!
> 
> Jean
 
I'll check ctxt from status for voluntary_ctxt_switches and nonvoluntary_ctxt_switches.
Comment 5 Jean-Tsung Hsiao 2016-08-16 13:33:40 UTC 
 * The following perf command sequence indicates that lots of context switches inside CPU #17 --- one of the four pmd cores.

perf record -e sched:sched_switch -C 17  sleep 10; perf report

*** Below is the tuna data:

[root@netqe5 jhsiao]# tuna -t ovs-vswitchd -CP
                      thread       ctxt_switches
    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd 
 6625    OTHER     0       17    162302          274    ovs-vswitchd  
  6720   OTHER     0       17         1            0       handler76  
  6626   OTHER     0       17     13991            0  dpdk_watchdog3  
  6627   OTHER     0       17     70221            0   vhost_thread1  
  6628   OTHER     0       17      9060            1           urcu2  
  6726   OTHER     0       17         5       316085           pmd82  
  6721   OTHER     0       17    280650            0   revalidator77  
  6728   OTHER     0       21         2            7           pmd83  
  6729   OTHER     0       23         1           10           pmd84  
  6727   OTHER     0       19         2        69927           pmd81  
  6722   OTHER     0       17    420793            0   revalidator78  
  6724   OTHER     0       17    280547            0   revalidator80  
  6723   OTHER     0       17    420780            0   revalidator79  
  6713   OTHER     0       17         1            0       handler74  
  6714   OTHER     0       17         1            0       handler69  
  6715   OTHER     0       17         1            0       handler72  
  6716   OTHER     0       17         1            0       handler70  
  6717   OTHER     0       17         1            0       handler71  
  6718   OTHER     0       17         1            0       handler73  
  6719   OTHER     0       17         1            0       handler75 

* Using the following script to remove all context switches from the four pmd cores does not improve the Xena to testpmd 64 bytes Mpps.

NON_PMD_CPU_RANGE="13,15"
OVS_PID="$(pidof ovs-vswitchd)"
for pid in $(ps -e -T  | grep ${OVS_PID} | grep -v 'pmd' | awk '{ print $2 }')
do
    taskset -p -c ${NON_PMD_CPU_RANGE} ${pid}
done
 
tuna -t ovs-vswitchd -CP
Comment 6 Flavio Leitner 2016-09-29 14:13:27 UTC 
Jean, pmd82 is running on CPU 17 with many other threads and would explain the issue.  You need to run that script after OVS has been configured otherwise new threads might fall into the wrong CPU.

It looks like the isolcpus isn't being used as well.
Comment 7 Jean-Tsung Hsiao 2016-10-04 17:06:37 UTC 
(In reply to Flavio Leitner from comment #6)
> Jean, pmd82 is running on CPU 17 with many other threads and would explain
> the issue.  You need to run that script after OVS has been configured
> otherwise new threads might fall into the wrong CPU.
> 
> It looks like the isolcpus isn't being used as well.

Hi Flavio,

The high nonvoluntary context switches for CPU 17 was obtained before the tuning. After running that tuning script(comment #5)  it's gone. But, the throughput rate didn't improve --- it remained at about 13.6 Mpps. I would think tunnings like this probably will only improve latency.

I'll run the test again with ovs-2.5.0-5 --- with and without tunning.

NOTE: To improve Mpps throughput substantially we definitely need the following enhancement:

[Bug 1378586] RFE: Inconsistent 64 bytes Xena to testpmd/ vhostuser throughput caused by mixed queues in a pmd core
Comment 8 Jean-Tsung Hsiao 2016-10-05 17:48:37 UTC 
Hi Flavio,

As I promised above I re-ran with isolcpus and other tunnings. The Mpps throughput rate remains at 13.6 Mpps.

Please see below for context switch data --- before and after the test.

Also, see below for tunning steps and scripts.

Please let me know for questions and suggestions.

Thanks!

Jean

== context switches before the test ==

[root@netqe5 dpdk-multique-scripts]# tuna -t ovs-vswitchd -CP
                      thread       ctxt_switches
    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd
 5968    OTHER     0    13,15     14936          194    ovs-vswitchd  
  6752   OTHER     0    13,15      3384          519   revalidator75  
  6753   OTHER     0    13,15      4001            0   revalidator76  
  6754   OTHER     0    13,15      3907          238   revalidator77  
  6755   OTHER     0    13,15      4120           23   revalidator78  
  6757   OTHER     0       17       365            5           pmd81  
  6758   OTHER     0       19       360            7           pmd82  
  6759   OTHER     0       21       416            5           pmd83  
  6760   OTHER     0       23       403            6           pmd84  
  5969   OTHER     0    13,15       805            0  dpdk_watchdog3  
  5970   OTHER     0    13,15      4039            0   vhost_thread1  
  5971   OTHER     0    13,15      4567            0           urcu2  
  6744   OTHER     0    13,15         1            0       handler70  
  6745   OTHER     0    13,15         1            0       handler80  
  6746   OTHER     0    13,15         1            1       handler79  
  6747   OTHER     0    13,15         1            0       handler69  
  6748   OTHER     0    13,15         1            0       handler71  
  6749   OTHER     0    13,15         1            0       handler72  
  6750   OTHER     0    13,15         1            0       handler73  
  6751   OTHER     0    13,15         1            0       handler74 

[root@netqe5 dpdk-multique-scripts]#  tuna -t qemu-kvm -CP
                      thread       ctxt_switches
    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd
 6809    OTHER     0      1,3     10218            3        qemu-kvm  
  6851   OTHER     0      1,3       106            0        qemu-kvm  
  6855   OTHER     0        0      9774            4       CPU 0/KVM  
  6856   OTHER     0        5      2681            1       CPU 1/KVM  
  6857   OTHER     0        7      2164            1       CPU 2/KVM  
  6858   OTHER     0        9      2675            1       CPU 3/KVM  
  6859   OTHER     0       11      2248            1       CPU 4/KVM  
  6861   OTHER     0      1,3         1            0      vnc_worker  
[root@netqe5 dpdk-multique-scripts]#
[root@netqe5 dpdk-multique-scripts]#
[root@netqe5 dpdk-multique-scripts]#
[root@netqe5 dpdk-multique-scripts]# #running the multiple_streams test

[root@netqe5 XenaScripts]# python multiple_streams 1000000 64 32 10
rate_fration =  1000000
packet_length 64
num_of_streams =  32
test_duration =  10
INFO:root:XenaSocket: Connected
INFO:root:XenaManager: Logged succefully
INFO:root:XenaPort: 1/0 starting traffic
INFO:root:XenaPort: 1/0 stopping traffic
Average: 13600300.00 pps

== context switches after the test ==

[root@netqe5 dpdk-multique-scripts]#
[root@netqe5 dpdk-multique-scripts]#
[root@netqe5 dpdk-multique-scripts]#  tuna -t qemu-kvm -CP
                      thread       ctxt_switches
    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd
 6809    OTHER     0      1,3     10231            3        qemu-kvm  
  6851   OTHER     0      1,3       106            0        qemu-kvm  
  6855   OTHER     0        0     10200            4       CPU 0/KVM  
  6856   OTHER     0        5      2681            1       CPU 1/KVM  
  6857   OTHER     0        7      2164            1       CPU 2/KVM  
  6858   OTHER     0        9      2675            1       CPU 3/KVM  
  6859   OTHER     0       11      2248            1       CPU 4/KVM  
  6861   OTHER     0      1,3         1            0      vnc_worker  
[root@netqe5 dpdk-multique-scripts]# tuna -t ovs-vswitchd -CP
                      thread       ctxt_switches
    pid SCHED_ rtpri affinity voluntary nonvoluntary             cmd
 5968    OTHER     0    13,15     15060          194    ovs-vswitchd  
  6752   OTHER     0    13,15      3766          519   revalidator75  
  6753   OTHER     0    13,15      4412           81   revalidator76  
  6754   OTHER     0    13,15      4377          280   revalidator77  
  6755   OTHER     0    13,15      4653           23   revalidator78  
  6757   OTHER     0       17       395            5           pmd81  
  6758   OTHER     0       19       384            7           pmd82  
  6759   OTHER     0       21       460            5           pmd83  
  6760   OTHER     0       23       426            6           pmd84  
  5969   OTHER     0    13,15       815            0  dpdk_watchdog3  
  5970   OTHER     0    13,15      4090            0   vhost_thread1  
  5971   OTHER     0    13,15      4646            0           urcu2  
  6744   OTHER     0    13,15         1            0       handler70  
  6745   OTHER     0    13,15         1            0       handler80  
  6746   OTHER     0    13,15         1            1       handler79  
  6747   OTHER     0    13,15         1            0       handler69  
  6748   OTHER     0    13,15         1            0       handler71  
  6749   OTHER     0    13,15         1            0       handler72  
  6750   OTHER     0    13,15         1            0       handler73  
  6751   OTHER     0    13,15         1            0       handler74

== Build and reboot the kernel with isolcpus=5,7,9,11,17,19,21,23 ==

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-510.el7.x86_64 root=/dev/mapper/rhel_netqe5-root ro isolcpus=5,7,9,11,17,19,21,23 intel_iommu=on default_hugepagesz=1GB hugepagesz=1G hugepages=32 crashkernel=auto rd.lvm.lv=rhel_netqe5/root rd.lvm.lv=rhel_netqe5/swap console=ttyS1,115200

NOTE: CPU 5,7,9,11 are for the vhostuser guest. And CPU 17,19,21,23 are for pmd cores.

== Disable kernel watchdog timer ==
echo 0 > /proc/sys/kernel/watchdog

== Run the the following script right before test ==

[root@netqe5 dpdk-multique-scripts]# cat ../set_non_pmd_cpu_range.sh
NON_PMD_CPU_RANGE="13,15"
OVS_PID="$(pidof ovs-vswitchd)"
for pid in $(ps -e -T  | grep ${OVS_PID} | grep -v 'pmd' | awk '{ print $2 }')
do
    taskset -p -c ${NON_PMD_CPU_RANGE} ${pid}
done
 
tuna -t ovs-vswitchd -CP

== Run the the following script right before test  ==
 
[root@netqe5 dpdk-multique-scripts]# cat ../set_non_kvm_cpu_range.sh
NON_KVM_CPU_RANGE="1,3"
OVS_PID="$(pidof qemu-kvm)"
for pid in $(ps -e -T  | grep ${OVS_PID} | grep -v 'CPU' | awk '{ print $2 }')
do
    taskset -p -c ${NON_KVM_CPU_RANGE} ${pid}
done
 
tuna -t qemu-kvm -CP
 
== pmd cores and queues alignment --- collected after openvswitch and OVS dpdk bridge stared ==

[root@netqe5 dpdk-multique-scripts]# ovs-appctl dpif-netdev/pmd-rxq-show
pmd thread numa_id 1 core_id 19:
    port: dpdk0    queue-id: 1
    port: dpdk1    queue-id: 0
    port: vhost0    queue-id: 0
    port: vhost1    queue-id: 0
pmd thread numa_id 1 core_id 17:
    port: dpdk0    queue-id: 0
    port: dpdk1    queue-id: 1
    port: vhost0    queue-id: 1
    port: vhost1    queue-id: 1
pmd thread numa_id 1 core_id 21:
    port: dpdk0    queue-id: 2
    port: dpdk1    queue-id: 2
    port: vhost0    queue-id: 2
    port: vhost1    queue-id: 2
pmd thread numa_id 1 core_id 23:
    port: dpdk0    queue-id: 3
    port: dpdk1    queue-id: 3
    port: vhost0    queue-id: 3
    port: vhost1    queue-id: 3

== Other data ==

root@netqe5 dpdk-multique-scripts]# rpm -qa | grep openvswitch
openvswitch-2.5.0-5.git20160628.el7fdb.x86_64
[root@netqe5 dpdk-multique-scripts]# uname -a
Linux netqe5.knqe.lab.eng.bos.redhat.com 3.10.0-510.el7.x86_64 #1 SMP Wed Sep 21 14:44:14 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Comment 9 Flavio Leitner 2016-12-03 22:19:49 UTC 
Jean,

Based on comment#8, this looks like this is a dup of bz#1378586 https://bugzilla.redhat.com/show_bug.cgi?id=1378586#c16

What do you think?
Comment 10 Jean-Tsung Hsiao 2016-12-04 04:26:29 UTC 
(In reply to Flavio Leitner from comment #9)
> Jean,
> 
> Based on comment#8, this looks like this is a dup of bz#1378586
> https://bugzilla.redhat.com/show_bug.cgi?id=1378586#c16
> 
> What do you think?

Yes.

Thanks!

Jean
Comment 11 Flavio Leitner 2016-12-21 15:24:12 UTC 

*** This bug has been marked as a duplicate of bug 1378586 ***