Hide Forgot
Description of problem: OSPD 10 + OVS DPDK two ports (bidirectional) performance tests. OVS + DPDK two ports performance tests show unstable results. Version-Release number of selected component (if applicable): RHOS 10 Product version: 10 Product core version: 10 Product core build: 2016-10-21.3 openvswitch-2.5.0-14.git20160727.el7fdb.x86_64 tuned-2.7.1-4.el7fdb.noarch How reproducible: Run back to back performance test with moongen. Steps to Reproduce: 1. Deploy RHOS 10 OVS DPDK environment with two DPDK ports. 2. Connect Compute node back to back with Moongen. 3. Run performance tests. Actual results: Three runs were performed. The results of the runs: 1) [INFO] Device 0->1 PASSED - frame loss (0, 0.00000000%) is less than or equal to the maximum (0.00000000%) [INFO] Device 1->0 PASSED - frame loss (-79, -0.00009754%) is less than or equal to the maximum (0.00000000%) [INFO] Test Result: PASSED [PARAMETERS] startRate: 10.000000 nrFlows: 256 frameSize: 64 runBidirec: true searchRunTime: 30 validationRunTime: 30 acceptableLossPct: 0.000000 ports: 1,2 [REPORT]Device 0->1: Tx frames: 80986815 Rx Frames: 80986815 frame loss: 0, 0.000000% Rx Mpps: 2.699713 [REPORT]Device 1->0: Tx frames: 80990532 Rx Frames: 80990611 frame loss: -79, -0.000098% Rx Mpps: 2.699715 [REPORT] total: Tx frames: 161977347 Rx Frames: 161977426 frame loss: -79, -0.000049% Tx Mpps: 5.399426 Rx Mpps: 5.399428 2) [INFO] Device 0->1 PASSED - frame loss (0, 0.00000000%) is less than or equal to the maximum (0.00000000%) [INFO] Device 1->0 PASSED - frame loss (-69, -0.00013506%) is less than or equal to the maximum (0.00000000%) [INFO] Test Result: PASSED [PARAMETERS] startRate: 10.000000 nrFlows: 256 frameSize: 64 runBidirec: true searchRunTime: 30 validationRunTime: 30 acceptableLossPct: 0.000000 ports: 1,2 [REPORT]Device 0->1: Tx frames: 51088023 Rx Frames: 51088023 frame loss: 0, 0.000000% Rx Mpps: 1.702952 [REPORT]Device 1->0: Tx frames: 51087645 Rx Frames: 51087714 frame loss: -69, -0.000135% Rx Mpps: 1.702955 [REPORT] total: Tx frames: 102175668 Rx Frames: 102175737 frame loss: -69, -0.000068% Tx Mpps: 3.405905 Rx Mpps: 3.405907 3) [INFO] Device 0->1 PASSED - frame loss (0, 0.00000000%) is less than or equal to the maximum (0.00000000%) [INFO] Device 1->0 PASSED - frame loss (-50, -0.00005814%) is less than or equal to the maximum (0.00000000%) [INFO] Test Result: PASSED [PARAMETERS] startRate: 10.000000 nrFlows: 256 frameSize: 64 runBidirec: true searchRunTime: 30 validationRunTime: 30 acceptableLossPct: 0.000000 ports: 1,2 [REPORT]Device 0->1: Tx frames: 85981203 Rx Frames: 85981203 frame loss: 0, 0.000000% Rx Mpps: 2.866902 [REPORT]Device 1->0: Tx frames: 86005521 Rx Frames: 86005571 frame loss: -50, -0.000058% Rx Mpps: 2.866899 [REPORT] total: Tx frames: 171986724 Rx Frames: 171986774 frame loss: -50, -0.000029% Tx Mpps: 5.733800 Rx Mpps: 5.733801 Expected results: ~6.12 Mpps. Additional info: Compute host (hypervisor): [root@compute-0 ~]# numactl -H available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 node 0 size: 32671 MB node 0 free: 14910 MB node 1 cpus: 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 node 1 size: 32768 MB node 1 free: 14801 MB node distances: node 0 1 0: 10 21 1: 21 10 [root@compute-0 ~]# tuna -t ovs-vswitchd -CP | grep pmd 5672 OTHER 0 22 0 1 pmd110 5669 OTHER 0 4 0 1 pmd111 5670 OTHER 0 6 0 2 pmd112 5671 OTHER 0 20 0 1 pmd109 [root@compute-0 ~]# virsh dumpxml instance-00000012 | grep vcpu <nova:vcpus>6</nova:vcpus> <vcpu placement='static'>6</vcpu> <vcpupin vcpu='0' cpuset='8'/> <vcpupin vcpu='1' cpuset='24'/> <vcpupin vcpu='2' cpuset='10'/> <vcpupin vcpu='3' cpuset='26'/> <vcpupin vcpu='4' cpuset='28'/> <vcpupin vcpu='5' cpuset='12'/> [root@compute-0 ~]# cat /etc/tuned/cpu-partitioning-variables.conf isolated_cores=4,6,8,10,12,14,16,18,20,22,24,26,28,30 Hyperthreading enabled on the compute node. Tuned profile (tuned-2.7.1-4) enabled and activated. Iommu=pt intel_iommu=on configured. Guest instance: Tuned profile (tuned-2.7.1-4) enabled and activated. Iommu=pt intel_iommu=on configured.
Have you tested on this HW before? What was the last known working version?
The last performance test was made on current hardware. It was mono directional perf test with one DPDK port. The results of the mono directional tests: 3 runs. Stable result over all runs. Mpps: 3.063646 Frame loss (0, 0.00000000%)
and what was the OVS version?
OVS version mentioned below "Product core build" version. It's openvswitch-2.5.0-14.git20160727.el7fdb.x86_64
Reassign to Yariv to provide numbers on the RHOSP10.0, and see if isolcpus is required ot not to get 3.7Mpps/core with zero packet loss.
OVS 2.6 with RHOS 10 1Q HT 4Mpps 0.0002 packet loss. is it ok? can we move it to verified 0 Packet loss lower than 3.7.
This is verified
Once it is moved to verified, please state the package NVR because if it is available in fdProd, this bug can be closed.
Hi, Currently, OSP moved to work with OVS2.6, I've verified the performance using openvswitch-2.6.1-10.git20161206.el7fdp.x86_64 And achieved 3.576437. Thanks.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1587