Bug 1616306
| Summary: | Performance Degradation running PVP on all packet sizes. rfc 2544 Throughput tests | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Bob Fubel <bfubel> | ||||||||||||||||
| Component: | openvswitch2.10 | Assignee: | Eelco Chaudron <echaudro> | ||||||||||||||||
| Status: | CLOSED DUPLICATE | QA Contact: | Bob Fubel <bfubel> | ||||||||||||||||
| Severity: | medium | Docs Contact: | |||||||||||||||||
| Priority: | medium | ||||||||||||||||||
| Version: | 7.5 | CC: | aloughla, atragler, bfubel, cfontain, ctrautma, fbaudin, maxime.coquelin, rkhan | ||||||||||||||||
| Target Milestone: | rc | ||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||
| Hardware: | Unspecified | ||||||||||||||||||
| OS: | Unspecified | ||||||||||||||||||
| Whiteboard: | |||||||||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||
| Last Closed: | 2018-09-17 06:40:07 UTC | Type: | Bug | ||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||
| Embargoed: | |||||||||||||||||||
| Attachments: |
|
||||||||||||||||||
Created attachment 1476172 [details]
guest cmds
Created attachment 1476173 [details]
Test Results
Per conversation with Rashid, assigning to Eelco. Was wondering if the system was properly tuned, as I do see quite some deviation in the runs, especially the 2.10 ones. Or is this expected with RFC2544? I don't have a Xena set up, but do see some lower numbers on a quick run for 256 bytes packets on the ovs_perf zero packet loss test. Will do a full run on my setup and see the results. If they look ok, will try to get some Xena ports. In the meantime, can you repeat the test for PP, so without the VM. If the problem is not seen there it might help to pinpoint the commit easier (vhost related). In addition was the 2.9 and 2.10 test ran on the same setup? What NIC's (and a which speed) where used? Can you get me a copy of the Xena configuration file you used for testing? Also how many traffic flows were you using, and what was changing? MAC/IP/UDP etc. I could not get this from the spreadsheet. Created attachment 1479376 [details]
Xena2544 tester file
The system was tuned correctly. There is sometimes a variation in the zero loss test. if a packet is dropped in the binary search the test is not repeated and that limits the top end of the test. To account for this I run the test several times to make sure the level we see is consitant. I have attached the Xena File. The test is being run using vsperf to drive the xena. If you would like you can loging to the system to look at any setting. netqe15.knqe.lab.eng.bos.redhat.com Just email/IRC me 'bfubel' to setup a time. so I will not have anything on the system and back in the state. NIC: Intel Ethernet 10G 2P X520 Created attachment 1479825 [details]
RFC2544 Configuration file for Xena
Comment on attachment 1479376 [details]
Xena2544 tester file
Wrong File
I've been running some tests on my setup using the Xena2544 interface directly with XL710 cards and the throughput for 5 iterations of a 60seconds run with 256 bytes packets are quite stable and do not differ between 2.9 and 2.10. I've attached two pictures with the results from the Xena console. It might be related to the driver you are using, or something else specific to your setup. Would it be possible to get access to your setup sometime next week during my day (CET time zone)? If you let me know the Xena and ports used I can manually trigger the tests. Login information for your system and how to restart the VM (testpmd) would also help. Created attachment 1479830 [details]
OVS 2.9 xena results
Created attachment 1479831 [details]
OVS 2.10 xena results
I change the Emulator Threads like you sugested and it didn't seem impact the preformace hits I am seeing. cmd : /bin/bash -c "sudo -E taskset -c 1 /usr/libexec/qemu-kvm -m 8192 -smp 3 -cpu host,migratable=off -drive if=ide,file=rhel1Q.qcow2 -boot c --enable-kvm -monitor unix:/tmp/vm0monitor,server,nowait -object memory-backend-file,id=mem,size=8192M,mem-path=/dev/hugepages,share=on -numa node,memdev=mem -mem-prealloc -nographic -vnc :0 -name Client0 -snapshot -net none -no-reboot -chardev socket,id=char0,path=/var/run/openvswitch/dpdkvhostuserclient0,server -netdev type=vhost-user,id=net1,chardev=char0,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=net1,csum=off,mrg_rxbuf=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off -chardev socket,id=char1,path=/var/run/openvswitch/dpdkvhostuserclient1,server -netdev type=vhost-user,id=net2,chardev=char1,vhostforce -device virtio-net-pci,mac=00:00:00:00:00:02,netdev=net2,csum=off,mrg_rxbuf=off,gso=off,guest_tso4=off,guest_tso6=off,guest_ecn=off" cmd : sudo taskset -c -p 1 72423 cmd : sudo taskset -c -p 9 72424 cmd : sudo taskset -c -p 11 72425 Closing this BZ as a duplicate of 1628965, as according to emails this is the new bug for tracking the issue. On 14 Sep 2018, at 18:57, Christian Trautman wrote: > Just to clarify, the performance issue is only when updating to qemu 2.12. > > All tests with 7.5 with qemu 2.10 have not had a problem with either ovs 2.9 or ovs 2.10. > > We closed the previous bug and opened a new one to get the information precise and easier to follow. > > https://bugzilla.redhat.com/show_bug.cgi?id=1628965 > > -Christian, > *** This bug has been marked as a duplicate of bug 1628965 *** |
Created attachment 1476170 [details] Host cmds Description of problem: Performace has dropped fom 2.9 to 2.10. running in ixgbe. only change was ugraded from 2.9 to 2.10 between runs. Version-Release number of selected component (if applicable): openvswitch2.10-2.10.0-0.20180810git58a7ce6.el7fdp.x86_64 openvswitch-2.9.0-55.el7fdp.x86_64 How reproducible: every time. Steps to Reproduce: attached log for host and guests. running xena traffic generator. RFC2544 throughtput test 0% loss test. some loss seen for 60 second trials and large loss seen on 20 minute trials. Actual results: attaced results: Expected results: Perfoamce should not have changed. Additional info: Using CPU-partioning tuned-adm profile. 7.5 on host and guest.