Bug 1313040
| Summary: | significant throughput drop between 1% and 0% loss RFC2544 tests for vhostuser | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Andrew Theurer <atheurer> |
| Component: | openvswitch-dpdk | Assignee: | Flavio Leitner <fleitner> |
| Status: | CLOSED NOTABUG | QA Contact: | Red Hat Kernel QE team <kernel-qe> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.4 | CC: | aloughla, atheurer, atragler, bmichalo, brault, fbaudin, fleitner, jhladky, mleitner, osabart, rkhan, sukulkar |
| Target Milestone: | rc | ||
| Target Release: | 7.3 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-09-29 14:53:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1349523 | ||
|
Description
Andrew Theurer
2016-02-29 19:28:04 UTC
We believe we have resolved this with tuning, namely: isolcpus, nohz_full, and rcu_nocbs in the host for all ovs-dpdk PMD threads and VM vcpu threads isolcpus, nohz_full, and rcu_nocbs in the guest for all DPDK application PMD threads sched:fifo-95 in the host for ovs-dpdk PMD threads and VM vcpu threads sched:fifo-95 in the guest for DPDK application PMD threads However, using sched:fifo can cause a problem, described here: https://bugzilla.redhat.com/show_bug.cgi?id=1328890 The proposed solution to that problem is only for RT kernel, so the RT kernel will likely be a requirement for high throughput packet processing with zero packet loss. (In reply to Andrew Theurer from comment #2) > We believe we have resolved this with tuning, namely: Thanks Andrew. I found the same results even for single queue case. Is there anything you consider worth fixing in OVS since those were actually events not under OVS control? Thanks, fbl Flavio, at this time I don't think there is anything to address in OVS since the mutex fix for RCU went in. We, however, have not exercised all possible paths in OVS (for example these tests do not use MAC learning mode) Eventually we may uncover something. For example a miss of EMC or something else might cause extra latency from OVS. Eventually we should document when OVS may not sustain zero packet loss. Going forward, in order to help understand if OVS is affecting packet loss, I would propose we run two types of tests: First is a non-OVS test with RT kernel, in order to confirm kernel and KVM is "good". This would involve using testpmd in both the guest and the host (with vhostuser used on host testpmd). We are confident testpmd should not do anything to cause latency spike -this allows us to conclude any latency spike in this test is from either kernel, kvm, or qemu. Second is a test using OVS instead of testpmd in the host. This is run once the first test passes. If this test fails, then we further investigate OVS. |