RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1412201 - OVS-DPDK : Improved performance/scaling with # of flows to address Telco use cases
Summary: OVS-DPDK : Improved performance/scaling with # of flows to address Telco use ...
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: openvswitch
Version: 7.4
Hardware: x86_64
OS: Unspecified
high
high
Target Milestone: pre-dev-freeze
: ---
Assignee: Kevin Traynor
QA Contact: Christian Trautman
URL:
Whiteboard:
Depends On:
Blocks: 1419948 1426175 1465537
TreeView+ depends on / blocked
 
Reported: 2017-01-11 14:14 UTC by Chegu Vinod
Modified: 2017-09-22 10:38 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1426175 (view as bug list)
Environment:
Last Closed: 2017-09-22 10:38:45 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Chegu Vinod 2017-01-11 14:14:43 UTC
Telco/NFV use cases demand support for a very large number of flows without reduction in througput performance. OVS-DPDK today doesn't scale well with the increased number of flows relative to some other alternatives (e.g. VPP). There has been some ongoing work on this topic on the upstream : Pl. refer to slide #16 in http://openvswitch.org/support/ovscon2016/8/1400-gray.pdf . 

Request is to ensure that by OSP 12 (Pike) release the version of the OVS+DPDK chosen has the relevant changes included in it. If those changes are not yet in that version of OVS+DPDK but changes are already committed in the master OVS+DPDK trees then the request them to be backported from the master trees to the versions that is being included in OSP 12 release.

Comment 3 Kevin Traynor 2017-02-10 17:51:07 UTC
Below is a list of the patches that were used in the referred to benchmarks. I expect that they will all be merged/rejected in upstream OVS by OVS 2.8 (late summer 2017?).

Probabilistic EMC insertion:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327576.html
 
PMD performance debug info:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327569.html
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327570.html
 
TX Batching:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-December/326348.html
 
TX to tunnel ports without recirculation (combine actions):
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327161.html
 
The following patches are currently upstreamed in to the master branch and will be part of OVS 2.7
 
DPCLS per in_port with sorted subtables:
Commit 3453b4d62a98f1c276a89ad560d4212b752c7468
 
Data structure alignment:
http://openvswitch.org/pipermail/dev/2016-October/080654.html
Commit d8a304b44765cb114ad19dfbc46369dc140db340
Commit 22ccf9c258a763a246270da71e66f035388998a1
Commit 63906f18d71318623307914dc99ad28acfeff399
Commit 38ee0814978c620bfa2ba5ee1069d8b42ff9de95
Commit ad9f05812d399bf794f9ab34449a8fc32ca08806
Commit db2e457d6c1cd3e3bf4ddfcbc568449dceb11285
Commit 0a0b5a7282063114d223b9bed386235b7e59bccd
Commit 96d96917e9fe0157a6d8ff1457b52161ff1206d4
Commit 2e4450aa35c6e7a9c97c930f5d43028cfbfbe07f
Commit a17408e432b7faba04219f48f00f8067010b8aed
Commit 165f5e467129d0032004de70dbf0a7d7f3e7950a
Commit f4b835bb0f2e7040a52bc3b0bbd6ba8fa4b20857

Comment 4 Kevin Traynor 2017-02-28 15:47:46 UTC
Current status of outstanding patches:

Probabilistic EMC insertion:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327576.html
- Merged.
 
PMD performance debug info:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327569.html
- Merged.

https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327570.html
- Not Merged. However this is just a user stats addition for debugging. It will not help improve performance.
 
TX Batching:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-December/326348.html
- In rework.
 
TX to tunnel ports without recirculation (combine actions):
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327161.html
- In rework.

Comment 5 Kevin Traynor 2017-05-12 13:23:15 UTC
Current status of outstanding patches:

Probabilistic EMC insertion:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327576.html
- Merged in OVS master @ OVS 2.7.90 (i.e. post OVS 2.7).
  
TX Batching:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-December/326348.html
- Max latency for this original patch is unacceptable. However, there is wip with some suggestions for how to improve latency.
 
TX to tunnel ports without recirculation (combine actions):
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327161.html
- Was merged, but now reverted due to some test fails and bugs. I expect it will be re-applied when they are resolved.

====

The following are part of the patches in the Description, however they are stats only and do not have an effect on performance. As such I am not considering them necessary for this Bz:

PMD performance debug info:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327569.html
- Not Merged (was incorrect status in Comment 4)

EMC % full stat
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327570.html
- Not Merged.

Comment 7 Kevin Traynor 2017-05-15 16:56:58 UTC
Current status of outstanding patches:

Probabilistic EMC insertion:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327576.html
- Merged in OVS master @ OVS 2.7.90 (i.e. post OVS 2.7).
- There is a known bug with this code. The 'random' number that is used as the basis for the probability of insert to the emc is not random enough. It means when setting that flows should be inserted after an average of 100 pkts, it may be many 100's of thousands pkts before flow is inserted into the emc. The author is working on fix for this.
  
TX Batching:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-December/326348.html
- Max latency for this original patch is unacceptable. 
- New batching code for dpdk and dpdkvhost ports is WIP
 
TX to tunnel ports without recirculation (combine actions):
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327161.html
- Reverted from master in commit f5f64552ec22f4c7452fdb554d52e73055b51cca

Comment 8 Kevin Traynor 2017-06-06 15:56:28 UTC
Current status of outstanding features:

Probabilistic EMC insertion:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327576.html
- Merged in OVS master @ OVS 2.7.90 (i.e. post OVS 2.7).
- Fix for random number bug is acked-by/tested-by upstream
- Backport for OVS 2.7 is ready pending upstream merge of random number bug
  
TX Batching:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-December/326348.html
- Max latency for this original patch is unacceptable. 
- New batching code with more acceptable latency for dpdk and addition of dpdkvhost ports is WIP upstream
 
TX to tunnel ports without recirculation (combine actions):
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327161.html
- Reverted from master in commit f5f64552ec22f4c7452fdb554d52e73055b51cca

Comment 10 Kevin Traynor 2017-06-23 14:38:04 UTC
From https://bugzilla.redhat.com/show_bug.cgi?id=1426175#c5
(In reply to Franck Baudin from comment #4)
> Will all the commits above included in OVS 2.7 that will be shipped with
> RHEL 7.4?
> 
> RHOSP12 will ship with ^

Probabilistic EMC insertion:
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327576.html
- Merged in OVS master @ OVS 2.7.90 (i.e. post OVS 2.7).
- Fix for random number bug is acked-by/tested-by upstream but minor rework is needed on the fix upstream
- Backport for OVS 2.7 is ready pending upstream merge of random number bug
  
TX Batching:
https://mail.openvswitch.org/pipermail/ovs-dev/2016-December/326348.html
- Max latency for this original patch is unacceptable. 
- New batching code with more acceptable latency is upstream and acked
 
TX to tunnel ports without recirculation (combine actions):
https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327161.html
- Reverted from master in commit f5f64552ec22f4c7452fdb554d52e73055b51cca
- No updates from upstream

These listed ones need to be fully upstream first. All other commits will be part of the OVS 2.7 package.

Comment 12 Kevin Traynor 2017-09-22 10:38:45 UTC
I want to wrap up this Bz. Most of the changes have merged and are available in OVS 2.7 packages. The exceptions are:

Probabilistic EMC insertion:
- This missed OVS 2.7 and there has been several bugs fixes needed
- There are still one or two minor issues
- The thinking is not to backport to OVS 2.7 as it would replace the current default scheme

Tx Batching:
- This is still being discussed upstream due to different implementations and impact on latency. It may merge in OVS 2.9 timeframe.

TX to tunnel ports without recirculation (combine actions):
- This merged, caused several bugs, was reverted and re-merged just before OVS 2.8
- Backporting is not possible due to major code churn in this area.

There's mixture of resolutions for the different individual changes, but I'm going to mark the overall Bz as closed/nextrelease, as all the changes should eventually merge and become stable.


Note You need to log in before you can comment on or make changes to this bug.