Red Hat Bugzilla – Bug 1366356
[ovs-dpdk] Prevent other OVS threads to run in the pmd-cpu-mask cores.
Last modified: 2017-01-10 03:38:20 EST
Description of problem: When using userspace datapath (DPDK), some non-PMD threads run on the same CPU running PMD (configured by pmd-cpu-mask). That causes the PMD to be preempted which causes latency spikes, drops, etc. This bug requests the OVS to set the affinity of the other threads to avoid the the cores masked in pmd-cpu-mask parameter. Version-Release number of selected component (if applicable): ovs 2.5 How reproducible: Always Steps to Reproduce: 1. configure ovs-dpdk with single queue, single device, with one or more PMDs 2. check with 'ps' that some threads are running in the PMD's core Actual results: The PMD thread is preempted by multiple other OVS threads which causes latency spikes and packet loss. Expected results: The PMD thread to run alone. Additional info: The following script works around the issue: ----8<---- NON_PMD_CPU_RANGE="21-22" OVS_PID="$(pidof ovs-vswitchd)" for pid in $(ps -e -T | grep ${OVS_PID} | grep -v 'pmd' | awk '{ print $2 }') do taskset -p -c ${NON_PMD_CPU_RANGE} ${pid} done tuna -t ovs-vswitchd -CP ----8<----
Is it possible to assign certain cpu core to specific pmd thread ? this might be useful when we have two dpdk interfaces from different numa nodes, but added to the same ovs bridge.
(In reply to zenghui.shi from comment #2) > Is it possible to assign certain cpu core to specific pmd thread ? > this might be useful when we have two dpdk interfaces from different numa > nodes, but added to the same ovs bridge. this question might be confusing, let me rephrase it : assume we have two NIC interfaces which are located on two numa nodes separately(interface 1 sits on numa node 1, interface 2 sits on numa node 2), set ovs parameters as below: 1) add the two interfaces into the same ovs bridge, add flows to forward bidirectional traffic, 2) 2 cpu cores(one from numa node 1, the other from numa node 2) be used for pmd treads ; 3) use taskset to pin cpu from numa node 1 to pmd thread 1 4) use taskset to pin cpu from numa node 2 to pmd thread 2 question is, how to make sure the pmd thread 1 poll from interface 1 and pmd thread 2 is polling from interface 2?
Shouldn't we move this BZ in openvswitch-dpdk component?
(In reply to Franck Baudin from comment #5) > Shouldn't we move this BZ in openvswitch-dpdk component? There is no openvswitch-dpdk component in Fast Datapath.
will openvswitch-dpdk component be "closed" and all BZs moved to openvswitch?
In OVS 2.5 the core that runs general OVS threads must be specified by the user while the PMD's may be. Using -c 0x<bits> and pmd-cpu-mask. That gives the user the ability to ensure they are not on the same cores. Is that a good enough resolution for this bz for OVS 2.5? If not, I think automagically keeping the general threads and pmd-cpu-mask on separate cores should only be done when the user does not specify pmd-cpu-mask. Otherwise, we would be overriding what the user has configured. OVS 2.6 threading is different and I think it would benefit from this work more.
I don't have access to the reproducer environment anymore, but based on what Kevin said and code review, it seems that if -c and pmd-cpu-mask are correct, all non-PMD threads run on cores specified by -c and PMD threads run on cores specified in pmd-cpu-mask, so they should not overlap (maybe that was the mistake before). That is enough to resolve this bugzilla. Thanks!
This shall be addressed in first-boot scripts
Created attachment 1229955 [details] Attached the post-install.yaml Modified the post install scripts
(In reply to Karthik Sundaravel from comment #12) > Created attachment 1229955 [details] > Attached the post-install.yaml > > Modified the post install scripts Solution suggested: Additional network-enviroment.yaml parameter needed NeutronDpdkCoreList Read later on by post-install.yaml
The parameter NeutronDpdkCoreList is already defined in network-environment.yaml and is used by post-install.yaml now
The post-install.yaml script provided by Karthik and attached to the BZ has been verified.
Document changes has already been included in the Release Notes at GA. You can find it in the Known Issues section: https://access.redhat.com/documentation/en/red-hat-openstack-platform/10/single/release-notes/#idm140265780821648.