Bug 1395537

Summary:	RFE: tuned should enable use of multi-queue where possible
Product:	Red Hat Enterprise Linux 7	Reporter:	Marko Myllynen <myllynen>
Component:	tuned	Assignee:	Jaroslav Škarvada <jskarvad>
Status:	CLOSED DEFERRED	QA Contact:	qe-baseos-daemons
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.3	CC:	atheurer, jeder, jskarvad
Target Milestone:	rc	Keywords:	FutureFeature
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-11-24 13:51:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Marko Myllynen 2016-11-16 06:25:32 UTC

Description of problem:
On OpenStack virtio-net multi-queue with OVS-DPDK is used to scale and enhance network performance, see https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html for more detailed description.

On instances the means that ethtool needs to be called to activate the use of multiple queues. Ideally, this manual step would not be needed and tuned would activate this when it determines more than one queue is available.

Another related scaling technique is XPS as described in https://www.kernel.org/doc/Documentation/networking/scaling.txt, however I'm not sure whether this is something that would be generally wise to enable by default.

I've pasting an example script below which can be used as inspiration (see scaling.txt for details on RFS and RPS):

#!/bin/bash

USE_XPS=0

nr_cpu=$(grep -c ^processor /proc/cpuinfo)
nr_dev=$(($(ip -o link show | wc -l) - 1))
fl_cnt=$((32768 / ($nr_cpu * $nr_dev)))
#echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
for dev in $(ip -o link show | cut -d: -f2); do
  [[ $dev == lo ]] && continue

  # Channels
  nr_queue=$(ethtool -l $dev 2> /dev/null | awk '/Combined/ {print $2;exit}')
  [[ $nr_queue != $nr_cpu || $nr_cpu -eq 1 ]] && continue
  ethtool -L $dev combined $nr_queue > /dev/null 2>&1

  # RFS/RPS/XPS
  [[ $USE_XPS -eq 0 ]] && continue
  for i in $(seq 0 $(($nr_cpu - 1))); do
    #echo $fl_cnt > /sys/class/net/$dev/queues/rx-$i/rps_flow_cnt
    #printf "%x" $((1<<$i)) > /sys/class/net/$dev/queues/rx-$i/rps_cpus
    printf "%x" $((1<<$i)) > /sys/class/net/$dev/queues/tx-$i/xps_cpus
  done
done

Thanks.

Comment 1 Marko Myllynen 2016-11-16 06:32:09 UTC

It should also be checked how irqbalance exactly helps on this, what is actually needed by tuned and if irqbalance already takes care of some of the above how to make sure tuned / irqbalance won't interfere. Thanks.

Comment 3 Jeremy Eder 2016-11-16 12:38:41 UTC

This isn't a tuned issue (yet).  I would suggest you consider it in the context of kernel/module defaults, first.

Comment 4 Marko Myllynen 2016-11-16 13:24:07 UTC

(In reply to Jeremy Eder from comment #3)
> This isn't a tuned issue (yet).  I would suggest you consider it in the
> context of kernel/module defaults, first.

Looks like virtio-blk does the right thing without any manual configuration sans expecting irqbalance to be running. All the OpenStack virtio-net multi-queue documentation is suggesting to run the above ethtool command but perhaps for the plain multi-queue enablement running irqbalance would be enough already? 

If that is the case then it would leave us with XPS, is it something generally safe to be configured by tuned.

All in all, the point is to setup this semiautomatically (e.g., by running irqbalance + tuned) without requiring users to run arbitrary commands, like shown in the script above.

Thanks.

Comment 7 Marko Myllynen 2016-11-18 16:25:01 UTC

Neil Horman points out that for enabling multi-queue there's even more elegant solution than enhancing tuned:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0f13b66b01c6e2ec4913a7812414183844d1cc4f

There's a new kernel RFE/BZ to backport this patch, I think could close this BZ:

https://bugzilla.redhat.com/show_bug.cgi?id=1396578

Thanks.

Comment 8 Marko Myllynen 2016-11-24 13:51:52 UTC

(In reply to Marko Myllynen from comment #7)
> 
> There's a new kernel RFE/BZ to backport this patch, I think could close this
> BZ:

I've created Fedora / upstream RFE about RSS/RPS/RFS/XPS, thus together with the above mentioned https://bugzilla.redhat.com/show_bug.cgi?id=1396578 this BZ is now obsolete, closing.

https://bugzilla.redhat.com/show_bug.cgi?id=1398345

Thanks.