Hide Forgot
Description of problem: On OpenStack virtio-net multi-queue with OVS-DPDK is used to scale and enhance network performance, see https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html for more detailed description. On instances the means that ethtool needs to be called to activate the use of multiple queues. Ideally, this manual step would not be needed and tuned would activate this when it determines more than one queue is available. Another related scaling technique is XPS as described in https://www.kernel.org/doc/Documentation/networking/scaling.txt, however I'm not sure whether this is something that would be generally wise to enable by default. I've pasting an example script below which can be used as inspiration (see scaling.txt for details on RFS and RPS): #!/bin/bash USE_XPS=0 nr_cpu=$(grep -c ^processor /proc/cpuinfo) nr_dev=$(($(ip -o link show | wc -l) - 1)) fl_cnt=$((32768 / ($nr_cpu * $nr_dev))) #echo 32768 > /proc/sys/net/core/rps_sock_flow_entries for dev in $(ip -o link show | cut -d: -f2); do [[ $dev == lo ]] && continue # Channels nr_queue=$(ethtool -l $dev 2> /dev/null | awk '/Combined/ {print $2;exit}') [[ $nr_queue != $nr_cpu || $nr_cpu -eq 1 ]] && continue ethtool -L $dev combined $nr_queue > /dev/null 2>&1 # RFS/RPS/XPS [[ $USE_XPS -eq 0 ]] && continue for i in $(seq 0 $(($nr_cpu - 1))); do #echo $fl_cnt > /sys/class/net/$dev/queues/rx-$i/rps_flow_cnt #printf "%x" $((1<<$i)) > /sys/class/net/$dev/queues/rx-$i/rps_cpus printf "%x" $((1<<$i)) > /sys/class/net/$dev/queues/tx-$i/xps_cpus done done Thanks.
It should also be checked how irqbalance exactly helps on this, what is actually needed by tuned and if irqbalance already takes care of some of the above how to make sure tuned / irqbalance won't interfere. Thanks.
This isn't a tuned issue (yet). I would suggest you consider it in the context of kernel/module defaults, first.
(In reply to Jeremy Eder from comment #3) > This isn't a tuned issue (yet). I would suggest you consider it in the > context of kernel/module defaults, first. Looks like virtio-blk does the right thing without any manual configuration sans expecting irqbalance to be running. All the OpenStack virtio-net multi-queue documentation is suggesting to run the above ethtool command but perhaps for the plain multi-queue enablement running irqbalance would be enough already? If that is the case then it would leave us with XPS, is it something generally safe to be configured by tuned. All in all, the point is to setup this semiautomatically (e.g., by running irqbalance + tuned) without requiring users to run arbitrary commands, like shown in the script above. Thanks.
Neil Horman points out that for enabling multi-queue there's even more elegant solution than enhancing tuned: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=0f13b66b01c6e2ec4913a7812414183844d1cc4f There's a new kernel RFE/BZ to backport this patch, I think could close this BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1396578 Thanks.
(In reply to Marko Myllynen from comment #7) > > There's a new kernel RFE/BZ to backport this patch, I think could close this > BZ: I've created Fedora / upstream RFE about RSS/RPS/RFS/XPS, thus together with the above mentioned https://bugzilla.redhat.com/show_bug.cgi?id=1396578 this BZ is now obsolete, closing. https://bugzilla.redhat.com/show_bug.cgi?id=1398345 Thanks.