Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
On OpenStack virtio-net multi-queue with OVS-DPDK is used to scale and enhance network performance, see https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html for more detailed description.
On instances the means that ethtool needs to be called to activate the use of multiple queues. Ideally, this manual step would not be needed and tuned would activate this when it determines more than one queue is available.
Another related scaling technique is XPS as described in https://www.kernel.org/doc/Documentation/networking/scaling.txt, however I'm not sure whether this is something that would be generally wise to enable by default.
I've pasting an example script below which can be used as inspiration (see scaling.txt for details on RFS and RPS):
#!/bin/bash
USE_XPS=0
nr_cpu=$(grep -c ^processor /proc/cpuinfo)
nr_dev=$(($(ip -o link show | wc -l) - 1))
fl_cnt=$((32768 / ($nr_cpu * $nr_dev)))
#echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
for dev in $(ip -o link show | cut -d: -f2); do
[[ $dev == lo ]] && continue
# Channels
nr_queue=$(ethtool -l $dev 2> /dev/null | awk '/Combined/ {print $2;exit}')
[[ $nr_queue != $nr_cpu || $nr_cpu -eq 1 ]] && continue
ethtool -L $dev combined $nr_queue > /dev/null 2>&1
# RFS/RPS/XPS
[[ $USE_XPS -eq 0 ]] && continue
for i in $(seq 0 $(($nr_cpu - 1))); do
#echo $fl_cnt > /sys/class/net/$dev/queues/rx-$i/rps_flow_cnt
#printf "%x" $((1<<$i)) > /sys/class/net/$dev/queues/rx-$i/rps_cpus
printf "%x" $((1<<$i)) > /sys/class/net/$dev/queues/tx-$i/xps_cpus
done
done
Thanks.
It should also be checked how irqbalance exactly helps on this, what is actually needed by tuned and if irqbalance already takes care of some of the above how to make sure tuned / irqbalance won't interfere. Thanks.
(In reply to Jeremy Eder from comment #3)
> This isn't a tuned issue (yet). I would suggest you consider it in the
> context of kernel/module defaults, first.
Looks like virtio-blk does the right thing without any manual configuration sans expecting irqbalance to be running. All the OpenStack virtio-net multi-queue documentation is suggesting to run the above ethtool command but perhaps for the plain multi-queue enablement running irqbalance would be enough already?
If that is the case then it would leave us with XPS, is it something generally safe to be configured by tuned.
All in all, the point is to setup this semiautomatically (e.g., by running irqbalance + tuned) without requiring users to run arbitrary commands, like shown in the script above.
Thanks.