Recently the kernel has gained support for lockless xmit on the tun device; which allows relevant performance improvement in networking performance towards VMs, if the related tun device is configured with a 'noqueue' qdisc. To leverage such change, libvirt has to explicitly set the tun qdisc to 'noqueue', after the tun device creation, since the default is pfifo_fast/mq. A configuration option can be added to retain the old/previous tun configuration, if needed.
Paolo, what a great news! I wonder if we could just set up the noqueue qdisc by default for every tun created in libvirt. Would there be any drawbacks? I'm mostly concerned about performance issues. Moreover, how does this cope with multiqueue feature? I guess it should be just fine, but I rather ask.
(In reply to Michal Privoznik from comment #1) > Paolo, what a great news! I wonder if we could just set up the noqueue qdisc > by default for every tun created in libvirt. Would there be any drawbacks? If libvirt does not setup any 'default' qdisc by itself. Should be no drawbacks. > I'm mostly concerned about performance issues. Moreover, how does this cope > with multiqueue feature? I guess it should be just fine, but I rather ask. It will actually improve the performance. And should work well with multiqueue. Thanks
Hi Michal, Sorry for coming back late to you. Actually Jason already answered the questions :-) (thanks Jason!) I tested the tun tx lockless with multiqueue virtio devices and it works well. AFAICS the only possible drawback is some users relaying on the old, default pfifo_fast qdisc to ensure the priority of some kind of network traffic towards the VM. Without the default qdisc that will be impossible. I can't be 100% sure, but I guess that said scenario is very unlikely. From the performance point of view, the advantages are very tangibles! Cheers, Paolo
Jan or Jason, now that I read the docs to kernel Documentation/sysctl/net.txt I've discovered default_qdisc file which allows the default qdisc to be created for new virtual devices. Is there a reason why mgmt application on the top of libvirt cannot set this instead of libvirt?
(In reply to Michal Privoznik from comment #8) > Jan or Jason, > > now that I read the docs to kernel Documentation/sysctl/net.txt I've > discovered default_qdisc file which allows the default qdisc to be created > for new virtual devices. Is there a reason why mgmt application on the top > of libvirt cannot set this instead of libvirt? Paolo may have an answer for this.
Also, libvirt runs on a lot of kernels and possibly other operating systems than Linux. So is there a way to detect that the kernel we are currently running under does support noqueue? My understanding of kernel's git log is that until 4.3 it wasn't possible to set noqueue via tc (which is how libvirt manipulates qdiscs). And I'm not aiming at "try & error" approach - we don't want to spam the logs.
Hi, (In reply to Michal Privoznik from comment #8) > now that I read the docs to kernel Documentation/sysctl/net.txt I've > discovered default_qdisc file which allows the default qdisc to be created > for new virtual devices. Is there a reason why mgmt application on the top > of libvirt cannot set this instead of libvirt? Sorry for the long delay, I lost track of this request. AFAIK, there are no hard constraints on setting the noqueue qdisc in libvirtd; I think this task was suggested for libvirtd since it creates the tun devices and can set configured them in the proper way before any traffic start flowing through them, so they always behave consistently. > So is there a way to detect that the kernel we are currently running > under does support noqueue? To obtain real gain from the noqueue qdisc, is required also lockless xmit support in the tun devices. Querying the available netdev features via the ETHTOOL_GFEATURES ioctl and testing the NETIF_F_LLTX will tell you. If the lockless xmit is available on the tun device, than also setting the noqueue should be possible (unless working on some 3rd party kernel with out-of-order backports) Paolo
Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-October/msg00636.html
Merged upstream as: 0b66196d86 qemu: Set noqueue qdisc for TAP devices 01559528e5 virnetdev: Introduce virNetDevSetRootQDisc() v6.8.0-311-g0b66196d86
Hi Michal, I have tested on libvirt-6.10.0-1, the behavior as below, could you please help to check if it is expected? 1) vnet* tun device will created default as "qdisc noqueue" with or without multiqueue setting; 2) When there is Qos setting, it will changed to "qdisc htb"; 3) If there is *no* multiqueue setting, after the Qos cleared by live update, the vnet* will change to "qdisc fq_codel" which is rhel8.4 default; 4) If there *is* multiqueue, after the Qos setting is cleared by live update, it will change to "qdisc mq"; I just checked ping for all the scenarios, no connection issue found. Do you think more performance test is needed? The details: Scenario 1: with multiqueue 1. start vm without any Qos setting, but with multiqueue setting, and check the qdisc type, it is "qdisc noqueue" # virsh dumpxml rhel | grep /interface -B9 <interface type='network'> <mac address='52:54:00:94:3e:39'/> <source network='default' portid='68f511ed-e0f6-4682-8357-394d9818f3dc' bridge='virbr0'/> <target dev='vnet5'/> <model type='virtio'/> <driver queues='4'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> # ip l | grep vnet5 18: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 **qdisc noqueue** master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000 # virsh domiftune rhel vnet5 inbound.average: 0 inbound.peak : 0 inbound.burst : 0 inbound.floor : 0 outbound.average: 0 outbound.peak : 0 outbound.burst : 0 2. set the Qos setting, and check the vnet device changed to "qdisc htb" # virsh domiftune rhel vnet5 200,300,300 300,300,200 # virsh domiftune rhel vnet5 inbound.average: 200 inbound.peak : 300 inbound.burst : 300 inbound.floor : 0 outbound.average: 300 outbound.peak : 300 outbound.burst : 200 # ip l | grep vnet5 18: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 **qdisc htb** master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000 login the guest and check the network function, it works well. 3. clear the Qos setting, it changed to "qdisc mq", check the network function, it works well. # virsh domiftune rhel vnet5 0 0 # ip l | grep vnet5 18: vnet5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 **qdisc mq** master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000 Scenario 2: without multiqueue 1. # virsh start rhel Domain rhel started # ip l | grep vnet 28: vnet15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000 2. Add Qos setting: # virsh domiftune rhel vnet15 100 # ip l | grep vnet 28: vnet15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 **qdisc htb** master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000 3.Clear the Qos setting: # virsh domiftune rhel vnet15 0 0 # ip l | grep vnet 28: vnet15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 **qdisc fq_codel** master virbr0 state UNKNOWN mode DEFAULT group default qlen 1000
Test on direct type interface, the result is different. 1. # virsh start rhel Domain rhel started # virsh dumpxml rhel | grep /interface -B12 <interface type='direct'> <mac address='52:54:00:dc:b5:7f'/> <source dev='eno1' mode='bridge'/> <target dev='macvtap5'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <interface type='direct'> <mac address='52:54:00:66:dd:6f'/> <source dev='eno1' mode='bridge'/> <target dev='macvtap6'/> <model type='virtio'/> <driver queues='4'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> # ip l | grep macvtap 30: macvtap5@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 500 31: macvtap6@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 500 2. # virsh domiftune rhel macvtap5 100 # virsh domiftune rhel macvtap6 100 # ip l | grep macvtap 30: macvtap5@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500 31: macvtap6@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500 3. # virsh domiftune rhel macvtap6 0 # virsh domiftune rhel macvtap5 0 # ip l | grep macvtap 30: macvtap5@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500 31: macvtap6@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 500
(In reply to yalzhang from comment #18) > Hi Michal, I have tested on libvirt-6.10.0-1, the behavior as below, could > you please help to check if it is expected? > > 1) vnet* tun device will created default as "qdisc noqueue" with or without > multiqueue setting; This is correct. > 2) When there is Qos setting, it will changed to "qdisc htb"; This is also correct. > 3) If there is *no* multiqueue setting, after the Qos cleared by live > update, the vnet* will change to "qdisc fq_codel" which is rhel8.4 default; This is not correct. > 4) If there *is* multiqueue, after the Qos setting is cleared by live > update, it will change to "qdisc mq"; This is somewhat correct, but we should use noqueue too, I guess. Patches posted upstream: https://www.redhat.com/archives/libvir-list/2020-December/msg00343.html
(In reply to yalzhang from comment #19) > Test on direct type interface, the result is different. This is also addressed by patches I've posted in comment 20.
Merged upstream: d53b092353 qemu: Restore default root qdisc when QoS is cleared out abb1554a2d qemu: Set default qdisc before setting bandwidth v6.10.0-324-gd53b092353
test on libvirt-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64 with below scenarios, the result is as expected. 1. check qdisc for tap device(connected to default nat network) 1) start vm, check the qdisc for vnet*, it is noqueue; 2) set Qos setting, and check the vnet*, it is htb, and the Qos works; 3) clear the Qos setting, check vnet*, recover to noqueue; 2. set multiqueue, and do the same test above, the result is the same as 1; 3. check qdisc for macvtap device, the result is the same as 1; 4. set multiqueue, and do the same test as 1, the result is the same as 1.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098
Hi Probably this thing broke ingress that set up by openvswitch... qemu 4.2 working ok, v6 ref: https://bugzilla.redhat.com/show_bug.cgi?id=2138557
Setting noqueue on OvS was fixed in https://gitlab.com/libvirt/libvirt/-/commit/e2c225f57c4 which is v7.6.0-rc1~234. What version of libvirt are you running?
(In reply to Michal Privoznik from comment #29) > Setting noqueue on OvS was fixed in > https://gitlab.com/libvirt/libvirt/-/commit/e2c225f57c4 which is > v7.6.0-rc1~234. What version of libvirt are you running? Hi It is 8.0.0 actually $ libvirtd -V libvirtd (libvirt) 8.0.0
question then is whether you told libvirt it's an OvS interface. Can you share what your <interface/> part of domain XML looks like?
It shows me next XML <interface type='ethernet'> <mac address='fa:16:3e:9c:15:bc'/> <target dev='tapfc2cf5b7-b0'/> <model type='virtio'/> <mtu size='1500'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
There you go. This does not tell libvirt that the TAP is plugged into an OvS bridge. In fact, it doesn't tell libvirt it's plugged into any bridge. If you'd change the XML to <interface type='bridge'> <mac address='fa:16:3e:9c:15:bc'/> <source bridge='ovsbr0'/> <target dev='tapfc2cf5b7-b0'/> <virtualport type='openvswitch'/> <model type='virtio'/> <mtu size='1500'/> </interface> then libvirt won't set noqueue qdisc.
So now im totally confused, but this xml is generated by nova-compute as far as i know
It seems to be a side effect of the following change in nova: https://opendev.org/openstack/nova/commit/a62dd42c0dbb6b2ab128e558e127d76962738446 In short, nova is using os-vif to wire up ports to OVS, instead of relying on libvirt. This is done to ensure that VMs are not able to send/receive traffic until the network (OpenFlow rules and other stuff) is fully configured, e.g. to avoid eavesdropping on a private traffic (CVE 2018-14636) and some other disruptive behavior. So, libvirt doesn't know that these ports will be plugged to OVS in the future. FWIW, we will be fixing the issue from the OVS side in BZ 2138339, so libvirt may remove the special case in the future.