Red Hat Bugzilla – Bug 1240439
Add multiqueue support for 'direct' interface types.
Last modified: 2016-11-03 14:19:41 EDT
Description of problem: We can currently enable multi-queue VM nics by specifiying <driver name="vhost" queues='X'/> and this works fine for bridges where a tap device is used. However, this syntax is not supported for 'direct' type interfaces or networks configured to use direct interfaces. This makes it impossible to enabling multiqueue support with mavtap devices. Version-Release number of selected component (if applicable): Any libvirt version How reproducible: Steps to Reproduce: 1. add the driver directive to direct interface definition or to a network that uses some set of interfaces. 2. attempt to start the domain. 3. Actual results: Expected results: Additional info:
*** Bug 1247444 has been marked as a duplicate of this bug. ***
Patches proposed upstream: https://www.redhat.com/archives/libvir-list/2015-December/msg00134.html
jason, when implementing this feature for TUN/TAP devices, I was advised that libvirt should set IFF_MULTI_QUEUE flag on each /dev/net/tun FD passed to libvirt. Is this the case here too? Libvirt's opening /dev/tapX corresponding to the macvtap device. Thanks in advance!
Anyway, I got confirmation from MST that libvirt should do that. So I've just pushed patches upstream: commit 81a110edc72a42f5c6316d554a5ea64f1d8d83b0 Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Fri Dec 4 11:47:22 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:44:44 2015 +0100 qemu: Enable multiqueue for macvtaps https://bugzilla.redhat.com/show_bug.cgi?id=1240439 Ta-da! Now that we know how to open a macvtap device multiple times, we can finally enable the multiqueue feature. Everything else is already prepared (e.g. command line generation) from the previous iteration where the feature was implemented for TUN/TAP devices. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> commit 08da97bfb9b459333d5f9efa7841ce8fe23f431d Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Fri Dec 4 11:31:17 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:44:43 2015 +0100 virNetDevMacVLanCreateWithVPortProfile: Rework to support multiple FDs For the multiqueue on macvtaps we are going to need to open the device multiple times. Currently, this is not supported. Rework the function, so that upper layers can be reworked too. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> commit 1e90c744d5b98d00af5957819afb9d9568f9daff Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Tue Dec 8 13:17:26 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:44:39 2015 +0100 virNetDevMacVLanTapSetup: Allow enabling of IFF_MULTI_QUEUE Like we are doing for TUN/TAP devices, we should do the same for macvtaps. Although, it's not as critical as in that case, we should do it for the consistency. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> commit 136fe2f7cc7035294d270f63c4dba13fa839390c Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Fri Dec 4 11:19:32 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:42:50 2015 +0100 virNetDevMacVLanTapSetup: Rework to support multiple FDs For the multiqueue on macvtaps we are going to need to open the device multiple times. Currently, this is not supported. Rework the function, so that upper layers can be reworked too. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> commit d36897c7658cd9a744f9a999c06c7c3d9f98fab8 Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Fri Dec 4 09:39:02 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:42:50 2015 +0100 virNetDevMacVLanTapOpen: Rework to support multiple FDs For the multiqueue on macvtaps we are going to need to open the device multiple times. Currently, this is not supported. Rework the function, so that upper layers can be reworked too. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> commit 025a87065fa92b9e48e9ffbe0a1f065440f85a0e Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Fri Dec 4 10:55:49 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:42:49 2015 +0100 virNetDevMacVLanTapOpen: Slightly rework There are few outdated things. Firstly, we don't need to undergo the torture of fopen, fscanf and fclose just to get the interface index when we have nice wrapper over that: virNetDevGetIndex. Secondly, we don't need to have statically allocated buffer for the path we are opening. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> commit 56e2171c6fee97d98241f9143da4fcd194c3633c Author: Michal Privoznik <mprivozn@redhat.com> AuthorDate: Thu Dec 3 11:33:55 2015 +0100 Commit: Michal Privoznik <mprivozn@redhat.com> CommitDate: Fri Dec 11 08:42:49 2015 +0100 virNetDevMacVLanCreateWithVPortProfile: Turn vnet_hdr into flag So yet again one of integer arguments that we use as a boolean. Since the argument count of the function is unbearably long enough, lets turn those booleans into flags. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> v1.3.0-57-g81a110e Moving to POST.
verify this bug with libvirt-1.3.1-1.el7.x86_64 1.1. prepare guest xml with macvtap/mq # cat rh7.xml |grep vhost -B 4 <interface type='network'> <mac address='52:54:00:4c:5a:62'/> <source network='net-bridge-bridge'/> <model type='virtio'/> <driver name='vhost' queues='2'/> # virt-xml-validate rh7.xml rh7.xml validates 1.2. define/start guest # virsh define rh7.xml Domain rh7 defined from rh7.xml # virsh start rh7 Domain rh7 started # virsh dumpxml rh7|grep vhost -B 5 <interface type='direct'> <mac address='52:54:00:4c:5a:62'/> <source network='net-bridge-bridge' dev='enp2s0' mode='bridge'/> <target dev='macvtap0'/> <model type='virtio'/> <driver name='vhost' queues='2'/> 1.3. check qemu cmd line # ps aux|grep vhost qemu 23093 73.7 9.0 1752304 345100 ? Sl 17:08 0:20 /usr/libexec/qemu-kvm -name rh7 -S -machine pc-q35-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d5ca4513-2e9d-4577-b5dc-2d736a5bceb8 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-rh7/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/rh7.img,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pcie.0,multifunction=on,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=25:27,id=hostnet0,vhost=on,vhostfds=28:29 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=52:54:00:4c:5a:62,bus=pci.2,addr=0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-rh7/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x6 -msg timestamp=on root 23096 0.0 0.0 0 0 ? S 17:08 0:00 [vhost-23093] root 23097 0.0 0.0 0 0 ? S 17:08 0:00 [vhost-23093] 1.4. login guest to check # ethtool -i eth0 driver: virtio_net version: 1.0.0 firmware-version: bus-info: 0000:02:02.0 supports-statistics: no supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no # ethtool -L eth0 combined 2 # ethtool -L eth0 combined 1 # ethtool -L eth0 combined 3 Cannot set device channel parameters: Invalid argument 1.5. check hotplug # cat interface.xml <interface type='direct'> <source network='net-bridge-bridge' dev='enp2s0' mode='bridge'/> <model type='virtio'/> <driver name='vhost' queues='2'/> </interface> # virsh attach-device rh7 interface.xml error: Failed to attach device from interface.xml error: unsupported configuration: Multiqueue network is not supported for: direct I file new bug 1313264 for this issue. 1.6. check hotunplugging # virsh detach-interface rh7 direct Interface detached successfully # virsh dumpxml r7 |grep vhost -B5 -A3 # ps aux|grep vhost qemu 14158 45.9 9.4 1758288 362084 ? Sl 18:14 0:37 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d4bfbeef-a959-4b35-aa97-870dd9ab7b31 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-r7/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x3 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.0,addr=0x4 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x1d.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x1f.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/r7.img,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=26:28,id=hostnet0,vhost=on,vhostfds=29:30 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=52:54:00:12:2b:f6,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-r7/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on root 14566 0.0 0.0 112644 924 pts/1 S+ 18:16 0:00 grep --color=auto vhost
verified on libvirt-1.3.4-1.el7.x86_64, the macvtap network support multiqueue both in hotplug and start a guest with multiqueue defined. So move the bug to verified.
Hi Michal, Could you please help to confirm if this is expected? Currently kernel supports up to 256 queues (see BZ 1308395#c4), and it did work for nat mode interface.(see BZ 1322661#c5) But for macvtap, as I tried, it can support 16 at most. When set queues=17 or more, the guest will not start and report error, I tired hotplug, the error is similiar: # virsh start rhel error: Failed to start domain rhel error: cannot stat tap fd -1: Bad file descriptor
(In reply to yalzhang@redhat.com from comment #8) > Hi Michal, > > Could you please help to confirm if this is expected? > Currently kernel supports up to 256 queues (see BZ 1308395#c4), and it did > work for nat mode interface.(see BZ 1322661#c5) > But for macvtap, as I tried, it can support 16 at most. When set queues=17 > or more, the guest will not start and report error, I tired hotplug, the > error is similiar: > > # virsh start rhel > error: Failed to start domain rhel > error: cannot stat tap fd -1: Bad file descriptor Nope! That is not expected and in fact lead me to a bug in our code. Moving back to ASSIGNED so we can pick up the fix for this case too. Good job!
Patches proposed upstream: https://www.redhat.com/archives/libvir-list/2016-August/msg00503.html
To POST: http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-August/msg00672.html
Hi Michal, Verified on libvirt-2.0.0-5.el7.x86_64 It still can not support multiqueues more than 16, when I set 17 or more, and the error message changed: # virsh start ovmf error: Failed to start domain ovmf error: cannot open macvtap tap device /dev/tap20: Device or resource busy Please take a look at it,tks~
(In reply to yalzhang@redhat.com from comment #13) > Hi Michal, > > Verified on > libvirt-2.0.0-5.el7.x86_64 > > It still can not support multiqueues more than 16, when I set 17 or more, > and the error message changed: > > # virsh start ovmf > error: Failed to start domain ovmf > error: cannot open macvtap tap device /dev/tap20: Device or resource busy > > Please take a look at it,tks~ Well, this means that in kernel there's not support for 17 or more queues for macvtaps. And when looking into kernel source code I can confirm this: include/linux/if_macvlan.h- include/linux/if_macvlan.h-/* include/linux/if_macvlan.h- * Maximum times a macvtap device can be opened. This can be used to include/linux/if_macvlan.h- * configure the number of receive queue, e.g. for multiqueue virtio. include/linux/if_macvlan.h- */ include/linux/if_macvlan.h:#define MAX_MACVTAP_QUEUES (NR_CPUS < 16 ? NR_CPUS : 16) Also, there are apparently different limits for tap devices and macvtap devices.
As it is kernel limitation, move the bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html