Bug 1240439

*** Bug 1247444 has been marked as a duplicate of this bug. ***

Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2015-December/msg00134.html

jason,

when implementing this feature for TUN/TAP devices, I was advised that libvirt should set IFF_MULTI_QUEUE flag on each /dev/net/tun FD passed to libvirt. Is this the case here too? Libvirt's opening /dev/tapX corresponding to the macvtap device. Thanks in advance!

Anyway, I got confirmation from MST that libvirt should do that.
So I've just pushed patches upstream:

commit 81a110edc72a42f5c6316d554a5ea64f1d8d83b0
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 11:47:22 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:44:44 2015 +0100

    qemu: Enable multiqueue for macvtaps
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1240439
    
    Ta-da! Now that we know how to open a macvtap device multiple
    times, we can finally enable the multiqueue feature. Everything
    else is already prepared (e.g. command line generation) from the
    previous iteration where the feature was implemented for
    TUN/TAP devices.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 08da97bfb9b459333d5f9efa7841ce8fe23f431d
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 11:31:17 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:44:43 2015 +0100

    virNetDevMacVLanCreateWithVPortProfile: Rework to support multiple FDs
    
    For the multiqueue on macvtaps we are going to need to open
    the device multiple times. Currently, this is not supported.
    Rework the function, so that upper layers can be reworked too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 1e90c744d5b98d00af5957819afb9d9568f9daff
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Dec 8 13:17:26 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:44:39 2015 +0100

    virNetDevMacVLanTapSetup: Allow enabling of IFF_MULTI_QUEUE
    
    Like we are doing for TUN/TAP devices, we should do the same for
    macvtaps. Although, it's not as critical as in that case, we
    should do it for the consistency.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 136fe2f7cc7035294d270f63c4dba13fa839390c
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 11:19:32 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:50 2015 +0100

    virNetDevMacVLanTapSetup: Rework to support multiple FDs
    
    For the multiqueue on macvtaps we are going to need to open
    the device multiple times. Currently, this is not supported.
    Rework the function, so that upper layers can be reworked too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit d36897c7658cd9a744f9a999c06c7c3d9f98fab8
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 09:39:02 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:50 2015 +0100

    virNetDevMacVLanTapOpen: Rework to support multiple FDs
    
    For the multiqueue on macvtaps we are going to need to open
    the device multiple times. Currently, this is not supported.
    Rework the function, so that upper layers can be reworked too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 025a87065fa92b9e48e9ffbe0a1f065440f85a0e
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 10:55:49 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:49 2015 +0100

    virNetDevMacVLanTapOpen: Slightly rework
    
    There are few outdated things. Firstly, we don't need to undergo
    the torture of fopen, fscanf and fclose just to get the interface
    index when we have nice wrapper over that: virNetDevGetIndex.
    Secondly, we don't need to have statically allocated buffer for
    the path we are opening.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 56e2171c6fee97d98241f9143da4fcd194c3633c
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Dec 3 11:33:55 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:49 2015 +0100

    virNetDevMacVLanCreateWithVPortProfile: Turn vnet_hdr into flag
    
    So yet again one of integer arguments that we use as a boolean.
    Since the argument count of the function is unbearably long
    enough, lets turn those booleans into flags.
    
    Signed-off-by: Michal Privoznik <mprivozn>


v1.3.0-57-g81a110e

Moving to POST.

verify this bug with libvirt-1.3.1-1.el7.x86_64

1.1. prepare guest xml with macvtap/mq 

# cat rh7.xml |grep vhost -B 4
    <interface type='network'>
      <mac address='52:54:00:4c:5a:62'/>
      <source network='net-bridge-bridge'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>

# virt-xml-validate rh7.xml 
rh7.xml validates


1.2. define/start guest

# virsh define rh7.xml 
Domain rh7 defined from rh7.xml

# virsh start rh7 
Domain rh7 started


# virsh dumpxml rh7|grep vhost -B 5
    <interface type='direct'>
      <mac address='52:54:00:4c:5a:62'/>
      <source network='net-bridge-bridge' dev='enp2s0' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>


1.3. check qemu cmd line

# ps aux|grep vhost 
qemu     23093 73.7  9.0 1752304 345100 ?      Sl   17:08   0:20 /usr/libexec/qemu-kvm -name rh7 -S -machine pc-q35-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d5ca4513-2e9d-4577-b5dc-2d736a5bceb8 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-rh7/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/rh7.img,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pcie.0,multifunction=on,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=25:27,id=hostnet0,vhost=on,vhostfds=28:29 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=52:54:00:4c:5a:62,bus=pci.2,addr=0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-rh7/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x6 -msg timestamp=on
root     23096  0.0  0.0      0     0 ?        S    17:08   0:00 [vhost-23093]
root     23097  0.0  0.0      0     0 ?        S    17:08   0:00 [vhost-23093]

1.4. login guest to check
# ethtool -i eth0
driver: virtio_net
version: 1.0.0
firmware-version: 
bus-info: 0000:02:02.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
# ethtool -L eth0 combined 2
# ethtool -L eth0 combined 1
# ethtool -L eth0 combined 3
Cannot set device channel parameters: Invalid argument

1.5. check hotplug

# cat interface.xml 
   <interface type='direct'>
      <source network='net-bridge-bridge' dev='enp2s0' mode='bridge'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
    </interface>
# virsh attach-device rh7 interface.xml 
error: Failed to attach device from interface.xml
error: unsupported configuration: Multiqueue network is not supported for: direct

I file new bug 1313264 for this issue.


1.6. check hotunplugging

# virsh detach-interface rh7 direct
Interface detached successfully

# virsh dumpxml r7 |grep vhost -B5 -A3


# ps aux|grep vhost
qemu     14158 45.9  9.4 1758288 362084 ?      Sl   18:14   0:37 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d4bfbeef-a959-4b35-aa97-870dd9ab7b31 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-r7/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x3 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.0,addr=0x4 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x1d.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x1f.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/r7.img,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=26:28,id=hostnet0,vhost=on,vhostfds=29:30 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=52:54:00:12:2b:f6,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-r7/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
root     14566  0.0  0.0 112644   924 pts/1    S+   18:16   0:00 grep --color=auto vhost

verified on libvirt-1.3.4-1.el7.x86_64, the macvtap network support multiqueue both in hotplug and start a guest with multiqueue defined. So move the bug to verified.

Hi Michal,

Could you please help to confirm if this is expected?
Currently kernel supports up to 256 queues (see BZ 1308395#c4), and it did work for nat mode interface.(see BZ 1322661#c5) 
But for macvtap, as I tried, it can support 16 at most. When set queues=17 or more, the guest will not start and report error, I tired hotplug, the error is similiar:

# virsh start rhel
error: Failed to start domain rhel
error: cannot stat tap fd -1: Bad file descriptor

(In reply to yalzhang from comment #8)
> Hi Michal,
> 
> Could you please help to confirm if this is expected?
> Currently kernel supports up to 256 queues (see BZ 1308395#c4), and it did
> work for nat mode interface.(see BZ 1322661#c5) 
> But for macvtap, as I tried, it can support 16 at most. When set queues=17
> or more, the guest will not start and report error, I tired hotplug, the
> error is similiar:
> 
> # virsh start rhel
> error: Failed to start domain rhel
> error: cannot stat tap fd -1: Bad file descriptor

Nope! That is not expected and in fact lead me to a bug in our code. Moving back to ASSIGNED so we can pick up the fix for this case too. Good job!

Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2016-August/msg00503.html

To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-August/msg00672.html

Hi Michal, 

Verified on
libvirt-2.0.0-5.el7.x86_64

It still can not support multiqueues more than 16, when I set 17 or more, and the error message changed:

# virsh start ovmf
error: Failed to start domain ovmf
error: cannot open macvtap tap device /dev/tap20: Device or resource busy

Please take a look at it,tks~

(In reply to yalzhang from comment #13)
> Hi Michal, 
> 
> Verified on
> libvirt-2.0.0-5.el7.x86_64
> 
> It still can not support multiqueues more than 16, when I set 17 or more,
> and the error message changed:
> 
> # virsh start ovmf
> error: Failed to start domain ovmf
> error: cannot open macvtap tap device /dev/tap20: Device or resource busy
> 
> Please take a look at it,tks~

Well, this means that in kernel there's not support for 17 or more queues for macvtaps. And when looking into kernel source code I can confirm this:

include/linux/if_macvlan.h-
include/linux/if_macvlan.h-/*
include/linux/if_macvlan.h- * Maximum times a macvtap device can be opened. This can be used to
include/linux/if_macvlan.h- * configure the number of receive queue, e.g. for multiqueue virtio.
include/linux/if_macvlan.h- */
include/linux/if_macvlan.h:#define MAX_MACVTAP_QUEUES   (NR_CPUS < 16 ? NR_CPUS : 16)

Also, there are apparently different limits for tap devices and macvtap devices.

As it is kernel limitation, move the bug to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html