RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1240439 - Add multiqueue support for 'direct' interface types.
Summary: Add multiqueue support for 'direct' interface types.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1247444 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-06 21:26 UTC by Vlad Yasevich
Modified: 2016-11-03 18:19 UTC (History)
6 users (show)

Fixed In Version: libvirt-2.0.0-5.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 18:19:41 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2577 0 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC

Description Vlad Yasevich 2015-07-06 21:26:47 UTC 
Description of problem:
We can currently enable multi-queue VM nics by specifiying
  <driver name="vhost" queues='X'/>

and this works fine for bridges where a tap device is used.

However, this syntax is not supported for 'direct' type interfaces
or networks configured to use direct interfaces.
This makes it impossible to enabling multiqueue support with mavtap devices.

Version-Release number of selected component (if applicable):
Any libvirt version

How reproducible:


Steps to Reproduce:
1. add the driver directive to direct interface definition or to a
network that uses some set of interfaces.
2. attempt to start the domain.
3.


Actual results:


Expected results:


Additional info:
Comment 1 Michal Privoznik 2015-12-04 12:29:42 UTC 
*** Bug 1247444 has been marked as a duplicate of this bug. ***
Comment 2 Michal Privoznik 2015-12-04 12:34:47 UTC 
Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2015-December/msg00134.html
Comment 3 Michal Privoznik 2015-12-07 10:20:59 UTC 
jason,

when implementing this feature for TUN/TAP devices, I was advised that libvirt should set IFF_MULTI_QUEUE flag on each /dev/net/tun FD passed to libvirt. Is this the case here too? Libvirt's opening /dev/tapX corresponding to the macvtap device. Thanks in advance!
Comment 4 Michal Privoznik 2015-12-11 08:00:08 UTC 
Anyway, I got confirmation from MST that libvirt should do that.
So I've just pushed patches upstream:

commit 81a110edc72a42f5c6316d554a5ea64f1d8d83b0
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 11:47:22 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:44:44 2015 +0100

    qemu: Enable multiqueue for macvtaps
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1240439
    
    Ta-da! Now that we know how to open a macvtap device multiple
    times, we can finally enable the multiqueue feature. Everything
    else is already prepared (e.g. command line generation) from the
    previous iteration where the feature was implemented for
    TUN/TAP devices.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 08da97bfb9b459333d5f9efa7841ce8fe23f431d
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 11:31:17 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:44:43 2015 +0100

    virNetDevMacVLanCreateWithVPortProfile: Rework to support multiple FDs
    
    For the multiqueue on macvtaps we are going to need to open
    the device multiple times. Currently, this is not supported.
    Rework the function, so that upper layers can be reworked too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 1e90c744d5b98d00af5957819afb9d9568f9daff
Author:     Michal Privoznik <mprivozn>
AuthorDate: Tue Dec 8 13:17:26 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:44:39 2015 +0100

    virNetDevMacVLanTapSetup: Allow enabling of IFF_MULTI_QUEUE
    
    Like we are doing for TUN/TAP devices, we should do the same for
    macvtaps. Although, it's not as critical as in that case, we
    should do it for the consistency.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 136fe2f7cc7035294d270f63c4dba13fa839390c
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 11:19:32 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:50 2015 +0100

    virNetDevMacVLanTapSetup: Rework to support multiple FDs
    
    For the multiqueue on macvtaps we are going to need to open
    the device multiple times. Currently, this is not supported.
    Rework the function, so that upper layers can be reworked too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit d36897c7658cd9a744f9a999c06c7c3d9f98fab8
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 09:39:02 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:50 2015 +0100

    virNetDevMacVLanTapOpen: Rework to support multiple FDs
    
    For the multiqueue on macvtaps we are going to need to open
    the device multiple times. Currently, this is not supported.
    Rework the function, so that upper layers can be reworked too.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 025a87065fa92b9e48e9ffbe0a1f065440f85a0e
Author:     Michal Privoznik <mprivozn>
AuthorDate: Fri Dec 4 10:55:49 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:49 2015 +0100

    virNetDevMacVLanTapOpen: Slightly rework
    
    There are few outdated things. Firstly, we don't need to undergo
    the torture of fopen, fscanf and fclose just to get the interface
    index when we have nice wrapper over that: virNetDevGetIndex.
    Secondly, we don't need to have statically allocated buffer for
    the path we are opening.
    
    Signed-off-by: Michal Privoznik <mprivozn>

commit 56e2171c6fee97d98241f9143da4fcd194c3633c
Author:     Michal Privoznik <mprivozn>
AuthorDate: Thu Dec 3 11:33:55 2015 +0100
Commit:     Michal Privoznik <mprivozn>
CommitDate: Fri Dec 11 08:42:49 2015 +0100

    virNetDevMacVLanCreateWithVPortProfile: Turn vnet_hdr into flag
    
    So yet again one of integer arguments that we use as a boolean.
    Since the argument count of the function is unbearably long
    enough, lets turn those booleans into flags.
    
    Signed-off-by: Michal Privoznik <mprivozn>


v1.3.0-57-g81a110e

Moving to POST.
Comment 6 Shanzhi Yu 2016-03-02 10:20:41 UTC 
verify this bug with libvirt-1.3.1-1.el7.x86_64

1.1. prepare guest xml with macvtap/mq 

# cat rh7.xml |grep vhost -B 4
    <interface type='network'>
      <mac address='52:54:00:4c:5a:62'/>
      <source network='net-bridge-bridge'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>

# virt-xml-validate rh7.xml 
rh7.xml validates


1.2. define/start guest

# virsh define rh7.xml 
Domain rh7 defined from rh7.xml

# virsh start rh7 
Domain rh7 started


# virsh dumpxml rh7|grep vhost -B 5
    <interface type='direct'>
      <mac address='52:54:00:4c:5a:62'/>
      <source network='net-bridge-bridge' dev='enp2s0' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>


1.3. check qemu cmd line

# ps aux|grep vhost 
qemu     23093 73.7  9.0 1752304 345100 ?      Sl   17:08   0:20 /usr/libexec/qemu-kvm -name rh7 -S -machine pc-q35-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d5ca4513-2e9d-4577-b5dc-2d736a5bceb8 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-rh7/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x1d.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/rh7.img,format=qcow2,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pcie.0,multifunction=on,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=25:27,id=hostnet0,vhost=on,vhostfds=28:29 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=52:54:00:4c:5a:62,bus=pci.2,addr=0x2 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-rh7/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pcie.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pcie.0,addr=0x6 -msg timestamp=on
root     23096  0.0  0.0      0     0 ?        S    17:08   0:00 [vhost-23093]
root     23097  0.0  0.0      0     0 ?        S    17:08   0:00 [vhost-23093]

1.4. login guest to check
# ethtool -i eth0
driver: virtio_net
version: 1.0.0
firmware-version: 
bus-info: 0000:02:02.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
# ethtool -L eth0 combined 2
# ethtool -L eth0 combined 1
# ethtool -L eth0 combined 3
Cannot set device channel parameters: Invalid argument

1.5. check hotplug

# cat interface.xml 
   <interface type='direct'>
      <source network='net-bridge-bridge' dev='enp2s0' mode='bridge'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
    </interface>
# virsh attach-device rh7 interface.xml 
error: Failed to attach device from interface.xml
error: unsupported configuration: Multiqueue network is not supported for: direct

I file new bug 1313264 for this issue.


1.6. check hotunplugging

# virsh detach-interface rh7 direct
Interface detached successfully

# virsh dumpxml r7 |grep vhost -B5 -A3


# ps aux|grep vhost
qemu     14158 45.9  9.4 1758288 362084 ?      Sl   18:14   0:37 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d4bfbeef-a959-4b35-aa97-870dd9ab7b31 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-r7/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pci-bridge,chassis_nr=1,id=pci.1,bus=pci.0,addr=0x3 -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.0,addr=0x4 -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x1d.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x1d -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x1d.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x1d.0x2 -device ahci,id=sata0,bus=pci.0,addr=0x1f.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x1 -drive file=/var/lib/libvirt/images/r7.img,format=raw,if=none,id=drive-virtio-disk0,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,multifunction=on,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fds=26:28,id=hostnet0,vhost=on,vhostfds=29:30 -device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=52:54:00:12:2b:f6,bus=pci.0,addr=0x7 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-r7/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel1,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
root     14566  0.0  0.0 112644   924 pts/1    S+   18:16   0:00 grep --color=auto vhost
Comment 7 yalzhang@redhat.com 2016-05-26 07:19:26 UTC 
verified on libvirt-1.3.4-1.el7.x86_64, the macvtap network support multiqueue both in hotplug and start a guest with multiqueue defined. So move the bug to verified.
Comment 8 yalzhang@redhat.com 2016-08-03 02:43:13 UTC 
Hi Michal,

Could you please help to confirm if this is expected?
Currently kernel supports up to 256 queues (see BZ 1308395#c4), and it did work for nat mode interface.(see BZ 1322661#c5) 
But for macvtap, as I tried, it can support 16 at most. When set queues=17 or more, the guest will not start and report error, I tired hotplug, the error is similiar:

# virsh start rhel
error: Failed to start domain rhel
error: cannot stat tap fd -1: Bad file descriptor
Comment 9 Michal Privoznik 2016-08-09 17:15:30 UTC 
(In reply to yalzhang from comment #8)
> Hi Michal,
> 
> Could you please help to confirm if this is expected?
> Currently kernel supports up to 256 queues (see BZ 1308395#c4), and it did
> work for nat mode interface.(see BZ 1322661#c5) 
> But for macvtap, as I tried, it can support 16 at most. When set queues=17
> or more, the guest will not start and report error, I tired hotplug, the
> error is similiar:
> 
> # virsh start rhel
> error: Failed to start domain rhel
> error: cannot stat tap fd -1: Bad file descriptor

Nope! That is not expected and in fact lead me to a bug in our code. Moving back to ASSIGNED so we can pick up the fix for this case too. Good job!
Comment 10 Michal Privoznik 2016-08-09 17:34:06 UTC 
Patches proposed upstream:

https://www.redhat.com/archives/libvir-list/2016-August/msg00503.html
Comment 11 Michal Privoznik 2016-08-10 15:06:27 UTC 
To POST:

http://post-office.corp.redhat.com/archives/rhvirt-patches/2016-August/msg00672.html
Comment 13 yalzhang@redhat.com 2016-08-11 04:56:49 UTC 
Hi Michal, 

Verified on
libvirt-2.0.0-5.el7.x86_64

It still can not support multiqueues more than 16, when I set 17 or more, and the error message changed:

# virsh start ovmf
error: Failed to start domain ovmf
error: cannot open macvtap tap device /dev/tap20: Device or resource busy

Please take a look at it,tks~
Comment 14 Michal Privoznik 2016-08-11 07:24:15 UTC 
(In reply to yalzhang from comment #13)
> Hi Michal, 
> 
> Verified on
> libvirt-2.0.0-5.el7.x86_64
> 
> It still can not support multiqueues more than 16, when I set 17 or more,
> and the error message changed:
> 
> # virsh start ovmf
> error: Failed to start domain ovmf
> error: cannot open macvtap tap device /dev/tap20: Device or resource busy
> 
> Please take a look at it,tks~

Well, this means that in kernel there's not support for 17 or more queues for macvtaps. And when looking into kernel source code I can confirm this:

include/linux/if_macvlan.h-
include/linux/if_macvlan.h-/*
include/linux/if_macvlan.h- * Maximum times a macvtap device can be opened. This can be used to
include/linux/if_macvlan.h- * configure the number of receive queue, e.g. for multiqueue virtio.
include/linux/if_macvlan.h- */
include/linux/if_macvlan.h:#define MAX_MACVTAP_QUEUES   (NR_CPUS < 16 ? NR_CPUS : 16)

Also, there are apparently different limits for tap devices and macvtap devices.
Comment 15 yalzhang@redhat.com 2016-08-11 08:54:34 UTC 
As it is kernel limitation, move the bug to verified.
Comment 17 errata-xmlrpc 2016-11-03 18:19:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html
Note You need to log in before you can comment on or make changes to this bug.