Bug 1099277

Summary: vectors of virtio-net-pci will be 0 when set vectors>=129
Product: Red Hat Enterprise Linux 7 Reporter: Jun Li <juli>
Component: qemu-kvmAssignee: Amos Kong <akong>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 7.0CC: acathrow, ailan, alex.williamson, amit.shah, dgilbert, hhuang, juzhang, michen, mst, qiguo, quintela, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Windows   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-08-04 06:52:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jun Li 2014-05-20 02:13:14 UTC
Description of problem:
vectors of virtio-net-pci will be 0 when set vectors>=129.

Version-Release number of selected component (if applicable):
qemu-kvm-1.5.3-60.el7_0.2.x86_64
3.10.0-123.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot qemu-kvm with following cli:
# /usr/libexec/qemu-kvm \
-netdev tap,id=dev1,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,queues=8 -device virtio-net-pci,netdev=dev1,mac=24:be:05:14:bb:aa,id=net1,vectors=129,mq=on \
-monitor stdio -nodefaults
2.using "info qtree" to check the value of vectors of virtio-net-pci.
(qemu)info qtree

Actual results:
vector of virtio-net-pci will be 0.

Expected results:
If vectors need <=128, when boot qemu-kvm with vectors >=129, could give some hints. Just like when set vectors with 10000000000, qemu-kvm will give hints like followings:
# /usr/libexec/qemu-kvm -netdev tap,id=dev1,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown,queues=8 -device virtio-net-pci,netdev=dev1,mac=24:be:05:14:bb:aa,id=net1,vectors=10000000000,mq=on -monitor stdio -nodefaults
QEMU 1.5.3 monitor - type 'help' for more information
(qemu) qemu-kvm: -device virtio-net-pci,netdev=dev1,mac=24:be:05:14:bb:aa,id=net1,vectors=10000000000,mq=on: Parameter 'vectors' expects uint32_t

Additional info:

Comment 2 Amos Kong 2014-05-22 08:54:18 UTC
From qapi-schema.json:

# @vectors: #optional number of MSI-x vectors, 0 to disable MSI-X


When hotplug a virtio-pci device, qemu will try to take over PCI BARs for the use of MSIX,
qemu only remains 4k memory for BAR & vector table, each msix entry takes 16 bytes. So it
only allows 4k /16 bytes = 128 MSIX.

When we assign a vectors that is larger than 128, the msix resource isn't enough, so msix
will be disabled.

So it's not a bug.


Related commit:
========================================================================
commit 53f949254ad2435bfd45cb0dee96f246a0bdd7e3
Author: Alex Williamson <alex.williamson>
Date:   Thu Jun 14 12:15:51 2012 -0600

    msix: Add simple BAR allocation MSIX setup functions
    
    msi_init() takes over a BAR without really specifying or allowing
    specification of how it does so.  Instead, let's split it into
    two interfaces, one fully specified, and one trivially easy.  This
    implements the latter.  msix_init_exclusive_bar() takes over
    allocating and filling a PCI BAR _exclusively_ for the use of MSIX.
    When used, the matching msi_uninit_exclusive_bar() should be used
    to tear it down.


+int msix_init_exclusive_bar(PCIDevice *dev, unsigned short nentries,
+                            uint8_t bar_nr)
+{
+    int ret;
+    char *name;
+
+    /*
+     * Migration compatibility dictates that this remains a 4k
+     * BAR with the vector table in the lower half and PBA in
+     * the upper half.  Do not use these elsewhere!
+     */
+#define MSIX_EXCLUSIVE_BAR_SIZE 4096
+#define MSIX_EXCLUSIVE_BAR_PBA_OFFSET (MSIX_EXCLUSIVE_BAR_SIZE / 2)
+
+    if (nentries * PCI_MSIX_ENTRY_SIZE > MSIX_EXCLUSIVE_BAR_PBA_OFFSET) {
+        return -EINVAL;
+    }

......

Comment 3 Amos Kong 2014-05-22 09:18:57 UTC
It's better to report an error when fail to init exclusive bars for msix.

[Qemu-devel] [PATCH] virtio-pci: report an error when disable msix
http://lists.nongnu.org/archive/html/qemu-devel/2014-05/msg04475.html

Comment 4 Amos Kong 2014-07-06 12:10:02 UTC
Upstream applied another similar patch from Fam, backport is needed.

[PATCH] virtio-pci: Report an error when msix vectors init fails

commit c7ff54825b74f27c3aac85aad540542801630d0a
Author: Fam Zheng <famz>
Date:   Mon Jun 23 17:06:25 2014 +0800

    virtio-pci: Report an error when msix vectors init fails
    
    Currently vectors silently cleared to 0 if the initialization is failed,
    but user should at least have one way to notice this.
    
    Signed-off-by: Fam Zheng <famz>
    Reviewed-by: Michael S. Tsirkin <mst>
    Signed-off-by: Michael S. Tsirkin <mst>

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index ce97514..57e1e61 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -976,6 +976,8 @@ static void virtio_pci_device_plugged(DeviceState *d)
 
     if (proxy->nvectors &&
         msix_init_exclusive_bar(&proxy->pci_dev, proxy->nvectors, 1)) {
+        error_report("unable to init msix vectors to %" PRIu32,
+                     proxy->nvectors);
         proxy->nvectors = 0;
     }

Comment 6 Amos Kong 2014-08-04 06:52:34 UTC
There are two bugs, they were found with different devices, but it's duplicated issue.

*** This bug has been marked as a duplicate of bug 1095645 ***