Bug 987025 - enable MSI-X for virtio-scsi
enable MSI-X for virtio-scsi
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
6.5
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Asias He
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-22 11:20 EDT by Paolo Bonzini
Modified: 2013-12-05 03:18 EST (History)
12 users (show)

See Also:
Fixed In Version: qemu-kvm-0.12.1.2-2.382.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-11-21 02:04:54 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Paolo Bonzini 2013-07-22 11:20:34 EDT
When virtio-scsi will support multiqueue in RHEL-7, the default number of interrupt vectors will have to change compared to RHEL-6.4.  This is because the default number of interrupts is 2, which is not enough to actually enable usage of multiple queues in the driver (the driver needs a separate interrupt for each request queue).

Backporting upstream commit 4c205d0cb1c8bde5a53f6acceda74dae1043a197 will simplify a bit maintainance of RHEL6-compatible machine types in RHEL-7.
Comment 1 Asias He 2013-07-22 20:51:51 EDT
Paolo, Do we need this as well?

commit d2ad7dd46e72118a577e16b3c6dffdc43c961476
Author: Paolo Bonzini <pbonzini@redhat.com>
Date:   Fri Apr 6 10:39:46 2012 +0200

    virtio-scsi: add multiqueue capability
    
    Adding multiqueue is as simple as creating more than one virtqueues,
    and saving the queue number for each request.
    
    Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Comment 4 Asias He 2013-07-23 03:22:51 EDT
(In reply to Paolo Bonzini from comment #0)
> When virtio-scsi will support multiqueue in RHEL-7, the default number of
> interrupt vectors will have to change compared to RHEL-6.4.  This is because
> the default number of interrupts is 2, which is not enough to actually
> enable usage of multiple queues in the driver (the driver needs a separate
> interrupt for each request queue).
> 
> Backporting upstream commit 4c205d0cb1c8bde5a53f6acceda74dae1043a197 will
> simplify a bit maintainance of RHEL6-compatible machine types in RHEL-7.

With commit 4c205d0cb1c8bde5a53f6acceda74dae1043a197 only or withoutit, we have this error in guest:
[    0.773819] virtio_scsi: probe of virtio1 failed with error -2

if i set num_queues to 2, 
  
   qemu-kvm -device virtio-scsi-pci,vectors=2,num_queues=2

We told guest we have 2 request queues, but actually we only have one!

Paolo, what do you think of backporting multi-queue support to 6.5 too?
Comment 5 Paolo Bonzini 2013-07-23 09:36:58 EDT
I don't think we need multi-queue in RHEL-6.  The upstream commit 4c205d0c can be heavily simplified in RHEL-6.  num_queues will always be 1, so you get:

- vectors=2 for RHEL <= 6.4 (with a compat property);

- vectors=4 for RHEL >= 6.5 (by changing the default; this matches upstream's num_queues + 3, with num_queues == 1).
Comment 7 Asias He 2013-07-24 01:21:05 EDT
(In reply to Paolo Bonzini from comment #5)
> I don't think we need multi-queue in RHEL-6.  The upstream commit 4c205d0c
> can be heavily simplified in RHEL-6.  num_queues will always be 1, so you
> get:
> 
> - vectors=2 for RHEL <= 6.4 (with a compat property);
> 
> - vectors=4 for RHEL >= 6.5 (by changing the default; this matches
> upstream's num_queues + 3, with num_queues == 1).

So you want this for rhel6.5?

--- a/hw/virtio-pci.c
+++ b/hw/virtio-pci.c
@@ -955,7 +955,10 @@ static int virtio_scsi_init_pci(PCIDevice *pci_dev)
         return -EINVAL;
     }
 
-    vdev->nvectors = proxy->nvectors;
+    vdev->nvectors = proxy->nvectors == DEV_NVECTORS_UNSPECIFIED
+                                        ? 1 + 3
+                                        : proxy->nvectors;
+
     virtio_init_pci(proxy, vdev,
                     PCI_VENDOR_ID_REDHAT_QUMRANET,
                     PCI_DEVICE_ID_VIRTIO_SCSI,
@@ -1056,7 +1059,7 @@ static PCIDeviceInfo virtio_info[] = {
         .init      = virtio_scsi_init_pci,
         .exit      = virtio_scsi_exit_pci,
         .qdev.props = (Property[]) {
-            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2),
+            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, DEV_NVECTORS_UNSPECIFIED),
             DEFINE_VIRTIO_SCSI_PROPERTIES(VirtIOPCIProxy, host_features, scsi),
             DEFINE_PROP_END_OF_LIST(),
Comment 8 Paolo Bonzini 2013-07-24 10:55:11 EDT
I want more simply this:

-            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2),
+            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 4),

Plus setting the compat property.
Comment 9 Paolo Bonzini 2013-07-24 10:57:59 EDT
Re. test case:

RHEL 6.4 will show in /proc/interrupts something like

26:            0   PCI-MSI-edge      virtio1-config
27:          256   PCI-MSI-edge      virtio1-virtqueues

RHEL 6.5 will show

28:            0   PCI-MSI-edge      virtio2-config
29:          256   PCI-MSI-edge      virtio2-control
30:            0   PCI-MSI-edge      virtio2-event
31:          256   PCI-MSI-edge      virtio2-request
Comment 17 Qunfang Zhang 2013-07-30 06:22:04 EDT
Reproduced with qemu-kvm-0.12.1.2-2.379.el6.x86_64. 

1. Boot a guest with virtio-scsi disk:

-drive file=/home/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1" -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1

2. Inside guest:
#cat /proc/interrupts:

 24:          0          0   PCI-MSI-edge      virtio0-config
 25:       5831          0   PCI-MSI-edge      virtio0-virtqueues


Verified on qemu-kvm-0.12.1.2-2.382.el6.x86_64: 

Test with virtio scsi hard disk:

#cat /proc/interrupts:
 24:          0          0   PCI-MSI-edge      virtio0-config
 25:          0          0   PCI-MSI-edge      virtio0-control
 26:          0          0   PCI-MSI-edge      virtio0-event
 27:       6554          0   PCI-MSI-edge      virtio0-request

Test with virtio scsi cdrom:

#cat /proc/interrupts:
 24:          0          0   PCI-MSI-edge      virtio0-config
 25:          0          0   PCI-MSI-edge      virtio0-control
 26:          0          0   PCI-MSI-edge      virtio0-event
 27:        528          0   PCI-MSI-edge      virtio0-request

So this bug is fixed.
Comment 19 errata-xmlrpc 2013-11-21 02:04:54 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1553.html

Note You need to log in before you can comment on or make changes to this bug.