Bug 987025

Summary: enable MSI-X for virtio-scsi
Product: Red Hat Enterprise Linux 6 Reporter: Paolo Bonzini <pbonzini>
Component: qemu-kvmAssignee: Asias He <asias>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.5CC: acathrow, areis, bsarathy, chayang, huding, juzhang, michen, mkenneth, pbonzini, qzhang, virt-maint, xtian
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-0.12.1.2-2.382.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-21 07:04:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Paolo Bonzini 2013-07-22 15:20:34 UTC
When virtio-scsi will support multiqueue in RHEL-7, the default number of interrupt vectors will have to change compared to RHEL-6.4.  This is because the default number of interrupts is 2, which is not enough to actually enable usage of multiple queues in the driver (the driver needs a separate interrupt for each request queue).

Backporting upstream commit 4c205d0cb1c8bde5a53f6acceda74dae1043a197 will simplify a bit maintainance of RHEL6-compatible machine types in RHEL-7.

Comment 1 Asias He 2013-07-23 00:51:51 UTC
Paolo, Do we need this as well?

commit d2ad7dd46e72118a577e16b3c6dffdc43c961476
Author: Paolo Bonzini <pbonzini>
Date:   Fri Apr 6 10:39:46 2012 +0200

    virtio-scsi: add multiqueue capability
    
    Adding multiqueue is as simple as creating more than one virtqueues,
    and saving the queue number for each request.
    
    Signed-off-by: Paolo Bonzini <pbonzini>

Comment 4 Asias He 2013-07-23 07:22:51 UTC
(In reply to Paolo Bonzini from comment #0)
> When virtio-scsi will support multiqueue in RHEL-7, the default number of
> interrupt vectors will have to change compared to RHEL-6.4.  This is because
> the default number of interrupts is 2, which is not enough to actually
> enable usage of multiple queues in the driver (the driver needs a separate
> interrupt for each request queue).
> 
> Backporting upstream commit 4c205d0cb1c8bde5a53f6acceda74dae1043a197 will
> simplify a bit maintainance of RHEL6-compatible machine types in RHEL-7.

With commit 4c205d0cb1c8bde5a53f6acceda74dae1043a197 only or withoutit, we have this error in guest:
[    0.773819] virtio_scsi: probe of virtio1 failed with error -2

if i set num_queues to 2, 
  
   qemu-kvm -device virtio-scsi-pci,vectors=2,num_queues=2

We told guest we have 2 request queues, but actually we only have one!

Paolo, what do you think of backporting multi-queue support to 6.5 too?

Comment 5 Paolo Bonzini 2013-07-23 13:36:58 UTC
I don't think we need multi-queue in RHEL-6.  The upstream commit 4c205d0c can be heavily simplified in RHEL-6.  num_queues will always be 1, so you get:

- vectors=2 for RHEL <= 6.4 (with a compat property);

- vectors=4 for RHEL >= 6.5 (by changing the default; this matches upstream's num_queues + 3, with num_queues == 1).

Comment 7 Asias He 2013-07-24 05:21:05 UTC
(In reply to Paolo Bonzini from comment #5)
> I don't think we need multi-queue in RHEL-6.  The upstream commit 4c205d0c
> can be heavily simplified in RHEL-6.  num_queues will always be 1, so you
> get:
> 
> - vectors=2 for RHEL <= 6.4 (with a compat property);
> 
> - vectors=4 for RHEL >= 6.5 (by changing the default; this matches
> upstream's num_queues + 3, with num_queues == 1).

So you want this for rhel6.5?

--- a/hw/virtio-pci.c
+++ b/hw/virtio-pci.c
@@ -955,7 +955,10 @@ static int virtio_scsi_init_pci(PCIDevice *pci_dev)
         return -EINVAL;
     }
 
-    vdev->nvectors = proxy->nvectors;
+    vdev->nvectors = proxy->nvectors == DEV_NVECTORS_UNSPECIFIED
+                                        ? 1 + 3
+                                        : proxy->nvectors;
+
     virtio_init_pci(proxy, vdev,
                     PCI_VENDOR_ID_REDHAT_QUMRANET,
                     PCI_DEVICE_ID_VIRTIO_SCSI,
@@ -1056,7 +1059,7 @@ static PCIDeviceInfo virtio_info[] = {
         .init      = virtio_scsi_init_pci,
         .exit      = virtio_scsi_exit_pci,
         .qdev.props = (Property[]) {
-            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2),
+            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, DEV_NVECTORS_UNSPECIFIED),
             DEFINE_VIRTIO_SCSI_PROPERTIES(VirtIOPCIProxy, host_features, scsi),
             DEFINE_PROP_END_OF_LIST(),

Comment 8 Paolo Bonzini 2013-07-24 14:55:11 UTC
I want more simply this:

-            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 2),
+            DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors, 4),

Plus setting the compat property.

Comment 9 Paolo Bonzini 2013-07-24 14:57:59 UTC
Re. test case:

RHEL 6.4 will show in /proc/interrupts something like

26:            0   PCI-MSI-edge      virtio1-config
27:          256   PCI-MSI-edge      virtio1-virtqueues

RHEL 6.5 will show

28:            0   PCI-MSI-edge      virtio2-config
29:          256   PCI-MSI-edge      virtio2-control
30:            0   PCI-MSI-edge      virtio2-event
31:          256   PCI-MSI-edge      virtio2-request

Comment 17 Qunfang Zhang 2013-07-30 10:22:04 UTC
Reproduced with qemu-kvm-0.12.1.2-2.379.el6.x86_64. 

1. Boot a guest with virtio-scsi disk:

-drive file=/home/RHEL-Server-6.4-64-virtio.qcow2,if=none,id=drive-system-disk,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop,serial="QEMU-DISK1" -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1

2. Inside guest:
#cat /proc/interrupts:

 24:          0          0   PCI-MSI-edge      virtio0-config
 25:       5831          0   PCI-MSI-edge      virtio0-virtqueues


Verified on qemu-kvm-0.12.1.2-2.382.el6.x86_64: 

Test with virtio scsi hard disk:

#cat /proc/interrupts:
 24:          0          0   PCI-MSI-edge      virtio0-config
 25:          0          0   PCI-MSI-edge      virtio0-control
 26:          0          0   PCI-MSI-edge      virtio0-event
 27:       6554          0   PCI-MSI-edge      virtio0-request

Test with virtio scsi cdrom:

#cat /proc/interrupts:
 24:          0          0   PCI-MSI-edge      virtio0-config
 25:          0          0   PCI-MSI-edge      virtio0-control
 26:          0          0   PCI-MSI-edge      virtio0-event
 27:        528          0   PCI-MSI-edge      virtio0-request

So this bug is fixed.

Comment 19 errata-xmlrpc 2013-11-21 07:04:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-1553.html