Bug 1320473 - virtio-pci doesn't reset VQ correctly at device reset
Summary: virtio-pci doesn't reset VQ correctly at device reset
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm-rhev
Version: 7.3
Hardware: x86_64
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Fam Zheng
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-23 10:09 UTC by Han Han
Modified: 2016-11-07 21:02 UTC (History)
13 users (show)

Fixed In Version: QEMU 2.6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-07 21:02:08 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2673 normal SHIPPED_LIVE qemu-kvm-rhev bug fix and enhancement update 2016-11-08 01:06:13 UTC

Description Han Han 2016-03-23 10:09:25 UTC
Description of problem:
For a virtio disk(non-system disk), set disable-legacy=on,disable-modern=off. The disk cannot be found in VM. 

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.5.0-2.el7.x86_64
libvirt-1.3.2-1.el7.x86_64
kernel-3.10.0-330.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a VM with following xml:
<disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/qemu-virtio-2.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
  <qemu:commandline>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/tmp/haha,format=raw,if=none,id=drive-virtio-disk0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2,disable-legacy=true,disable-modern=false'/>
    <qemu:arg value='-monitor'/>
    <qemu:arg value='telnet:0.0.0.0:1234,server,nowait'/>
  </qemu:commandline>

The hda is VM's os image.

2. Check the virtio disk:
In guest,
# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   10G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0  9.5G  0 part 
  ├─fedora-root 253:0    0  8.5G  0 lvm  /
  └─fedora-swap 253:1    0    1G  0 lvm  [SWAP]

# lspci
00:03.0 SCSI storage controller: Red Hat, Inc Device 1042 (rev 01)
        Subsystem: Red Hat, Inc Device 1100
        Physical Slot: 3
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at fc056000 (32-bit, non-prefetchable) [size=4K]
        Memory at fe000000 (64-bit, prefetchable) [size=8M]
        Capabilities: [98] MSI-X: Enable- Count=2 Masked-
        Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
        Capabilities: [70] Vendor Specific Information: VirtIO: Notify
        Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
        Capabilities: [50] Vendor Specific Information: VirtIO: ISR
        Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
        Kernel driver in use: virtio-pci
        Kernel modules: virtio_pci

# dmesg |grep virtio -i
[    4.597846] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    4.739341] virtio-pci 0000:00:08.0: virtio_pci: leaving for legacy driver
[    4.746057] virtio_blk: probe of virtio0 failed with error -2

Actual results:
as step2

Expected results:
virtio1.0 disk works well.

Additional info:
qemu-kvm-rhev-2.3.0-31.el7_2.10.x86_64 works well.
virtio0.9(disable-legacy=off disable-modern=on) is ok in qemu-kvm-rhev-2.5.0-2.el7.x86_64.

Comment 3 Han Han 2016-03-23 10:59:40 UTC
The bug can also be reproduced on qemu-kvm-2.5.0-11.fc25.x86_64 qemu-kvm-rhev-2.5.0-3.el7.x86_64

Comment 4 Fam Zheng 2016-04-06 00:38:12 UTC
Can you try with upstream QEMU? If that works, it's fixed in QEMU 2.6.

There is a reset bug fixed recently:

commit 75fd6f13af8513f1e14add754549141e415f8704
Author: Gerd Hoffmann <kraxel@redhat.com>
Date:   Thu Jan 28 16:08:07 2016 +0100

    virtio-pci: call pci reset variant when guest requests reset.

    Actually fixes linux not finding virtio 1.0 device virtqueues after
    reboot.  Which is new I think, any chance linux kernel virtio code
    became more strict in 4.3?

    Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
    Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
    Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
    Reviewed-by: Fam Zheng <famz@redhat.com>
    Tested-by: Fam Zheng <famz@redhat.com>

Comment 5 Han Han 2016-04-06 05:27:57 UTC
(In reply to Fam Zheng from comment #4)
> Can you try with upstream QEMU? If that works, it's fixed in QEMU 2.6.
> 
> There is a reset bug fixed recently:
> 
> commit 75fd6f13af8513f1e14add754549141e415f8704
> Author: Gerd Hoffmann <kraxel@redhat.com>
> Date:   Thu Jan 28 16:08:07 2016 +0100
> 
>     virtio-pci: call pci reset variant when guest requests reset.
> 
>     Actually fixes linux not finding virtio 1.0 device virtqueues after
>     reboot.  Which is new I think, any chance linux kernel virtio code
>     became more strict in 4.3?
> 
>     Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
>     Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>     Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>     Reviewed-by: Fam Zheng <famz@redhat.com>
>     Tested-by: Fam Zheng <famz@redhat.com>

Well, I built qemu-2.6.0-rc0 and test as comment1. There is no problem. Disk is ok to be read&write with virtio1.0 on.

Comment 6 Fam Zheng 2016-04-06 06:01:55 UTC
OK, thanks for the testing! Let's wait for the 7.3 rebase to pick this up. Setting to POST therefore.

Comment 8 huiqingding 2016-05-19 08:42:12 UTC
Reproduce this bug using the following version:
qemu-kvm-rhev-2.5.0-2.el7.x86_64
kernel-3.10.0-393.el7.x86_64

1. Boot rhel7.3 guest with virtio-blk disk, set disable-legacy=on,disable-modern=off.
# /usr/libexec/qemu-kvm -name qemu,debug-threads=on -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off,vmport=off -cpu Penryn -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid d597f0f5-e78f-4d58-a9f4-37527c64dc0c -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-34-qemu/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/home/qemu-1.qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=29,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:95:e1:20,bus=pci.0,addr=0x9 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channel/target/domain-34-qemu/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -spice port=5901,addr=0.0.0.0,disable-ticketing,image-compression=off,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -drive file=/tmp/haha,format=raw,if=none,id=drive-virtio-disk1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x13,drive=drive-virtio-disk1,id=virtio-disk1,bootindex=2,disable-legacy=true,disable-modern=false -msg timestamp=on -monitor stdio

2. Inside guest, the virtio-blk disk cannot be found.
# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   10G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0  9.5G  0 part 
  ├─fedora-root 253:0    0  8.5G  0 lvm  /
  └─fedora-swap 253:1    0    1G  0 lvm  [SWAP]

# dmesg |grep virtio -i
[    4.597846] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    4.739341] virtio-pci 0000:00:08.0: virtio_pci: leaving for legacy driver
[    4.746057] virtio_blk: probe of virtio0 failed with error -2

Comment 9 huiqingding 2016-05-19 08:48:18 UTC
Verify this bug using the following version:
qemu-kvm-rhev-2.6.0-1.el7.x86_64
kernel-3.10.0-393.el7.x86_64

Use the commandline of comment8 to boot RHEL7.3 guest, inside guest, the virtio-blk disk can be found.
# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           252:0    0    3G  0 disk 
├─vda1        252:1    0  500M  0 part /boot
└─vda2        252:2    0  2.5G  0 part 
  ├─rhel-root 253:0    0  2.2G  0 lvm  /
  └─rhel-swap 253:1    0  308M  0 lvm  [SWAP]
vdb           252:16   0  100M  0 disk
# dmesg|grep virtio -i
[    1.717794] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    1.864379] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    1.903274] virtio-pci 0000:00:08.0: virtio_pci: leaving for legacy driver
[    1.927226] virtio-pci 0000:00:03.0: irq 24 for MSI/MSI-X
[    1.927241] virtio-pci 0000:00:03.0: irq 25 for MSI/MSI-X
[    1.929059] virtio-pci 0000:00:05.0: irq 26 for MSI/MSI-X
[    1.929073] virtio-pci 0000:00:05.0: irq 27 for MSI/MSI-X
[    1.931968] virtio-pci 0000:00:13.0: irq 28 for MSI/MSI-X
[    1.931982] virtio-pci 0000:00:13.0: irq 29 for MSI/MSI-X

# mount /dev/vdb /mnt
# cd /mnt
# dd if=/dev/zero of=file bs=1M count=10
# ll file
-rw-r--r--. 1 root root 10485760 May 19 16:45 file

Additional info:
The cpu of host is as following:
# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Model name:            Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
Stepping:              7
CPU MHz:               1599.726
BogoMIPS:              6784.95
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

Comment 11 Han Han 2016-08-05 09:34:08 UTC
Hi fam, I what to know qemu will enable virtio-legacy or virtio-modern or both when not setting disable-legacy and disable-modern flag?

Comment 12 Fam Zheng 2016-08-08 02:47:05 UTC
This can be seen from the code

hw/virtio/virtio-pci.c:

    DEFINE_PROP_BIT("disable-legacy", VirtIOPCIProxy, flags,
                    VIRTIO_PCI_FLAG_DISABLE_LEGACY_BIT, false),
    DEFINE_PROP_BIT("disable-modern", VirtIOPCIProxy, flags,
                    VIRTIO_PCI_FLAG_DISABLE_MODERN_BIT, true),

disable-legacy is false by default. disable-mordern is true by default. So, by default we use legacy.

Comment 14 errata-xmlrpc 2016-11-07 21:02:08 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html


Note You need to log in before you can comment on or make changes to this bug.