Bug 1250798

Summary: Boot guest with virtio-blk data-plane, do "drive-mirror", guest cannot shutdown and qemu-kvm hang.
Product: Red Hat Enterprise Linux 7 Reporter: huiqingding <huding>
Component: qemu-kvm-rhevAssignee: Stefan Hajnoczi <stefanha>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 7.2CC: huding, juzhang, knoel, stefanha, virt-maint, xfu, yama
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-06-21 13:53:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description huiqingding 2015-08-06 03:27:05 UTC
Description of problem:
Boot guest with virtio-blk data-plane, do "drive-mirror", guest cannot shutdown and qemu-kvm hang.

Version-Release number of selected component (if applicable):
kernel-3.10.0-302.el7.x86_64
qemu-kvm-rhev-2.3.0-15.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. boot a guest with virtio-blk data-plane
# /usr/libexec/qemu-kvm -name rhel7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 4096 -cpu SandyBridge -realtime mlock=on -sandbox off -smp 4,maxcpus=4,sockets=4,cores=1,threads=1 -object iothread,id=iothread0 -drive file=/home/rhel7.2.qcow2,if=none,id=data-disk1,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=data-disk1,iothread=iothread0,bus=pci.0,addr=0x6,id=data-disk1 -drive file=/home/data-disk.qcow2,if=none,id=data-disk2,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=data-disk2,iothread=iothread0,bus=pci.0,addr=0x7,id=data-disk2 -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=b6:72:66:f5:7d:93,bus=pci.0,addr=0x9 -device sga -spice port=5910,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -qmp tcp:0:4466,server,nowait -serial unix:/tmp/ttym,server,nowait -monitor stdio

2. do drive-mirror
{"execute":"qmp_capabilities"}
{"return": {}}
{ "execute": "drive-mirror", "arguments": { "device": "data-disk2", "target": "/root/sn1", "format": "qcow2", "sync": "full"}}
{"return": {}}

3. inside guest, do shutdown
# shutdown -h now

Actual results:
after step2, "drive-mirror" can return without error info
after step3, guest cannot shutdown and qemu-kvm hang.

Expected results:
after step3, guest can shutdown and qemu-kvm can quit without error info.

Additional info:
On host, check the serial console:

# nc -U /tmp/ttym
Red Hat Enterprise Linux Server 7.2 Beta (Maipo)
Kernel 3.10.0-302.el7.x86_64 on an x86_64

dhcp-9-29 login: Powering off.
[   90.737329] Power down.

Comment 1 huiqingding 2015-08-06 03:32:31 UTC
Remove virtio-blk data-plane and boot the guest:
/usr/libexec/qemu-kvm -name rhel7 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -m 4096 -cpu SandyBridge -realtime mlock=on -sandbox off -smp 4,maxcpus=4,sockets=4,cores=1,threads=1  -drive file=/home/rhel7.2.qcow2,if=none,id=data-disk1,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=data-disk1,bus=pci.0,addr=0x6,id=data-disk1 -drive file=/home/data-disk.qcow2,if=none,id=data-disk2,format=qcow2,cache=none,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,drive=data-disk2,bus=pci.0,addr=0x7,id=data-disk2 -netdev tap,id=hostnet0,vhost=on,script=/etc/ovs-ifup,downscript=/etc/ovs-ifdown -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=b6:72:66:f5:7d:93,bus=pci.0,addr=0x9 -device sga -spice port=5910,password=redhat-vga,disable-ticketing -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -qmp tcp:0:4466,server,nowait -serial unix:/tmp/ttym,server,nowait -monitor stdio

Do drive-mirror and shutdown guest same as step2 and step3 in comment #0, the guest can shutdown and qemu-kvm can quit normally.

Comment 3 Stefan Hajnoczi 2015-09-11 16:27:48 UTC
Looks like a bdrv_set_aio_context() or bdrv_drain_all() issue.  Deferring to RHEL 7.2 since the scenario to trigger it doesn't appear critical enough for an exception.

Comment 5 Stefan Hajnoczi 2016-06-21 13:53:04 UTC
This bug has the same root cause as bz#1265179.  I have posted a backport for that bug.

*** This bug has been marked as a duplicate of bug 1265179 ***