Bug 1263046

Summary: when drive-mirror from raw disk to qcow2 disk, guest hang
Product: Red Hat Enterprise Linux 7 Reporter: Pei Zhang <pezhang>
Component: qemu-kvm-rhevAssignee: Jeff Cody <jcody>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.2CC: areis, chayang, famz, huding, jcody, juzhang, knoel, kwolf, michen, mst, pbonzini, shu, virt-maint, xfu
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-17 06:46:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pei Zhang 2015-09-15 02:45:41 UTC
Description of problem:
When drive-mirror from raw disk to qcow2 disk, the guest hang.

Version-Release number of selected component (if applicable):
Host:rhel7.2
Kernel:3.10.0-315.el7.x86_64
qemu-kvm-rhev:qemu-kvm-rhev-2.3.0-22.el7.x86_64

Guest: rhel 6.7z
Kernel:2.6.32-573.7.1.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1.boot guest with a raw disk
# /usr/libexec/qemu-kvm -name rhel6.7z -machine pc-i440fx-rhel7.2.0,accel=kvm \
-cpu SandyBridge -m 4G,slots=256,maxmem=40G -numa node \
-smp 4,sockets=2,cores=2,threads=1 \
-uuid 82b1a01e-5f6c-4f5f-8d27-3855a74e6b61 \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=12:54:00:5c:88:61 \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16 \
-vnc 0.0.0.0:1 \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-serial unix:/tmp/monitor,server,nowait \
-drive file=/mnt/disk2/rhel6.7_virtio.raw,format=raw,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0 \

2. drive-mirror to qcow2 disk, guest hang.
{ "execute": "drive-mirror", "arguments": { "device": "drive-virtio-blk0","target": "/mnt/disk3/mirror0.qcow2", "format": "qcow2", "mode":
"absolute-paths", "sync": "full", "speed": 9223372036854775807, "on-source-error": "stop","on-target-error": "stop" } }


Actual results:
Guest hang.


Expected results:
Guest works well.

Additional info:

Comment 1 Pei Zhang 2015-09-15 05:22:54 UTC
More additional infos:

1. qeury drive-mirror job in qmp (twice)
{ "execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-blk0", "busy": false, "len": 0, "offset": 0, "paused": false, "speed": 9223372036854775807, "ready": false, "type": "mirror"}]}

{ "execute" : "query-block-jobs", "arguments" : {} }
{"return": [{"io-status": "ok", "device": "drive-virtio-blk0", "busy": false, "len": 0, "offset": 0, "paused": false, "speed": 9223372036854775807, "ready": false, "type": "mirror"}]}

2. qemu still shows  guest is running(actually hang)
(qemu) info status
VM status: running

3. In host( no other applications except qemu), always 1/8 cpus sy is 100%.
# top
top - 12:55:05 up  1:24,  4 users,  load average: 0.97, 0.58, 0.29
Tasks: 255 total,   2 running, 253 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,100.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7968276 total,  6158768 free,   981436 used,   828072 buff/cache
KiB Swap:  8257532 total,  8257532 free,        0 used.  6736312 avail Mem 

4. Also tested with qemu-kvm-rhev of low versions, below are the version and test results.
(1)qemu-kvm-rhev-2.3.0-14.el7.x86_64: guest hang, 1/8 cpus sy is 100%, (qemu works well)
(2)qemu-kvm-rhev-2.3.0-13.el7.x86_64: guest hang, 1/8 cpus sy is 100%, (qemu works well)
(3)qemu-kvm-rhev-2.3.0-12.el7.x86_64: guest hang, 1/8 cpus sy is 100%, and qemu hang
(4)qemu-kvm-rhev-2.1.2-23.el7.x86_64: guest hang, 1/8 cpus sy is 100%, and qemu hang

Comment 2 Markus Stockhausen 2015-09-15 19:37:58 UTC
Can you check if that fixes the error:

http://git.qemu.org/?p=qemu.git;a=commit;h=4c0cbd6fec7db182a6deb52d5a8a8e7b0c5cbe64

Comment 3 Karen Noel 2015-09-15 21:47:15 UTC
(In reply to Markus Stockhausen from comment #2)
> Can you check if that fixes the error:
> 
> http://git.qemu.org/?p=qemu.git;a=commit;
> h=4c0cbd6fec7db182a6deb52d5a8a8e7b0c5cbe64

commit 4c0cbd6fec7db182a6deb52d5a8a8e7b0c5cbe64
Author: Fam Zheng <famz>
Date:   Wed May 13 11:11:13 2015 +0800

    block/mirror: Sleep periodically during bitmap scanning
    
This patch is already in RHEL 7.2: qemu-kvm-rhev-2.3.0-13.el7

Bug 1233826 - issueing drive-mirror command causes monitor unresponsive
https://bugzilla.redhat.com/show_bug.cgi?id=1233826

Comment 5 Pei Zhang 2015-09-16 03:26:06 UTC
(In reply to Markus Stockhausen from comment #2)
> Can you check if that fixes the error:
> 
> http://git.qemu.org/?p=qemu.git;a=commit;
> h=4c0cbd6fec7db182a6deb52d5a8a8e7b0c5cbe64

I retested with the latest qemu-kvm-rhev-2.3.0-23.el7, the guest works well. KVM QE does not sure if this is same issue with Bug 1259229. And this bug has been fixed.

Host:
Kernel:3.10.0-315.el7.x86_64
qemu-kvm-rhev:qemu-kvm-rhev-2.3.0-23.el7.x86_64

Steps:
1. boot guest with a raw disk
# /usr/libexec/qemu-kvm -name rhel6.7z -machine pc-i440fx-rhel7.2.0,accel=kvm \
-cpu SandyBridge -m 4G,slots=256,maxmem=40G -numa node \
-smp 4,sockets=2,cores=2,threads=1 \
-uuid 82b1a01e-5f6c-4f5f-8d27-3855a74e6b61 \
-netdev tap,id=hostnet0 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=12:54:00:5c:88:61 \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16 \
-vnc 0.0.0.0:1 \
-monitor stdio \
-qmp tcp:0:5555,server,nowait \
-serial unix:/tmp/monitor,server,nowait \
-drive file=/mnt/disk4/rhel6.7_virtio.raw,format=raw,if=none,id=drive-virtio-blk0,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-blk0,id=virtio-blk0 \

2. drive-mirror to qcow2 disk, guest hang.
{ "execute": "drive-mirror", "arguments": { "device": "drive-virtio-blk0","target": "/mnt/disk3/mirror0.qcow2", "format": "qcow2", "mode":
"absolute-paths", "sync": "full", "speed": 9223372036854775807, "on-source-error": "stop","on-target-error": "stop" } }
{"return": {}}
{"timestamp": {"seconds": 1442373542, "microseconds": 77287}, "event": "BLOCK_JOB_READY", "data": {"device": "drive-virtio-blk0", "len": 21475688448, "offset": 21475688448, "speed": 9223372036854775807, "type": "mirror"}}

3. reopen 
{"execute": "block-job-complete", "arguments": { "device": "drive-virtio-blk0"} }
{"return": {}}
{"timestamp": {"seconds": 1442373572, "microseconds": 995752}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-virtio-blk0", "len": 21476147200, "offset": 21476147200, "speed": 9223372036854775807, "type": "mirror"}}

(qemu) info block
drive-virtio-blk0: /mnt/disk3/mirror0.qcow2 (qcow2)
    Cache mode:       writeback
...

4. Guest works well. After reboot guest sever times, it still works well.

Comment 6 Fam Zheng 2015-09-17 06:46:50 UTC

*** This bug has been marked as a duplicate of bug 1259229 ***