Bug 971214

Summary: qemu will be core dump when Storage vm migration with mutiple storage on rhel7.0
Product: Red Hat Enterprise Linux 7 Reporter: zhonglinzhang <zhzhang>
Component: qemu-kvmAssignee: Ademar Reis <areis>
Status: CLOSED DUPLICATE QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: medium    
Version: 7.0CC: acathrow, hhuang, jcody, juzhang, knoel, kwolf, michen, pbonzini, shu, sluo, virt-maint, xfu, zhzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-02 11:00:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1076185    

Description zhonglinzhang 2013-06-06 02:44:23 UTC
Description of problem:
1.after completing the configuration of src and des, sometimes qemu will be core dump on des
2. but no occur core dump, from src to des, failed to copy image.

Version-Release number of selected component (if applicable):
# uname -r
3.10.0-0.rc3.58.el7.x86_64
qemu-kvm-1.5.0-2.el7.x86_64

How reproducible:
50%

Steps to Reproduce:
1. des 
qemu-img create -f qcow2 -o compat=1.1 test1.qcow2 35G 
/usr/libexec/qemu-kvm  -cpu SandyBridge -M pc-i440fx-1.4 -enable-kvm -m 4G -smp 4,sockets=2,cores=2,threads=1 -name scalability-test -uuid 389d06a7-e631-4fae-baf4-87bdb9b5594e -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/test1.qcow3,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,drive=drive-system-disk,id=system-disk,addr=0x6    -drive file=/home/test2.qcow3,if=none,id=drive-s1-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,drive=drive-s1-disk,id=s1-disk,addr=0x7  -drive file=/home/test3.qcow3,if=none,id=drive-s2-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,drive=drive-s2-disk,id=s2-disk,addr=0x8      -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup  -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:22:15:27:54:3d,bus=pci.0,addr=0x9 -k en-us -boot menu=on -vnc :0  -vga cirrus -monitor stdio -qmp tcp:0:5555,server,nowait -incoming tcp:0:5888
{"execute": "qmp_capabilities"} 
{ "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet", "data": { "host": "10.66.4.112", "port": "7777" } } } }
(qemu) nbd_server_add -w drive-system-disk 
(qemu) nbd_server_add -w drive-s1-disk 
(qemu) nbd_server_add -w drive-s2-disk 

2. src
/usr/libexec/qemu-kvm -cpu SandyBridge -M pc-i440fx-1.4 -enable-kvm -m 4G -smp 4,sockets=2,cores=2,threads=1 -name scalability-test -uuid 389d06a7-e631-4fae-baf4-87bdb9b5594e -rtc base=localtime,clock=host,driftfix=slew -drive file=/home/rhel7cp5.qcow3,if=none,id=drive-system-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,drive=drive-system-disk,id=system-disk,addr=0x6 -drive file=/home/new1.qcow3,if=none,id=drive-s1-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,drive=drive-s1-disk,id=s1-disk,addr=0x7 -drive file=/home/new2.qcow3,if=none,id=drive-s2-disk,media=disk,format=qcow2,aio=native,werror=stop,rerror=stop -device virtio-blk-pci,bus=pci.0,drive=drive-s2-disk,id=s2-disk,addr=0x8 -netdev tap,id=hostnet0,vhost=on,script=/etc/qemu-ifup -device virtio-net-pci,netdev=hostnet0,id=virtio-net-pci0,mac=00:22:15:27:56:3d,bus=pci.0,addr=0x9 -k en-us -boot menu=on -vnc :0 -vga cirrus -monitor stdio -qmp tcp:0:5555,server,nowait
{"execute": "qmp_capabilities"}
{ "execute": "drive-mirror", "arguments": { "device": "drive-system-disk", "target": "nbd://10.66.4.112:7777/drive-system-disk", "sync": "full", "format": "raw", "mode": "existing" } }
{ "execute": "drive-mirror", "arguments": { "device": "drive-s1-disk", "target": "nbd://10.66.4.112:7777/drive-s1-disk", "sync": "full", "format": "raw", "mode": "existing" } }
{ "execute": "drive-mirror", "arguments": { "device": "drive-s2-disk", "target": "nbd://10.66.4.112:7777/drive-s2-disk", "sync": "full", "format": "raw", "mode": "existing" } }

3. migrate -d tcp:$des-host-ip:$port

Actual results:
1. des:
(qemu) qemu-system-x86_64: block/qcow2-cache.c:67: qcow2_cache_destroy: Assertion `c->entries[i].ref == 0' failed.
2. {"timestamp": {"seconds": 1370484715, "microseconds": 973943}, "event": "BLOCK_IO_ERROR", "data": {"device": "drive-system-disk", "operation": "write", "action": "stop"}}


Expected results:
qemu work normally
copy image from src to des should be work well 

Additional info:

Comment 3 Paolo Bonzini 2014-03-13 12:44:54 UTC
Kevin, could storage migration interfere with the patches you committed recently that affect caching with BDRV_O_INCOMING?  Either by making things better, or by making them much worse.

Would it make any sense for writes to clear BDRV_O_INCOMING?

Comment 5 Kevin Wolf 2014-04-01 15:56:08 UTC
Is this still reproducible with the latest packages? Looks like a potential
duplicate of bug 1048575.

Comment 6 juzhang 2014-04-02 01:23:41 UTC
(In reply to Kevin Wolf from comment #5)
> Is this still reproducible with the latest packages? Looks like a potential
> duplicate of bug 1048575.

Hi Sluo,

Could you have a try and update testing result in the bz?

Best Regards,
Junyi

Comment 7 Sibiao Luo 2014-04-02 03:27:07 UTC
Retried it on the latest qemu-kvm and kernel version with the same steps as comment #0. I tried 3 times storage vm migration with nbd which all copy image from src to dest successfully without any core dumped. So I think this issue has gone correctly.

host info:
# uname -r && rpm -q qemu-kvm-rhev
3.10.0-115.el7.x86_64
qemu-kvm-rhev-1.5.3-59.el7ev.x86_64
guest info:
# uname -r
3.10.0-115.el7.x86_64

- dest:
{"execute":"qmp_capabilities"}
{"return": {}}
{ "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet", "data": { "host": "10.66.8.202","port":"7777"}}}}
{"return": {}}

{"timestamp": {"seconds": 1396408939, "microseconds": 164689}, "event": "RESUME"}     <-------------Resume successfully.

- sorc:
{"execute":"qmp_capabilities"}
{"return": {}}
{ "execute": "drive-mirror", "arguments": { "device": "drive-data-disk", "target": "nbd://10.66.8.202:7777/drive-data-disk", "sync": "full", "format": "raw", "mode": "existing" } }
{"timestamp": {"seconds": 1396408402, "microseconds": 708874}, "event": "BLOCK_JOB_READY", "data": {"device": "drive-data-disk", "len": 10737418240, "offset": 10737418240, "speed": 0, "type": "mirror"}}
{"return": {}}
{"timestamp": {"seconds": 1396408406, "microseconds": 532662}, "event": "NIC_RX_FILTER_CHANGED", "data": {"name": "virtio-net-pci0", "path": "/machine/peripheral/virtio-net-pci0/virtio-backend"}}

{ "execute": "drive-mirror", "arguments": { "device": "drive-system-disk", "target": "nbd://10.66.8.202:7777/drive-system-disk", "sync": "full", "format": "raw", "mode": "existing" } }
{"return": {}}

{"timestamp": {"seconds": 1396408890, "microseconds": 899804}, "event": "BLOCK_JOB_READY", "data": {"device": "drive-system-disk", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "mirror"}}

{"execute": "migrate","arguments":{"uri": "tcp:10.66.8.202:5888"}}
{"return": {}}
{"timestamp": {"seconds": 1396408939, "microseconds": 65155}, "event": "STOP"}
{"timestamp": {"seconds": 1396408939, "microseconds": 70076}, "event": "SPICE_MIGRATE_COMPLETED"}
{"timestamp": {"seconds": 1396408941, "microseconds": 143644}, "event": "SHUTDOWN"}
{"timestamp": {"seconds": 1396408941, "microseconds": 144028}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-system-disk", "len": 21474836480, "offset": 21474836480, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1396408941, "microseconds": 145512}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "drive-data-disk", "len": 10737418240, "offset": 10737418240, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1396408941, "microseconds": 146769}, "event": "DEVICE_TRAY_MOVED", "data": {"device": "ide1-cd0", "tray-open": true}}
{"timestamp": {"seconds": 1396408941, "microseconds": 146849}, "event": "DEVICE_TRAY_MOVED", "data": {"device": "floppy0", "tray-open": true}}

Best Regards,
sluo

Comment 8 Kevin Wolf 2014-04-02 11:00:13 UTC

*** This bug has been marked as a duplicate of bug 1048575 ***