Bug 1652906
| Summary: | Qemu core dump when do migration after hot plugging a backend image with 'blockdev-add' (without the frontend) | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | aihua liang <aliang> |
| Component: | qemu-kvm | Assignee: | Virtualization Maintenance <virt-maint> |
| qemu-kvm sub component: | General | QA Contact: | Virtualization Bugs <virt-bugs> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | medium | ||
| Priority: | high | CC: | aliang, chayang, coli, juzhang, knoel, ngu, qzhang, rbalakri, ribarry, virt-maint, ymankad |
| Version: | --- | Keywords: | TestOnly |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-05-29 16:04:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1633536 | ||
| Bug Blocks: | |||
Test it on rhel7.6, also hit this issue.
kernel version: 3.10.0-957.1.2.el7.x86_64
qemu-kvm version: qemu-kvm-rhev-2.12.0-19.el7.x86_64
Test steps:
1.Start guest in src and set its migration capabilities:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idkp9HYI \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \
-netdev tap,id=idDdrbRp,vhost=on \
-m 8192 \
-smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \
-cpu 'IvyBridge',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=localtime,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:3000,server,nowait \
-blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2 \
-device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \
-blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 \
-device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
-blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 \
-device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \
qmp cmds:
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}}
2. Start guest in dst with "-incoming" param and set its migration capabilities:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9q,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=idkp9HYI \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \
-netdev tap,id=idDdrbRp,vhost=on \
-m 8192 \
-smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \
-cpu 'IvyBridge',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :1 \
-rtc base=localtime,clock=host,driftfix=slew \
-boot order=cdn,once=c,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:3001,server,nowait \
-incoming tcp:0:5000 \
-blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2 \
-device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \
-blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 \
-device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
-blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 \
-device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \
qmp cmds:
{"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}}
3. Create a full backup image and a dirty bitmap in src:
{'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn1','size':21474836480},'job-id':'job1'}}
{'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn1','filename':'/root/sn1'}}
{'execute':'blockdev-create','arguments':{'options': {'cluster-size':524288, 'driver': 'qcow2','file':'drive_sn1','size':21474836480},'job-id':'job2'}}
{'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn1','file':'drive_sn1'}}
{'execute':'job-dismiss','arguments':{'id':'job1'}}
{'execute':'job-dismiss','arguments':{'id':'job2'}}
{"execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": {"node": "drive_image1", "name": "bitmap0"} }, {"type": "blockdev-backup", "data":{ "device": "drive_image1", "sync":"full","target":"sn1","job-id":"j1"}}]}}
4. Check bitmap info:
{"execute":"query-block"}
{"execute":"x-debug-block-dirty-bitmap-sha256","arguments":{"node":"drive_image1","name":"bitmap0"}}
5. Do migration from src to dst:
{"execute": "migrate","arguments":{"uri": "tcp:10.66.144.34:5000"}}
*****output of src*************
{"timestamp": {"seconds": 1543297362, "microseconds": 315207}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}}
{"timestamp": {"seconds": 1543297362, "microseconds": 430038}, "event": "MIGRATION_PASS", "data": {"pass": 1}}
{"timestamp": {"seconds": 1543297362, "microseconds": 430125}, "event": "MIGRATION", "data": {"status": "active"}}
{"timestamp": {"seconds": 1543297426, "microseconds": 680522}, "event": "MIGRATION_PASS", "data": {"pass": 2}}
{"timestamp": {"seconds": 1543297427, "microseconds": 878857}, "event": "MIGRATION_PASS", "data": {"pass": 3}}
{"timestamp": {"seconds": 1543297427, "microseconds": 978783}, "event": "MIGRATION_PASS", "data": {"pass": 4}}
{"timestamp": {"seconds": 1543297427, "microseconds": 980312}, "event": "STOP"}
{"timestamp": {"seconds": 1543297428, "microseconds": 92269}, "event": "MIGRATION_PASS", "data": {"pass": 5}}
**********output of dst****************
(qemu) qemu-kvm: Failed to load virtio_pci/modern_queue_state:used
qemu-kvm: Failed to load virtio_pci/modern_state:vqs
qemu-kvm: Failed to load virtio/extra_state:extra_state
qemu-kvm: Failed to load virtio-blk:virtio
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:05.0/virtio-blk'
qemu-kvm: load of migration failed: Input/output error
Actual Result:
After migration, src core dump with info:
(qemu) qemu-kvm: block.c:855: bdrv_child_cb_inactivate: Assertion `bs->open_flags & 0x0800' failed.
src_core.txt: line 38: 22221 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idkp9HYI -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 -netdev tap,id=idDdrbRp,vhost=on -m 8192 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'IvyBridge',+kvm_pv_unhalt -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:0:3000,server,nowait -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/rhel76-64-virtio-scsi.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0
Test on qemu-kvm-2.12.0-43.module+el8+2255+c8d2cbae.x86_64, also hit this issue.
kernel version:4.18.0-44.el8.x86_64
Qemu cmds in source:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181128-054651-YPtCwugS,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181128-054651-YPtCwugS,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=id9FildC \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181128-054651-YPtCwugS,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20181128-054651-YPtCwugS,path=/var/tmp/seabios-20181128-054651-YPtCwugS,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20181128-054651-YPtCwugS,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
-blockdev driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device virtio-net-pci,mac=9a:6f:70:71:72:73,id=idJzalak,vectors=4,netdev=idIEbgxj,bus=pci.0,addr=0x4 \
-netdev tap,id=idIEbgxj,vhost=on \
-m 14336 \
-smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \
-cpu 'Broadwell',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :0 \
-rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=d,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
Core dump info in source:
(qemu) qemu-kvm: block.c:855: bdrv_child_cb_inactivate: Assertion `bs->open_flags & BDRV_O_INACTIVE' failed.
live_backup.txt: line 36: 7812 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181128-054651-YPtCwugS,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181128-054651-YPtCwugS,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=id9FildC -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181128-054651-YPtCwugS,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181128-054651-YPtCwugS,path=/var/tmp/seabios-20181128-054651-YPtCwugS,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181128-054651-YPtCwugS,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 -blockdev driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 -device scsi-hd,id=image1,drive=drive_image1 -device virtio-net-pci,mac=9a:6f:70:71:72:73,id=idJzalak,vectors=4,netdev=idIEbgxj,bus=pci.0,addr=0x4 -netdev tap,id=idIEbgxj,vhost=on -m 14336 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=d,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:0:3000,server,nowait -spice ipv4,port=7000,disable-ticketing
***********Dst**************
Qemu cmds in dst:
/usr/libexec/qemu-kvm \
-name 'avocado-vt-vm1' \
-sandbox off \
-machine pc \
-nodefaults \
-device VGA,bus=pci.0,addr=0x2 \
-chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181128-054651-YPtCwugT,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181128-054651-YPtCwugS,server,nowait \
-mon chardev=qmp_id_catch_monitor,mode=control \
-device pvpanic,ioport=0x505,id=id9FildC \
-chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181128-054651-YPtCwugS,server,nowait \
-device isa-serial,chardev=serial_id_serial0 \
-chardev socket,id=seabioslog_id_20181128-054651-YPtCwugS,path=/var/tmp/seabios-20181128-054651-YPtCwugS,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20181128-054651-YPtCwugS,iobase=0x402 \
-device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
-device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
-device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
-device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
-device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pci.0,addr=0x3 \
-blockdev driver=qcow2,node-name=drive_image1,cache.no-flush=on,cache.direct=off,file.driver=file,file.aio=threads,file.filename=/home/kvm_autotest_root/images/rhel80-64-virtio-scsi.qcow2 \
-device scsi-hd,id=image1,drive=drive_image1 \
-device virtio-net-pci,mac=9a:6f:70:71:72:73,id=idJzalak,vectors=4,netdev=idIEbgxj,bus=pci.0,addr=0x4 \
-netdev tap,id=idIEbgxj,vhost=on \
-m 14336 \
-smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \
-cpu 'Broadwell',+kvm_pv_unhalt \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
-vnc :1 \
-rtc base=utc,clock=host,driftfix=slew \
-boot order=cdn,once=d,menu=off,strict=off \
-enable-kvm \
-monitor stdio \
-qmp tcp:0:3001,server,nowait \
-incoming tcp:0:5000 \
-qmp tcp:0:3000,server,nowait \
Failure info in dst:
(qemu) qemu-kvm: Failed to load virtqueue_state:vring.avail
qemu-kvm: Failed to load virtio/virtqueues:vq
qemu-kvm: Failed to load virtio-net:virtio
qemu-kvm: error while loading state for instance 0x0 of device '0000:00:04.0/virtio-net'
qemu-kvm: load of migration failed: Input/output error
Verified on qemu-kvm-3.1.0-1.module+el8+2538+1516be75.x86_64, don't hit this issue any more, set bug's status to "Verified". Test Env: kernel version: 4.18.0-50.el8.x86_64 qemu-kvm version:qemu-kvm-3.1.0-1.module+el8+2538+1516be75.x86_64 Test steps: Same as that in bug description. Actual Result: Migration executed successfully after blockdev-add, and both src and dst can quit successfully. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1293 |
Description of problem: Qemu core dump after dirty bitmap migration with shared storage Version-Release number of selected component (if applicable): kernel version: 4.18.0-32.el8.x86_64 qemu-kvm version: qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64 How reproducible: 1/1 Steps to Reproduce: 1.Start src guest and set its migration capabilities: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idkp9HYI \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \ -netdev tap,id=idDdrbRp,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \ -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \ -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 \ -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \ qmp cmds: {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}} 2.Start dst guest with the same system image with src, set its migration capabilites: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9q,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idkp9HYI \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \ -netdev tap,id=idDdrbRp,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3001,server,nowait \ -incoming tcp:0:5000 \ -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \ -blockdev driver=raw,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.raw \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \ -blockdev driver=raw,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.raw \ -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \ qmp cmds: {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true}]}} 3. Create a full backup image and create a dirty bitmap {'execute':'blockdev-create','arguments':{'options': {'driver':'file','filename':'/root/sn1','size':32212254720},'job-id':'job1'}} {'execute':'blockdev-add','arguments':{'driver':'file','node-name':'drive_sn1','filename':'/root/sn1'}} {'execute':'blockdev-create','arguments':{'options': {'cluster-size':524288, 'driver': 'qcow2','file':'drive_sn1','size':32212254720},'job-id':'job2'}} {'execute':'blockdev-add','arguments':{'driver':'qcow2','node-name':'sn1','file':'drive_sn1'}} {'execute':'job-dismiss','arguments':{'id':'job1'}} {'execute':'job-dismiss','arguments':{'id':'job2'}} {"execute": "transaction", "arguments": { "actions": [ {"type": "block-dirty-bitmap-add", "data": {"node": "drive_image1", "name": "bitmap0"} }, {"type": "blockdev-backup", "data":{ "device": "drive_image1", "sync":"full","target":"sn1","job-id":"j1"}}]}} 4. Query bitmap info: {"execute":"query-block"} {"execute":"x-debug-block-dirty-bitmap-sha256","arguments": "node":"drive_image1","name":"bitmap0"}} 5. Create file in src guest, then check bitmap info: {"execute":"query-block"} {"execute":"x-debug-block-dirty-bitmap-sha256","arguments": "node":"drive_image1","name":"bitmap0"}} 6. Migrate from src to dst. {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.67:5000"}} ********output********** {"timestamp": {"seconds": 1542977753, "microseconds": 411111}, "event": "MIGRATION", "data": {"status": "setup"}} {"return": {}} {"timestamp": {"seconds": 1542977753, "microseconds": 568305}, "event": "MIGRATION_PASS", "data": {"pass": 1}} {"timestamp": {"seconds": 1542977753, "microseconds": 568471}, "event": "MIGRATION", "data": {"status": "active"}} {"timestamp": {"seconds": 1542977837, "microseconds": 831264}, "event": "MIGRATION_PASS", "data": {"pass": 2}} {"timestamp": {"seconds": 1542977843, "microseconds": 629177}, "event": "MIGRATION_PASS", "data": {"pass": 3}} {"timestamp": {"seconds": 1542977844, "microseconds": 328032}, "event": "MIGRATION_PASS", "data": {"pass": 4}} {"timestamp": {"seconds": 1542977844, "microseconds": 527092}, "event": "MIGRATION_PASS", "data": {"pass": 5}} {"timestamp": {"seconds": 1542977844, "microseconds": 529924}, "event": "STOP"} {"timestamp": {"seconds": 1542977844, "microseconds": 553455}, "event": "MIGRATION_PASS", "data": {"pass": 6}} Actual results: After the migration, src qemu core dump with info: (qemu) qemu-kvm: block.c:855: bdrv_child_cb_inactivate: Assertion `bs->open_flags & BDRV_O_INACTIVE' failed. mig_src.txt: line 38: 15197 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idkp9HYI -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 -netdev tap,id=idDdrbRp,vhost=on -m 14336 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:0:3000,server,nowait -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 Expected results: Src guest in paused status after migration and no core dump. Additional info: Will attach core_dump file link later.