Bug 1652873
| Summary: | Src qemu core dump when quit vm after dirty bitmap migration failed | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | aihua liang <aliang> |
| Component: | qemu-kvm | Assignee: | Virtualization Maintenance <virt-maint> |
| Status: | CLOSED DUPLICATE | QA Contact: | aihua liang <aliang> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | high | ||
| Version: | 8.0 | CC: | chayang, coli, juzhang, michen, ngu, qzhang, rbalakri, virt-maint |
| Target Milestone: | rc | ||
| Target Release: | 8.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-11-26 01:39:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1652424 *** Also hit this issue on qemu-kvm-2.12.0-43.module+el8+2255+c8d2cbae.x86_64. |
Description of problem: Qemu core dump when quit vm after dirty bitmap migration failed Version-Release number of selected component (if applicable): kernel version: 4.18.0-32.el8.x86_64 qemu-kvm version: qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64 How reproducible: 1/1 Steps to Reproduce: 1.Start guest in src and set migration capabilities: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idkp9HYI \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \ -netdev tap,id=idDdrbRp,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \ -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \ -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 \ -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \ qmp cmds: {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true},{"capability":"pause-before-switchover","state":true}]}} 2. Start guest in dst with -incoming param, set migration capabilities and export the empty disk: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9q,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idkp9HYI \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \ -netdev tap,id=idDdrbRp,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3001,server,nowait \ -incoming tcp:0:5000 \ -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/migrate.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \ -blockdev driver=raw,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.raw \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \ -blockdev driver=raw,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.raw \ -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \ qmp cmds： {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true},{"capability":"pause-before-switchover","state":true}]}} { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host":"10.73.196.67", "port": "3333" } } } } { "execute": "nbd-server-add", "arguments":{ "device": "drive_image1", "writable": true } } 3.In src, stop vm and create a dirty bitmap, then check bitmap info (qemu) stop {"execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}} {"execute":"query-block"} {"execute": "x-debug-block-dirty-bitmap-sha256","arguments":{"node":"drive_image1","name":"bitmap0"}} 4. Do block-mirror from src to dst: {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror0","server":{"type":"inet","host":"10.73.196.67","port":"3333"},"export":"drive_image1"}} {"execute": "blockdev-mirror", "arguments": { "device":"drive_image1","target": "mirror0", "sync": "full", "job-id":"j1"}} 5. After mirror job is ready, migrate from source to dst. {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.67:5000"}} ********output********** {"timestamp": {"seconds": 1542964601, "microseconds": 832614}, "event": "MIGRATION", "data": {"status": "setup"}} {"return": {}} {"timestamp": {"seconds": 1542964601, "microseconds": 989813}, "event": "MIGRATION_PASS", "data": {"pass": 1}} {"timestamp": {"seconds": 1542964601, "microseconds": 990019}, "event": "MIGRATION", "data": {"status": "active"}} {"timestamp": {"seconds": 1542964601, "microseconds": 990092}, "event": "MIGRATION", "data": {"status": "failed"}} {"timestamp": {"seconds": 1542964997, "microseconds": 553155}, "event": "SHUTDOWN", "data": {"guest": false}} {"timestamp": {"seconds": 1542964997, "microseconds": 553308}, "event": "BLOCK_JOB_ERROR", "data": {"device": "j1", "operation": "write", "action": "report"}} {"timestamp": {"seconds": 1542964997, "microseconds": 553714}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "j1"}} {"timestamp": {"seconds": 1542964997, "microseconds": 554021}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "j1", "len": 32212254720, "offset": 32212254720, "speed": 0, "type": "mirror"}} {"timestamp": {"seconds": 1542964997, "microseconds": 554111}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}} {"timestamp": {"seconds": 1542964997, "microseconds": 554171}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}} 6. Quit vm (qemu)quit Actual results: After step6, qemu core dump with info: qemu-kvm: block.c:3415: bdrv_close_all: Assertion `QTAILQ_EMPTY(&all_bdrv_states)' failed. mig_src.txt: line 38: 1721 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idkp9HYI -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 -netdev tap,id=idDdrbRp,vhost=on -m 14336 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:0:3000,server,nowait -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 Expected results: Qemu quit successfully. Additional info: [root@ibm-x3650m5-07 coredump]# gdb core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-3.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... BFD: warning: /var/lib/systemd/coredump/core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 is truncated: expected core file size >= 15369699328, found: 2147483648 BFD: warning: /var/lib/systemd/coredump/core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 is truncated: expected core file size >= 15369699328, found: 2147483648 [New LWP 1721] [New LWP 1780] [New LWP 1792] [New LWP 1781] [New LWP 1786] [New LWP 1788] [New LWP 1785] [New LWP 1779] [New LWP 1777] [New LWP 1790] [New LWP 1722] [New LWP 1784] [New LWP 1787] [New LWP 1789] Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64.debug...done. done. Cannot access memory at address 0x7f3748f6f128 Cannot access memory at address 0x7f3748f6f120 Failed to read a valid object file image from memory. Core was generated by `/usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f3743dc793f in ?? () [Current thread is 1 (LWP 1721)] (gdb) bt #0 0x00007f3743dc793f in ?? () Backtrace stopped: Cannot access memory at address 0x7ffc325bb030 Attachment is the core dump file.