Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem: Qemu core dump when quit vm after dirty bitmap migration failed Version-Release number of selected component (if applicable): kernel version: 4.18.0-32.el8.x86_64 qemu-kvm version: qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64 How reproducible: 1/1 Steps to Reproduce: 1.Start guest in src and set migration capabilities: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idkp9HYI \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \ -netdev tap,id=idDdrbRp,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3000,server,nowait \ -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \ -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \ -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 \ -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \ qmp cmds: {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true},{"capability":"pause-before-switchover","state":true}]}} 2. Start guest in dst with -incoming param, set migration capabilities and export the empty disk: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9q,server,nowait \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idkp9HYI \ -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \ -device isa-serial,chardev=serial_id_serial0 \ -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \ -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 \ -netdev tap,id=idDdrbRp,vhost=on \ -m 14336 \ -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :1 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:0:3001,server,nowait \ -incoming tcp:0:5000 \ -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/migrate.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \ -blockdev driver=raw,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.raw \ -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \ -blockdev driver=raw,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.raw \ -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \ qmp cmds： {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true},{"capability":"pause-before-switchover","state":true}]}} { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host":"10.73.196.67", "port": "3333" } } } } { "execute": "nbd-server-add", "arguments":{ "device": "drive_image1", "writable": true } } 3.In src, stop vm and create a dirty bitmap, then check bitmap info (qemu) stop {"execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}} {"execute":"query-block"} {"execute": "x-debug-block-dirty-bitmap-sha256","arguments":{"node":"drive_image1","name":"bitmap0"}} 4. Do block-mirror from src to dst: {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror0","server":{"type":"inet","host":"10.73.196.67","port":"3333"},"export":"drive_image1"}} {"execute": "blockdev-mirror", "arguments": { "device":"drive_image1","target": "mirror0", "sync": "full", "job-id":"j1"}} 5. After mirror job is ready, migrate from source to dst. {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.67:5000"}} ********output********** {"timestamp": {"seconds": 1542964601, "microseconds": 832614}, "event": "MIGRATION", "data": {"status": "setup"}} {"return": {}} {"timestamp": {"seconds": 1542964601, "microseconds": 989813}, "event": "MIGRATION_PASS", "data": {"pass": 1}} {"timestamp": {"seconds": 1542964601, "microseconds": 990019}, "event": "MIGRATION", "data": {"status": "active"}} {"timestamp": {"seconds": 1542964601, "microseconds": 990092}, "event": "MIGRATION", "data": {"status": "failed"}} {"timestamp": {"seconds": 1542964997, "microseconds": 553155}, "event": "SHUTDOWN", "data": {"guest": false}} {"timestamp": {"seconds": 1542964997, "microseconds": 553308}, "event": "BLOCK_JOB_ERROR", "data": {"device": "j1", "operation": "write", "action": "report"}} {"timestamp": {"seconds": 1542964997, "microseconds": 553714}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "j1"}} {"timestamp": {"seconds": 1542964997, "microseconds": 554021}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "j1", "len": 32212254720, "offset": 32212254720, "speed": 0, "type": "mirror"}} {"timestamp": {"seconds": 1542964997, "microseconds": 554111}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}} {"timestamp": {"seconds": 1542964997, "microseconds": 554171}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}} 6. Quit vm (qemu)quit Actual results: After step6, qemu core dump with info: qemu-kvm: block.c:3415: bdrv_close_all: Assertion `QTAILQ_EMPTY(&all_bdrv_states)' failed. mig_src.txt: line 38: 1721 Aborted (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idkp9HYI -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 -netdev tap,id=idDdrbRp,vhost=on -m 14336 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:0:3000,server,nowait -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 Expected results: Qemu quit successfully. Additional info: [root@ibm-x3650m5-07 coredump]# gdb core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 GNU gdb (GDB) Red Hat Enterprise Linux 8.2-3.el8 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... BFD: warning: /var/lib/systemd/coredump/core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 is truncated: expected core file size >= 15369699328, found: 2147483648 BFD: warning: /var/lib/systemd/coredump/core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 is truncated: expected core file size >= 15369699328, found: 2147483648 [New LWP 1721] [New LWP 1780] [New LWP 1792] [New LWP 1781] [New LWP 1786] [New LWP 1788] [New LWP 1785] [New LWP 1779] [New LWP 1777] [New LWP 1790] [New LWP 1722] [New LWP 1784] [New LWP 1787] [New LWP 1789] Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64.debug...done. done. Cannot access memory at address 0x7f3748f6f128 Cannot access memory at address 0x7f3748f6f120 Failed to read a valid object file image from memory. Core was generated by `/usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f3743dc793f in ?? () [Current thread is 1 (LWP 1721)] (gdb) bt #0 0x00007f3743dc793f in ?? () Backtrace stopped: Cannot access memory at address 0x7ffc325bb030 Attachment is the core dump file.