RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1652873 - Src qemu core dump when quit vm after dirty bitmap migration failed
Summary: Src qemu core dump when quit vm after dirty bitmap migration failed
Keywords:
Status: CLOSED DUPLICATE of bug 1652424
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
unspecified
Target Milestone: rc
: 8.0
Assignee: Virtualization Maintenance
QA Contact: aihua liang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-23 11:22 UTC by aihua liang
Modified: 2019-05-21 09:14 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-26 01:39:33 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description aihua liang 2018-11-23 11:22:07 UTC
Description of problem:
 Qemu core dump when quit vm after dirty bitmap migration failed

Version-Release number of selected component (if applicable):
 kernel version: 4.18.0-32.el8.x86_64
 qemu-kvm version: qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64

How reproducible:
 1/1 

Steps to Reproduce:
1.Start guest in src and set migration capabilities:
   /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idkp9HYI  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7  \
    -netdev tap,id=idDdrbRp,vhost=on \
    -m 14336  \
    -smp 10,maxcpus=10,cores=5,threads=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :0  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3000,server,nowait \
    -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \
    -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
    -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 \
    -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \

qmp cmds:
    {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true},{"capability":"pause-before-switchover","state":true}]}}



2. Start guest in dst with -incoming param, set migration capabilities and export the empty disk:
    /usr/libexec/qemu-kvm \
    -name 'avocado-vt-vm1'  \
    -sandbox off  \
    -machine pc  \
    -nodefaults \
    -device VGA,bus=pci.0,addr=0x2  \
    -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9q,server,nowait \
    -mon chardev=qmp_id_qmpmonitor1,mode=control  \
    -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idkp9HYI  \
    -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait \
    -device isa-serial,chardev=serial_id_serial0  \
    -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 \
    -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \
    -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \
    -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \
    -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \
    -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7  \
    -netdev tap,id=idDdrbRp,vhost=on \
    -m 14336  \
    -smp 10,maxcpus=10,cores=5,threads=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1  \
    -vnc :1  \
    -rtc base=localtime,clock=host,driftfix=slew  \
    -boot order=cdn,once=c,menu=off,strict=off \
    -enable-kvm \
    -monitor stdio \
    -qmp tcp:0:3001,server,nowait \
    -incoming tcp:0:5000 \
    -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/migrate.qcow2 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 \
    -blockdev driver=raw,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.raw \
    -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 \
    -blockdev driver=raw,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.raw \
    -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0 \

qmp cmds:
    {"execute":"migrate-set-capabilities","arguments":{"capabilities":[{"capability":"events","state":true},{"capability":"dirty-bitmaps","state":true},{"capability":"pause-before-switchover","state":true}]}}
    { "execute": "nbd-server-start", "arguments": { "addr": { "type": "inet","data": { "host":"10.73.196.67", "port": "3333" } } } }
    { "execute": "nbd-server-add", "arguments":{ "device": "drive_image1", "writable": true } }

3.In src, stop vm and create a dirty bitmap, then check bitmap info
   (qemu) stop
   {"execute": "block-dirty-bitmap-add", "arguments": {"node": "drive_image1", "name": "bitmap0"}}
   {"execute":"query-block"}
   {"execute": "x-debug-block-dirty-bitmap-sha256","arguments":{"node":"drive_image1","name":"bitmap0"}}

4. Do block-mirror from src to dst:
    {"execute":"blockdev-add","arguments":{"driver":"nbd","node-name":"mirror0","server":{"type":"inet","host":"10.73.196.67","port":"3333"},"export":"drive_image1"}}
    {"execute": "blockdev-mirror", "arguments": { "device":"drive_image1","target": "mirror0", "sync": "full", "job-id":"j1"}}

5. After mirror job is ready, migrate from source to dst.
    {"execute": "migrate","arguments":{"uri": "tcp:10.73.196.67:5000"}}
    
 ********output**********
    {"timestamp": {"seconds": 1542964601, "microseconds": 832614}, "event": "MIGRATION", "data": {"status": "setup"}}
{"return": {}}
{"timestamp": {"seconds": 1542964601, "microseconds": 989813}, "event": "MIGRATION_PASS", "data": {"pass": 1}}
{"timestamp": {"seconds": 1542964601, "microseconds": 990019}, "event": "MIGRATION", "data": {"status": "active"}}
{"timestamp": {"seconds": 1542964601, "microseconds": 990092}, "event": "MIGRATION", "data": {"status": "failed"}}
{"timestamp": {"seconds": 1542964997, "microseconds": 553155}, "event": "SHUTDOWN", "data": {"guest": false}}
{"timestamp": {"seconds": 1542964997, "microseconds": 553308}, "event": "BLOCK_JOB_ERROR", "data": {"device": "j1", "operation": "write", "action": "report"}}
{"timestamp": {"seconds": 1542964997, "microseconds": 553714}, "event": "JOB_STATUS_CHANGE", "data": {"status": "aborting", "id": "j1"}}
{"timestamp": {"seconds": 1542964997, "microseconds": 554021}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "j1", "len": 32212254720, "offset": 32212254720, "speed": 0, "type": "mirror"}}
{"timestamp": {"seconds": 1542964997, "microseconds": 554111}, "event": "JOB_STATUS_CHANGE", "data": {"status": "concluded", "id": "j1"}}
{"timestamp": {"seconds": 1542964997, "microseconds": 554171}, "event": "JOB_STATUS_CHANGE", "data": {"status": "null", "id": "j1"}}

6. Quit vm
 (qemu)quit

Actual results:
After step6, qemu core dump with info:
  qemu-kvm: block.c:3415: bdrv_close_all: Assertion `QTAILQ_EMPTY(&all_bdrv_states)' failed.
mig_src.txt: line 38:  1721 Aborted                 (core dumped) /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -device VGA,bus=pci.0,addr=0x2 -chardev socket,id=qmp_id_qmpmonitor1,path=/var/tmp/monitor-qmpmonitor1-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_qmpmonitor1,mode=control -chardev socket,id=qmp_id_catch_monitor,path=/var/tmp/monitor-catch_monitor-20181107-005924-PkIxnG9p,server,nowait -mon chardev=qmp_id_catch_monitor,mode=control -device pvpanic,ioport=0x505,id=idkp9HYI -chardev socket,id=serial_id_serial0,path=/var/tmp/serial-serial0-20181107-005924-PkIxnG9p,server,nowait -device isa-serial,chardev=serial_id_serial0 -chardev socket,id=seabioslog_id_20181107-005924-PkIxnG9p,path=/var/tmp/seabios-20181107-005924-PkIxnG9p,server,nowait -device isa-debugcon,chardev=seabioslog_id_20181107-005924-PkIxnG9p,iobase=0x402 -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -device virtio-net-pci,mac=9a:44:45:46:47:48,id=iddDGLIi,vectors=4,netdev=idDdrbRp,bus=pci.0,addr=0x7 -netdev tap,id=idDdrbRp,vhost=on -m 14336 -smp 10,maxcpus=10,cores=5,threads=1,sockets=2 -cpu 'Broadwell',+kvm_pv_unhalt,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:0:3000,server,nowait -blockdev driver=qcow2,node-name=drive_image1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/kvm_autotest_root/images/win10-32-virtio.qcow2.bak -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data1,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data1.qcow2 -device virtio-blk-pci,id=data1,drive=drive_data1,bus=pci.0 -blockdev driver=qcow2,node-name=drive_data2,file.aio=threads,cache.no-flush=off,cache.direct=on,file.driver=file,file.filename=/home/data2.qcow2 -device virtio-blk-pci,id=data2,drive=drive_data2,bus=pci.0

Expected results:
  Qemu quit successfully.

Additional info:
[root@ibm-x3650m5-07 coredump]# gdb core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-3.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
BFD: warning: /var/lib/systemd/coredump/core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 is truncated: expected core file size >= 15369699328, found: 2147483648
BFD: warning: /var/lib/systemd/coredump/core.qemu-kvm.0.5d4dd52e7ce24262bea79bdbfd44ae76.1721.1542964997000000 is truncated: expected core file size >= 15369699328, found: 2147483648
[New LWP 1721]
[New LWP 1780]
[New LWP 1792]
[New LWP 1781]
[New LWP 1786]
[New LWP 1788]
[New LWP 1785]
[New LWP 1779]
[New LWP 1777]
[New LWP 1790]
[New LWP 1722]
[New LWP 1784]
[New LWP 1787]
[New LWP 1789]
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm-2.12.0-42.module+el8+2173+537e5cb5.x86_64.debug...done.
done.
Cannot access memory at address 0x7f3748f6f128
Cannot access memory at address 0x7f3748f6f120
Failed to read a valid object file image from memory.
Core was generated by `/usr/libexec/qemu-kvm -name avocado-vt-vm1 -sandbox off -machine pc -nodefaults'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f3743dc793f in ?? ()
[Current thread is 1 (LWP 1721)]
(gdb) bt
#0  0x00007f3743dc793f in ?? ()
Backtrace stopped: Cannot access memory at address 0x7ffc325bb030


Attachment is the core dump file.

Comment 2 Gu Nini 2018-11-26 01:39:33 UTC

*** This bug has been marked as a duplicate of bug 1652424 ***

Comment 3 aihua liang 2018-11-29 07:35:23 UTC
Also hit this issue on qemu-kvm-2.12.0-43.module+el8+2255+c8d2cbae.x86_64.


Note You need to log in before you can comment on or make changes to this bug.