Bug 1268038 - Observing QEMU monitor events via `virsh` results in "double free or corruption (fasttop): 0x0000556923c72ed0"
Observing QEMU monitor events via `virsh` results in "double free or corrupti...
Status: CLOSED NOTABUG
Product: Virtualization Tools
Classification: Community
Component: libvirt (Show other bugs)
unspecified
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Libvirt Maintainers
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-01 12:15 EDT by Kashyap Chamarthy
Modified: 2016-04-11 01:59 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-04-11 01:59:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Kashyap Chamarthy 2015-10-01 12:15:55 EDT
Reproducer
==========

On shell #1: Observe QEMU monitor events via `virsh`:

    $ virsh -c qemu:///system qemu-monitor-event \
        --event block --regex --no-case


On shell #1: Perform  live disk copy:

    $ virsh dumpxml --inactive cvm1 > /var/tmp/cvm1.xml
    $ virsh undefine cirrvm
    $ virsh blockcopy --domain cvm1 vda
        /var/lib/libvirt/images/copy2-cvm1.qcow2 \
        --wait --verbose --finish
    Block Copy: [100 %]
    Successfully copied



Actual Result
=============

--------------------------
Observe this on shell #2 
$ virsh -c qemu:///system qemu-monitor-event \
    --event block --regex --no-case
event BLOCK_JOB_READY at 1443714804.600609 for domain cvm1: {"device":"drive-virtio-disk0","len":41126912,"offset":41126912,"speed":0,"type":"mirror"}
events received: 1

error: One or more references were leaked after disconnect from the hypervisor
*** Error in `virsh': double free or corruption (fasttop): 0x0000556923c72ed0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x77a8d)[0x7f7b0269ba8d]
/lib64/libc.so.6(cfree+0x5cd)[0x7f7b026a7d2d]
/lib64/libvirt.so.0(virFree+0x1b)[0x7f7b068a92bb]
/lib64/libvirt.so.0(+0x11021e)[0x7f7b0694d21e]
/lib64/libvirt.so.0(virObjectEventStateFree+0x5f)[0x7f7b0694bb1f]
/lib64/libvirt.so.0(+0x1b22fc)[0x7f7b069ef2fc]
/lib64/libvirt.so.0(+0x14e515)[0x7f7b0698b515]
/lib64/libvirt.so.0(virObjectUnref+0xfb)[0x7f7b068ec61b]
/lib64/libvirt.so.0(+0x10ea26)[0x7f7b0694ba26]
/lib64/libvirt.so.0(virEventPollRunOnce+0x592)[0x7f7b068c65c2]
/lib64/libvirt.so.0(virEventRunDefaultImpl+0x41)[0x7f7b068c4f61]
virsh(+0x257b5)[0x556922e9d7b5]
/lib64/libvirt.so.0(+0xc5f0e)[0x7f7b06902f0e]
/lib64/libpthread.so.0(+0x7555)[0x7f7b029eb555]
/lib64/libc.so.6(clone+0x6d)[0x7f7b02726b9d]
$ virsh -c qemu:///system qemu-monitor-event --event block --regex --no-case
event BLOCK_JOB_READY at 1443714804.600609 for domain cvm1: {"device":"drive-virtio-disk0","len":41126912,"offset":41126912,"speed":0,"type":"mirror"}
events received: 1

error: One or more references were leaked after disconnect from the hypervisor
*** Error in `virsh': double free or corruption (fasttop): 0x0000556923c72ed0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x77a8d)[0x7f7b0269ba8d]
/lib64/libc.so.6(cfree+0x5cd)[0x7f7b026a7d2d]
/lib64/libvirt.so.0(virFree+0x1b)[0x7f7b068a92bb]
/lib64/libvirt.so.0(+0x11021e)[0x7f7b0694d21e]
/lib64/libvirt.so.0(virObjectEventStateFree+0x5f)[0x7f7b0694bb1f]
/lib64/libvirt.so.0(+0x1b22fc)[0x7f7b069ef2fc]
/lib64/libvirt.so.0(+0x14e515)[0x7f7b0698b515]
/lib64/libvirt.so.0(virObjectUnref+0xfb)[0x7f7b068ec61b]
/lib64/libvirt.so.0(+0x10ea26)[0x7f7b0694ba26]
/lib64/libvirt.so.0(virEventPollRunOnce+0x592)[0x7f7b068c65c2]
/lib64/libvirt.so.0(virEventRunDefaultImpl+0x41)[0x7f7b068c4f61]
virsh(+0x257b5)[0x556922e9d7b5]
/lib64/libvirt.so.0(+0xc5f0e)[0x7f7b06902f0e]
/lib64/libpthread.so.0(+0x7555)[0x7f7b029eb555]
/lib64/libc.so.6(clone+0x6d)[0x7f7b02726b9d]
======= Memory map: ========
556922e78000-556922eed000 r-xp 00000000 fc:03 10281402                   /usr/bin/virsh
5569230ec000-5569230ff000 r--p 00074000 fc:03 10281402                   /usr/bin/virsh
5569230ff000-556923100000 rw-p 00087000 fc:03 10281402                   /usr/bin/virsh
556923c4a000-556923c88000 rw-p 00000000 00:00 0                          [heap]
7f7af0000000-7f7af0021000 rw-p 00000000 00:00 0 
7f7af0021000-7f7af4000000 ---p 00000000 00:00 0 
7f7af4c7a000-7f7af4c7b000 ---p 00000000 00:00 0 

7f7af4c7b000-7f7af547b000 rw-p 00000000 00:00 0                          [stack:5686]
7f7af547b000-7f7afbb35000 r--p 00000000 fc:03 17320577                   /usr/lib/locale/locale-archive
7f7afbb35000-7f7afbb38000 r-xp 00000000 fc:03 16900621                   /usr/lib64/libkeyutils.so.1.5
7f7afbb38000-7f7afbd37000 ---p 00003000 fc:03 16900621                   /usr/lib64/libkeyutils.so.1.5
7f7afbd37000-7f7afbd38000 r--p 00002000 fc:03 16900621                   /usr/lib64/libkeyutils.so.1.5
7f7afbd38000-7f7afbd39000 rw-p 00003000 fc:03 16900621                   /usr/lib64/libkeyutils.so.1.5
7f7afbd39000-7f7afbd46000 r-xp 00000000 fc:03 17329406                   /usr/lib64/libkrb5support.so.0.1
7f7afbd46000-7f7afbf46000 ---p 0000d000 fc:03 17329406                   /usr/lib64/libkrb5support.so.0.1
7f7afbf46000-7f7afbf47000 r--p 0000d000 fc:03 17329406                   /usr/lib64/libkrb5support.so.0.1
7f7afbf47000-7f7afbf48000 rw-p 0000e000 fc:03 17329406                   /usr/lib64/libkrb5support.so.0.1
7f7afbf48000-7f7afbfbf000 r-xp 00000000 fc:03 16875407                   /usr/lib64/libfreebl3.so
7f7afbfbf000-7f7afc1be000 ---p 00077000 fc:03 16875407                   /usr/lib64/libfreebl3.so
7f7afc1be000-7f7afc1c0000 r--p 00076000 fc:03 16875407                   /usr/lib64/libfreebl3.so
7f7afc1c0000-7f7afc1c1000 rw-p 00078000 fc:03 16875407                   /usr/lib64/libfreebl3.so
7f7afc1c1000-7f7afc1c5000 rw-p 00000000 00:00 0 
7f7afc1c5000-7f7afc1cc000 r-xp 00000000 fc:03 16900199                   /usr/lib64/libffi.so.6.0.2
7f7afc1cc000-7f7afc3cb000 ---p 00007000 fc:03 16900199                   /usr/lib64/libffi.so.6.0.2
7f7afc3cb000-7f7afc3cc000 r--p 00006000 fc:03 16900199                   /usr/lib64/libffi.so.6.0.2
7f7afc3cc000-7f7afc3cd000 rw-p 00007000 fc:03 16900199                   /usr/lib64/libffi.so.6.0.2
7f7afc3cd000-7f7afc3dc000 r-xp 00000000 fc:03 16899846                   /usr/lib64/libbz2.so.1.0.6
7f7afc3dc000-7f7afc5db000 ---p 0000f000 fc:03 16899846                   /usr/lib64/libbz2.so.1.0.6
7f7afc5db000-7f7afc5dc000 r--p 0000e000 fc:03 16899846                   /usr/lib64/libbz2.so.1.0.6
7f7afc5dc000-7f7afc5dd000 rw-p 0000f000 fc:03 16899846                   /usr/lib64/libbz2.so.1.0.6
7f7afc5dd000-7f7afc5f2000 r-xp 00000000 fc:03 16899845                   /usr/lib64/libelf-0.163.so
7f7afc5f2000-7f7afc7f1000 ---p 00015000 fc:03 16899845                   /usr/lib64/libelf-0.163.so
7f7afc7f1000-7f7afc7f2000 r--p 00014000 fc:03 16899845                   /usr/lib64/libelf-0.163.so
7f7afc7f2000-7f7afc7f3000 rw-p 00015000 fc:03 16899845                   /usr/lib64/libelf-0.163.so
7f7afc7f3000-7f7afc7f7000 r-xp 00000000 fc:03 16900161                   /usr/lib64/libattr.so.1.1.0
7f7afc7f7000-7f7afc9f6000 ---p 00004000 fc:03 16900161                   /usr/lib64/libattr.so.1.1.0
7f7afc9f6000-7f7afc9f7000 r--p 00003000 fc:03 16900161                   /usr/lib64/libattr.so.1.1.0
7f7afc9f7000-7f7afc9f8000 rw-p 00004000 fc:03 16900161                   /usr/lib64/libattr.so.1.1.0
7f7afc9f8000-7f7afca1e000 r-xp 00000000 fc:03 17329380                   /usr/lib64/libtinfo.so.5.9
7f7afca1e000-7f7afcc1d000 ---p 00026000 fc:03 17329380                   /usr/lib64/libtinfo.so.5.9
7f7afcc1d000-7f7afcc21000 r--p 00025000 fc:03 17329380                   /usr/lib64/libtinfo.so.5.9
7f7afcc21000-7f7afcc22000 rw-p 00029000 fc:03 17329380                   /usr/lib64/libtinfo.so.5.9
7f7afcc22000-7f7afcc26000 r-xp 00000000 fc:03 16899906                   /usr/lib64/libuuid.so.1.3.0
7f7afcc26000-7f7afce25000 ---p 00004000 fc:03 16899906                   /usr/lib64/libuuid.so.1.3.0
7f7afce25000-7f7afce26000 r--p 00003000 fc:03 16899906                   /usr/lib64/libuuid.so.1.3.0
7f7afce26000-7f7afce27000 rw-p 00004000 fc:03 16899906                   /usr/lib64/libuuid.so.1.3.0
7f7afce27000-7f7afce2e000 r-xp 00000000 fc:03 19876886                   /usr/lib64/libblktapctl.so.1.0.0
7f7afce2e000-7f7afd02d000 ---p 00007000 fc:03 19876886                   /usr/lib64/libblktapctl.so.1.0.0
7f7afd02d000-7f7afd02e000 r--p 00006000 fc:03 19876886                   /usr/lib64/libblktapctl.so.1.0.0
7f7afd02e000-7f7afd02f000 rw-p 00007000 fc:03 19876886                   /usr/lib64/libblktapctl.so.1.0.0
7f7b07004000-7f7b0700c000 rw-p 00000000 00:00 0 
7f7b0700c000-7f7b07034000 r-xp 00000000 fc:03 16902175                   /usr/lib64/libsystemd.so.0.6.0
7f7b07034000-7f7b07035000 ---p 00028000 fc:03 16902175                   /usr/lib64/libsystemd.so.0.6.0
7f7b07035000-7f7b07036000 r--p 00028000 fc:03 16902175                   /usr/lib64/libsystemd.so.0.6.0
7f7b07036000-7f7b07037000 rw-p 00029000 fc:03 16902175                   /usr/lib64/libsystemd.so.0.6.0
7f7b07037000-7f7b0703a000 rw-p 00000000 00:00 0 
7f7b0703c000-7f7b0703e000 rw-p 00000000 00:00 0 
7f7b0703e000-7f7b07045000 r--s 00000000 fc:03 17329342                   /usr/lib64/gconv/gconv-modules.cache
7f7b07045000-7f7b07046000 rw-p 00000000 00:00 0 
7f7b07046000-7f7b07047000 r--p 00020000 fc:03 17320576                   /usr/lib64/ld-2.21.so
7f7b07047000-7f7b07048000 rw-p 00021000 fc:03 17320576                   /usr/lib64/ld-2.21.so
7f7b07048000-7f7b07049000 rw-p 00000000 00:00 0 
7fff8bf7c000-7fff8bf9d000 rw-p 00000000 00:00 0                          [stack]
7fff8bfd0000-7fff8bfd2000 r--p 00000000 00:00 0                          [vvar]
7fff8bfd2000-7fff8bfd4000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
--------------------------


Expected Result
===============

`virsh` should not error out this way, but exit gracefully.


Version:
=======

libvirt-daemon-kvm-1.2.13.1-2.fc22.x86_64
qemu-system-x86-2.3.1-7.fc22.x86_64

*NOTE*: The QEMU build is regular F22, *plus* the below upstream patch

    http://git.qemu.org/?p=qemu.git;a=commit;h=e424aff 
    -- mirror: Fix coroutine reentrance


QEMU scratch (note: these builds will get auto-deleted in 13 more days) build for reference:

    http://koji.fedoraproject.org/koji/taskinfo?taskID=11265946
Comment 1 Kashyap Chamarthy 2015-10-01 12:17:42 EDT
I should note that I noticed this behavior only when I do the blockcopy operation multiple times in successsion.
Comment 2 Cole Robinson 2016-04-10 18:28:47 EDT
I don't really have a setup to test this. Kashyap do you know if this is still relevant on f23 or f24?
Comment 3 Kashyap Chamarthy 2016-04-11 01:59:27 EDT
Just tested on Fedora-23, and I can't reproduce the "double free or corruption" any more.

I tested with these versions:

    qemu-system-x86-2.5.0-10.fc23.x86_64
    libvirt-daemon-kvm-1.3.2-3.fc23.x86_64
    4.4.6-300.fc23.x86_64

I did the 'blockcopy' on one shell:

$ virsh blockcopy --domain cvm1 vda \
    /var/lib/libvirt/images/copy2-cvm1.qcow2 \
    --wait --verbose --finish

While monitoring events on a different shell:

$ virsh -c qemu:///system qemu-monitor-event --event block --regex --no-case
event BLOCK_JOB_READY at 1460354010.949239 for domain cvm1: {"device":"drive-virtio-disk0","len":19267584,"offset":19267584,"speed":0,"type":"mirror"}
events received: 1

- - -

So, closing this as NOTABUG.

[/me wishes there was an option to close a bug as: "CLOSED; NOT-REPRODUCIBLE/INTERMITTENT"]

Note You need to log in before you can comment on or make changes to this bug.