Bug 1445174
| Summary: | [RHEV7.4] [guest memory dump]dump-guest-memory QMP command with "detach" param makes qemu-kvm process aborted | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | hachen <hachen> |
| Component: | qemu-kvm-rhev | Assignee: | Fam Zheng <famz> |
| Status: | CLOSED ERRATA | QA Contact: | hachen <hachen> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.4 | CC: | aliang, chayang, coli, hachen, juzhang, knoel, marcandre.lureau, michen, mrezanin, ngu, pbonzini, pingl, shuang, virt-maint, xfu, xuhan, xutian, xuwei |
| Target Milestone: | rc | Keywords: | Regression |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-rhev-2.9.0-5.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-02 04:35:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Host:
qemu-kvm-rhev-2.8.0-5.el7.x86_64
kernel-3.10.0-566.el7.x86_64
kernel-debuginfo-common-x86_64-3.10.0-566.el7.x86_64
kernel-debuginfo-3.10.0-566.el7.x86_64
I have also tried on rhel7.3 host with qemu 2.8,for step 2:
QMP:
{ "execute": "qmp_capabilities" }
{"return": {}}
{"execute": "dump-guest-memory", "arguments": { "detach": true, "paging": false, "protocol": "file:/home/dump.normal"}}
{"timestamp": {"seconds": 1493102889, "microseconds": 979138}, "event": "STOP"}
{"return": {}}
{"timestamp": {"seconds": 1493102893, "microseconds": 330485}, "event": "DUMP_COMPLETED", "data": {"result": {"total": 4301455360, "status": "completed", "completed": 4301455360}}}
{"timestamp": {"seconds": 1493102893, "microseconds": 330953}, "event": "RESUME"}
{"execute": "query-dump"}
{"return": {"total": 4301455360, "status": "completed", "completed": 4301455360}}
HMP:
works fine
I'm having difficulties reproducing locally. Can you provide a stack backtrace? hachen helped me to reproduce on a lab machine.
Start qemu-kvm under gdb as follows:
# gdb --args `sed '/Westmere/d' <dump.sh | tr -d '\\\\'`
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-98.el7
[...]
Reading symbols from /usr/libexec/qemu-kvm...Reading symbols from /usr/lib/debug/usr/libexec/qemu-kvm.debug...done.
done.
(gdb) r
Starting program: /usr/libexec/qemu-kvm -name \'avocado-vt-vm1\' -sandbox off -machine pc -nodefaults -vga cirrus -device pvpanic,ioport=0x505,id=idHT1RPm -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x3 -device virtio-net-pci,mac=9a:1b:1c:1d:1e:1f,id=id8xeo6O,vectors=4,netdev=idBP1nUD,bus=pci.0,addr=0x4 -netdev tap,id=idBP1nUD -m 8192 -smp 4,cores=2,threads=1,sockets=2 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:localhost:4444,server,nowait
QMP conversation in a second terminal:
# telnet localhost 4444
Trying ::1...
Connected to localhost.
Escape character is '^]'.
{"QMP": {"version": {"qemu": {"micro": 0, "minor": 9, "major": 2}, "package": "(qemu-kvm-rhev-2.9.0-1.el7)"}, "capabilities": []}}
{ "execute": "qmp_capabilities" }
{"return": {}}
{"execute": "dump-guest-memory", "arguments": { "detach": true, "paging": false, "protocol": "file:/home/dump.normal"}}
{"timestamp": {"seconds": 1493713144, "microseconds": 617728}, "event": "STOP"}
{"return": {}}
{"timestamp": {"seconds": 1493713162, "microseconds": 369668}, "event": "DUMP_COMPLETED", "data": {"result": {"total": 8606908416, "status": "completed", "completed": 8606908416}}}
Takes a few seconds to the STOP event, then some more to the DUMP_COMPLETED event. It seems to crash right after. Backtrace:
qemu-kvm: /builddir/build/BUILD/qemu-2.9.0/memory.c:914: memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()' failed.
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe3028700 (LWP 32154)]
0x00007fffed9461f7 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install boost-system-1.53.0-27.el7.x86_64 boost-thread-1.53.0-27.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 celt051-0.5.1.3-8.el7.x86_64 dbus-libs-1.6.12-17.el7.x86_64 elfutils-libelf-0.168-5.el7.x86_64 elfutils-libs-0.168-5.el7.x86_64 flac-libs-1.3.0-5.el7_1.x86_64 gmp-6.0.0-15.el7.x86_64 gsm-1.0.13-11.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-5.el7.x86_64 libICE-1.0.9-5.el7.x86_64 libSM-1.2.2-2.el7.x86_64 libX11-1.6.4-4.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXi-1.7.9-1.el7.x86_64 libXtst-1.2.3-1.el7.x86_64 libasyncns-0.8-7.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libdb-5.3.21-20.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libibverbs-13-4.el7.x86_64 libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libnl3-3.2.28-3.el7_3.x86_64 libogg-1.3.0-7.el7.x86_64 librdmacm-13-4.el7.x86_64 libselinux-2.5-11.el7.x86_64 libsndfile-1.0.25-10.el7.x86_64 libtasn1-4.10-1.el7.x86_64 libunwind-1.2-2.el7.x86_64 libvorbis-1.3.3-8.el7.x86_64 libxcb-1.12-1.el7.x86_64 nspr-4.13.1-1.0.el7.x86_64 nss-3.28.3-5.el7.x86_64 nss-softokn-freebl-3.28.3-4.el7.x86_64 nss-util-3.28.3-3.el7.x86_64 openldap-2.4.44-3.el7.x86_64 openssl-libs-1.0.2k-5.el7.x86_64 p11-kit-0.23.5-1.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-32.el7.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64
(gdb) set height 0
(gdb) thread apply all backtrace full
Thread 9 (Thread 0x7fffe3028700 (LWP 32154)):
#0 0x00007fffed9461f7 in __GI_raise (sig=sig@entry=6)
at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 32053
selftid = 32154
#1 0x00007fffed9478e8 in __GI_abort () at abort.c:90
save_stage = 2
act =
{__sigaction_handler = {sa_handler = 0x7fffffffe0a0, sa_sigaction = 0x7fffffffe0a0}, sa_mask = {__val = {140737180659568, 93824997708984, 914, 93825044486464, 140737179290563, 4, 140737001976688, 1483905344, 12899545671512211968, 93825022016112, 0, 0, 0, 21474836480, 140737180659568, 140737180671592}}, sa_flags = -135835648, sa_restorer = 0x7fffeda91e68}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007fffed93f266 in __assert_fail_base (fmt=0x7fffeda91e68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x555555a87316 "qemu_mutex_iothread_locked()", file=file@entry=0x555555a8d4b8 "/builddir/build/BUILD/qemu-2.9.0/memory.c", line=line@entry=914, function=function@entry=0x555555a8dc20 <__PRETTY_FUNCTION__.28716> "memory_region_transaction_commit") at assert.c:92
str = 0x555556ca99e0 ""
total = 4096
#3 0x00007fffed93f312 in __GI___assert_fail (assertion=assertion@entry=0x555555a87316 "qemu_mutex_iothread_locked()", file=file@entry=0x555555a8d4b8 "/builddir/build/BUILD/qemu-2.9.0/memory.c", line=line@entry=914, function=function@entry=0x555555a8dc20 <__PRETTY_FUNCTION__.28716> "memory_region_transaction_commit")
at assert.c:101
#4 0x00005555557af9ca in memory_region_transaction_commit ()
at /usr/src/debug/qemu-2.9.0/memory.c:914
as = <optimized out>
#5 0x00005555557b1611 in memory_region_add_eventfd (mr=mr@entry=0x55555904ce20, addr=<optimized out>, size=size@entry=0, match_data=<optimized out>, data=<optimized out>, e=<optimized out>) at /usr/src/debug/qemu-2.9.0/memory.c:1989
mrfd =
{addr = {start = 0x00000000000000000000000000000000, size = 0x00000000000000000000000000000000}, match_data = false, data = 0, e = 0x5555590de068}
i = <optimized out>
#6 0x00005555559569b4 in virtio_pci_ioeventfd_assign (d=0x55555904c000, notifier=0x5555590de068, n=0, assign=<optimized out>) at hw/virtio/virtio-pci.c:304
proxy = 0x55555904c000
vdev = <optimized out>
vq = <optimized out>
legacy = true
modern = true
fast_mmio = <optimized out>
modern_pio = false
modern_mr = 0x55555904ce20
modern_notify_mr = 0x55555904cf30
legacy_mr = 0x55555904c9f0
modern_addr = <optimized out>
legacy_addr = 16
#7 0x000055555595a320 in virtio_bus_set_host_notifier (bus=<optimized out>, n=n@entry=0, assign=assign@entry=true) at hw/virtio/virtio-bus.c:283
vdev = 0x555559054510
k = 0x555556cecb40
__func__ = "virtio_bus_set_host_notifier"
proxy = 0x55555904c000
vq = <optimized out>
notifier = 0x5555590de068
r = 0
#8 0x00005555557c7155 in virtio_blk_data_plane_start (vdev=<optimized out>)
at /usr/src/debug/qemu-2.9.0/hw/block/dataplane/virtio-blk.c:188
vblk = 0x555559054510
__func__ = "virtio_blk_data_plane_start"
s = 0x55555907b380
qbus = 0x555559054498
k = <optimized out>
i = 0
nvqs = 1
r = <optimized out>
#9 0x0000555555959fda in virtio_bus_start_ioeventfd (bus=0x555559054498)
at hw/virtio/virtio-bus.c:223
k = 0x555556cecb40
__func__ = "virtio_bus_start_ioeventfd"
proxy = 0x55555904c000
vdev = 0x555559054510
vdc = 0x555556ccc000
r = <optimized out>
#10 0x00005555557eb5e4 in virtio_vmstate_change (opaque=0x555559054510, running=<optimized out>, state=<optimized out>)
at /usr/src/debug/qemu-2.9.0/hw/virtio/virtio.c:2230
vdev = 0x555559054510
qbus = 0x555559054498
__func__ = "virtio_vmstate_change"
k = 0x555556cecb40
backend_run = <optimized out>
#11 0x0000555555877a02 in vm_state_notify (running=running@entry=1, state=state@entry=RUN_STATE_RUNNING) at vl.c:1595
e = <optimized out>
next = 0x5555587299e0
#12 0x000055555579a300 in vm_prepare_start ()
at /usr/src/debug/qemu-2.9.0/cpus.c:1821
requested = RUN_STATE__MAX
res = 0
#13 0x000055555579a369 in vm_start () at /usr/src/debug/qemu-2.9.0/cpus.c:1831
#14 0x00005555557b9c75 in dump_cleanup (s=s@entry=0x555556062f60 <dump_state_global>) at /usr/src/debug/qemu-2.9.0/dump.c:80
#15 0x00005555557ba8f4 in dump_process (s=0x555556062f60 <dump_state_global>, errp=errp@entry=0x7fffe3027980) at /usr/src/debug/qemu-2.9.0/dump.c:1687
local_err = 0x0
result = 0x555558729520
__PRETTY_FUNCTION__ = "dump_process"
#16 0x00005555557bc094 in dump_thread (data=<optimized out>)
at /usr/src/debug/qemu-2.9.0/dump.c:1694
err = 0x0
s = <optimized out>
#17 0x00007fffedcdbdc5 in start_thread (arg=0x7fffe3028700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7fffe3028700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140737001981696, 5111790796432296765, 1, 140737001982400, 140737001981696, 50, -5111730018288579779, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#18 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 8 (Thread 0x7ffdde9ff700 (LWP 32129)):
#0 0x00007fffedcdf6d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000555555a6de39 in qemu_cond_wait (cond=cond@entry=0x555556cb6b80, mutex=mutex@entry=0x555556cb6bb0) at util/qemu-thread-posix.c:133
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x00005555559ac84b in vnc_worker_thread_loop (queue=queue@entry=0x555556cb6b80) at ui/vnc-jobs.c:205
job = <optimized out>
entry = <optimized out>
tmp = <optimized out>
vs =
{sioc = 0x0, ioc = 0x0, ioc_tag = 0, disconnecting = 0, dirty = {{0, 0, 0} <repeats 2048 times>}, lossy_rect = 0x0, vd = 0x0, need_update = 0, force_update = 0, has_dirty = 0, features = 0, absolute = 0, last_x = 0, last_y = 0, last_bmask = 0, client_width = 0, client_height = 0, share_mode = 0, vnc_encoding = 0, major = 0, minor = 0, auth = 0, subauth = 0, challenge = '\000' <repeats 15 times>, tls = 0x0, sasl = {conn = 0x0, wantSSF = false, runSSF = false, waitWriteSSF = 0, encoded = 0x0, encodedLength = 0, encodedOffset = 0, username = 0x0, mechlist = 0x0}, encode_ws = false, websocket = false, info = 0x0, output = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, input = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, write_pixels = 0x0, client_pf = {bits_per_pixel = 0 '\000', bytes_per_pixel = 0 '\000', depth = 0 '\000', rmask = 0, gmask = 0, bmask = 0, amask = 0, rshift = 0 '\000', gshift = 0 '\000', bshift = 0 '\000', ashift = 0 '\000', rmax = 0 '\000', gmax = 0 '\000', bmax = 0 '\000', amax = 0 '\000', rbits = 0 '\000', gbits = 0 '\000', bbits = 0 '\000', abits = 0 '\000'}, client_format = 0, client_be = false, audio_cap = 0x0, as = {freq = 0, nchannels = 0, fmt = AUD_FMT_U8, endianness = 0}, read_handler = 0x0, read_handler_expect = 0, modifiers_state = '\000' <repeats 255 times>, abort = false, output_mutex = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, bh = 0x0, jobs_buffer = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, tight = {type = 0, quality = 0 '\000', compression = 0 '\000', pixel24 = 0 '\000', tight = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, tmp = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, zlib = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, gradient = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, png = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, levels = {0, 0, 0, 0}, stream = {{next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0x0, zfree = 0x0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0x0, zfree = 0x0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0x0, zfree = 0x0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0x0, zfree = 0x0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}}}, zlib = {zlib = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, tmp = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, stream = {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0x0, zfree = 0x0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, level = 0}, hextile = {send_tile = 0x0}, zrle = {type = 0, fb = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, zrle = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, tmp = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, zlib = {name = 0x0, capacity = 0, offset = 0, avg_size = 0, buffer = 0x0}, stream = {next_in = 0x0, avail_in = 0, total_in = 0, next_out = 0x0, avail_out = 0, total_out = 0, msg = 0x0, state = 0x0, zalloc = 0x0, zfree = 0x0, opaque = 0x0, data_type = 0, adler = 0, reserved = 0}, palette = {pool = {{idx = 0, color = 0, next = {le_next = 0x0, le_prev = 0x0}} <repeats 256 times>}, size = 0, max = 0, bpp = 0, table = {{lh_first = 0x0} <repeats 256 times>}}}, zywrle = {buf = {0 <repeats 4096 times>}}, mouse_mode_notifier = {notify = 0x0, node = {le_next = 0x0, le_prev = 0x0}}, next = {tqe_next = 0x0, tqe_prev = 0x0}}
n_rectangles = <optimized out>
saved_offset = <optimized out>
#3 0x00005555559acd88 in vnc_worker_thread (arg=0x555556cb6b80)
at ui/vnc-jobs.c:312
queue = 0x555556cb6b80
#4 0x00007fffedcdbdc5 in start_thread (arg=0x7ffdde9ff700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7ffdde9ff700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140728338478848, 5111790796432296765, 1, 140728338479552, 140728338478848, 93825016753024, -5112984591657514179, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 7 (Thread 0x7fffe0c20700 (LWP 32074)):
#0 0x00007fffedcdf6d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000555555a6de39 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x555556060fe0 <qemu_global_mutex>) at util/qemu-thread-posix.c:133
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x000055555579947b in qemu_kvm_cpu_thread_fn (cpu=<optimized out>)
at /usr/src/debug/qemu-2.9.0/cpus.c:1085
cpu = 0x5555571e4000
r = <optimized out>
#3 0x000055555579947b in qemu_kvm_cpu_thread_fn (arg=0x5555571e4000)
at /usr/src/debug/qemu-2.9.0/cpus.c:1123
cpu = 0x5555571e4000
r = <optimized out>
#4 0x00007fffedcdbdc5 in start_thread (arg=0x7fffe0c20700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7fffe0c20700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140736964200192, 5111790796432296765, 1, 140736964200896, 140736964200192, 93825022181376, -5111731646081184963, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 6 (Thread 0x7fffe1421700 (LWP 32073)):
#0 0x00007fffedcdf6d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000555555a6de39 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x555556060fe0 <qemu_global_mutex>) at util/qemu-thread-posix.c:133
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x000055555579947b in qemu_kvm_cpu_thread_fn (cpu=<optimized out>)
at /usr/src/debug/qemu-2.9.0/cpus.c:1085
cpu = 0x5555571c0000
r = <optimized out>
#3 0x000055555579947b in qemu_kvm_cpu_thread_fn (arg=0x5555571c0000)
at /usr/src/debug/qemu-2.9.0/cpus.c:1123
cpu = 0x5555571c0000
r = <optimized out>
#4 0x00007fffedcdbdc5 in start_thread (arg=0x7fffe1421700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7fffe1421700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140736972592896, 5111790796432296765, 1, 140736972593600, 140736972592896, 93825022033920, -5111734967701517507, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 5 (Thread 0x7fffe1c22700 (LWP 32071)):
#0 0x00007fffedcdf6d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000555555a6de39 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x555556060fe0 <qemu_global_mutex>) at util/qemu-thread-posix.c:133
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x000055555579947b in qemu_kvm_cpu_thread_fn (cpu=<optimized out>)
at /usr/src/debug/qemu-2.9.0/cpus.c:1085
cpu = 0x5555571a4000
r = <optimized out>
#3 0x000055555579947b in qemu_kvm_cpu_thread_fn (arg=0x5555571a4000)
at /usr/src/debug/qemu-2.9.0/cpus.c:1123
cpu = 0x5555571a4000
r = <optimized out>
#4 0x00007fffedcdbdc5 in start_thread (arg=0x7fffe1c22700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7fffe1c22700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140736980985600, 5111790796432296765, 1, 140736980986304, 140736980985600, 93825021919232, -5111733867653018819, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 4 (Thread 0x7fffe2423700 (LWP 32069)):
#0 0x00007fffedcdf6d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x0000555555a6de39 in qemu_cond_wait (cond=<optimized out>, mutex=mutex@entry=0x555556060fe0 <qemu_global_mutex>) at util/qemu-thread-posix.c:133
err = <optimized out>
__func__ = "qemu_cond_wait"
#2 0x000055555579947b in qemu_kvm_cpu_thread_fn (cpu=<optimized out>)
at /usr/src/debug/qemu-2.9.0/cpus.c:1085
cpu = 0x55555712a000
r = <optimized out>
#3 0x000055555579947b in qemu_kvm_cpu_thread_fn (arg=0x55555712a000)
at /usr/src/debug/qemu-2.9.0/cpus.c:1123
cpu = 0x55555712a000
r = <optimized out>
#4 0x00007fffedcdbdc5 in start_thread (arg=0x7fffe2423700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7fffe2423700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140736989378304, 5111790796432296765, 1, 140736989379008, 140736989378304, 93825021419520, -5111728371705492675, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 2 (Thread 0x7fffe3a3c700 (LWP 32057)):
#0 0x00007fffeda037f9 in syscall ()
at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x0000555555a6e146 in qemu_event_wait (val=<optimized out>, f=<optimized out>) at /usr/src/debug/qemu-2.9.0/include/qemu/futex.h:26
value = <optimized out>
#2 0x0000555555a6e146 in qemu_event_wait (ev=ev@entry=0x55555647e544 <rcu_call_ready_event>) at util/qemu-thread-posix.c:399
value = <optimized out>
#3 0x0000555555a7d76e in call_rcu_thread (opaque=<optimized out>)
at util/rcu.c:249
tries = 0
n = <optimized out>
node = <optimized out>
#4 0x00007fffedcdbdc5 in start_thread (arg=0x7fffe3a3c700)
at pthread_create.c:308
__res = <optimized out>
pd = 0x7fffe3a3c700
now = <optimized out>
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {140737012549376, 5111790796432296765, 1, 140737012550080, 140737012549376, 0, -5111728641751561411, -5111760132582192323}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
#5 0x00007fffeda0934d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 1 (Thread 0x7ffff7f7ccc0 (LWP 32053)):
#0 0x00007fffef1cba10 in g_array_set_size (farray=0x555556c89d00, length=length@entry=0) at garray.c:548
array = 0x555556c89d00
__FUNCTION__ = "g_array_set_size"
#1 0x0000555555a6add7 in main_loop_wait (nonblocking=nonblocking@entry=0)
at util/main-loop.c:501
ret = -1
timeout = 4294967295
timeout_ns = <optimized out>
#2 0x000055555575bcdc in main () at vl.c:1898
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = <optimized out>
boot_once = 0x555556c80888 "c"
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
hda_opts = <optimized out>
icount_opts = <optimized out>
accel_opts = <optimized out>
olist = <optimized out>
optind = 45
optarg = 0x7fffffffe4c3 "tcp:localhost:4444,server,nowait"
loadvm = <optimized out>
machine_class = 0x0
cpu_model = <optimized out>
vga_model = 0x7fffffffe0ea "cirrus"
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
defconfig = <optimized out>
userconfig = <optimized out>
nographic = <optimized out>
display_type = <optimized out>
display_remote = <optimized out>
log_mask = <optimized out>
log_file = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
err = 0x0
list_data_dirs = <optimized out>
bdo_queue = {sqh_first = 0x0, sqh_last = 0x7fffffffda10}
__func__ = "main"
__FUNCTION__ = "main"
#3 0x000055555575bcdc in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at vl.c:4720
i = <optimized out>
snapshot = <optimized out>
linux_boot = <optimized out>
initrd_filename = <optimized out>
kernel_filename = <optimized out>
kernel_cmdline = <optimized out>
boot_order = <optimized out>
boot_once = 0x555556c80888 "c"
cyls = <optimized out>
heads = <optimized out>
secs = <optimized out>
translation = <optimized out>
opts = <optimized out>
machine_opts = <optimized out>
hda_opts = <optimized out>
icount_opts = <optimized out>
accel_opts = <optimized out>
olist = <optimized out>
optind = 45
optarg = 0x7fffffffe4c3 "tcp:localhost:4444,server,nowait"
loadvm = <optimized out>
machine_class = 0x0
cpu_model = <optimized out>
vga_model = 0x7fffffffe0ea "cirrus"
qtest_chrdev = <optimized out>
qtest_log = <optimized out>
pid_file = <optimized out>
incoming = <optimized out>
defconfig = <optimized out>
userconfig = <optimized out>
nographic = <optimized out>
display_type = <optimized out>
display_remote = <optimized out>
log_mask = <optimized out>
log_file = <optimized out>
trace_file = <optimized out>
maxram_size = <optimized out>
ram_slots = <optimized out>
vmstate_dump_file = <optimized out>
main_loop_err = 0x0
err = 0x0
list_data_dirs = <optimized out>
bdo_queue = {sqh_first = 0x0, sqh_last = 0x7fffffffda10}
__func__ = "main"
__FUNCTION__ = "main"
Proposed a fix for upstream: https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg00429.html Fix included in qemu-kvm-rhev-2.9.0-5.el7 I test on
qemu-kvm-rhev-2.9.0-5.el7.x86_64
Host:
kernel-debuginfo-3.10.0-656.el7.x86_64
kernel-3.10.0-656.el7.x86_64
kernel-debuginfo-common-x86_64-3.10.0-656.el7.x86_64
{ "execute": "qmp_capabilities" }
{"return": {}}
{"timestamp": {"seconds": 1495609901, "microseconds": 378052}, "event": "VNC_CONNECTED", "data": {"server": {"auth": "none", "family": "ipv4", "service": "5900", "host": "0.0.0.0", "websocket": false}, "client": {"family": "ipv4", "service": "58234", "host": "10.66.61.77", "websocket": false}}}
{"timestamp": {"seconds": 1495609901, "microseconds": 396866}, "event": "VNC_INITIALIZED", "data": {"server": {"auth": "none", "family": "ipv4", "service": "5900", "host": "0.0.0.0", "websocket": false}, "client": {"family": "ipv4", "service": "58234", "host": "10.66.61.77", "websocket": false}}}
{"execute": "dump-guest-memory", "arguments": { "detach": true, "paging": false, "protocol": "file:/home/dump.normal"}}
{"timestamp": {"seconds": 1495609923, "microseconds": 605537}, "event": "STOP"}
{"return": {}}
{"timestamp": {"seconds": 1495609942, "microseconds": 84745}, "event": "DUMP_COMPLETED", "data": {"result": {"total": 8606908416, "status": "completed", "completed": 8606908416}}}
{"timestamp": {"seconds": 1495609942, "microseconds": 85550}, "event": "RESUME"}
Given information above, bug verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:2392 |
Description of problem: [guest memory dump]dump-guest-memory QMP command with "detach" param makes qemu-kvm process aborted. (qemu) qemu-kvm: /builddir/build/BUILD/qemu-2.9.0/memory.c:914: memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()' failed. Version-Release number of selected component (if applicable): Test on: qemu-kvm-rhev-2.9.0-1.el7.x86_64 Host: kernel-debuginfo-3.10.0-656.el7.x86_64 kernel-3.10.0-656.el7.x86_64 kernel-debuginfo-common-x86_64-3.10.0-656.el7.x86_64 How reproducible: 3/3 Steps to Reproduce: 1.Boot up a guest eg, /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox off \ -machine pc \ -nodefaults \ -vga cirrus \ -device pvpanic,ioport=0x505,id=idHT1RPm \ -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 \ -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 \ -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 \ -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 \ -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x3 \ -device virtio-net-pci,mac=9a:1b:1c:1d:1e:1f,id=id8xeo6O,vectors=4,netdev=idBP1nUD,bus=pci.0,addr=0x4 \ -netdev tap,id=idBP1nUD \ -m 4086 \ -smp 4,cores=2,threads=1,sockets=2 \ -cpu 'Westmere',+kvm_pv_unhalt \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot order=cdn,once=c,menu=off,strict=off \ -enable-kvm \ -monitor stdio \ -qmp tcp:localhost:4444,server,nowait \ 2. Check dump-guest-memory command. #telnet localhost 4444 QMP: { "execute": "qmp_capabilities" } {"return": {}} {"execute": "dump-guest-memory", "arguments": { "detach": true, "paging": false, "protocol": "file:/home/dump.normal"}} Actual results: QMP: {"timestamp": {"seconds": 1493106109, "microseconds": 745723}, "event": "STOP"} {"return": {}} {"timestamp": {"seconds": 1493106114, "microseconds": 299349}, "event": "DUMP_COMPLETED", "data": {"result": {"total": 4301455360, "status": "completed", "completed": 4301455360}}} Connection closed by foreign host. HMP: (qemu) qemu-kvm: /builddir/build/BUILD/qemu-2.9.0/memory.c:914: memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()' failed. guest_dump.sh: line 26: 7342 Aborted /usr/libexec/qemu-kvm -name 'avocado-vt-vm1' -sandbox off -machine pc -nodefaults -vga cirrus -device pvpanic,ioport=0x505,id=idHT1RPm -device ich9-usb-ehci1,id=usb1,addr=0x1d.7,multifunction=on,bus=pci.0 -device ich9-usb-uhci1,id=usb1.0,multifunction=on,masterbus=usb1.0,addr=0x1d.0,firstport=0,bus=pci.0 -device ich9-usb-uhci2,id=usb1.1,multifunction=on,masterbus=usb1.0,addr=0x1d.2,firstport=2,bus=pci.0 -device ich9-usb-uhci3,id=usb1.2,multifunction=on,masterbus=usb1.0,addr=0x1d.4,firstport=4,bus=pci.0 -drive id=drive_image1,if=none,snapshot=off,aio=native,cache=none,format=qcow2,file=/home/kvm_autotest_root/images/rhel74-64-virtio.qcow2 -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,bus=pci.0,addr=0x3 -device virtio-net-pci,mac=9a:1b:1c:1d:1e:1f,id=id8xeo6O,vectors=4,netdev=idBP1nUD,bus=pci.0,addr=0x4 -netdev tap,id=idBP1nUD -m 4086 -smp 4,cores=2,threads=1,sockets=2 -cpu 'Westmere',+kvm_pv_unhalt -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 -vnc :0 -rtc base=utc,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -monitor stdio -qmp tcp:localhost:4444,server,nowait Expected results: {"timestamp": {"seconds": 1489137867, "microseconds": 606103}, "event": "STOP"} {"return": {}} {"timestamp": {"seconds": 1489137868, "microseconds": 716173}, "event": "DUMP_COMPLETED", "data": {"result": {"total": 2164457472, "status": "completed", "completed": 2164457472}}} {"timestamp": {"seconds": 1489137868, "microseconds": 716518}, "event": "RESUME"} Additional info: