Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Hi,Julio
Could you provide some clear steps to reproduce this issue ?
It is hard for me to reproduce it without steps
Thanks very much
Wenlong
Comment 5Julio Entrena Perez
2013-03-25 11:58:13 UTC
(In reply to comment #4)
> Hi,Julio
>
> Could you provide some clear steps to reproduce this issue ?
> It is hard for me to reproduce it without steps
> Thanks very much
>
> Wenlong
Not really: we're not sure how to trigger the condition.
libvirt-0.10.2-18.el6_4.3.x86_64
vdsm-4.10.2-13.0.el6ev.x86_64
qemu-kvm-rhev-0.12.1.2-2.355.el6_4.2.x86_64
I use one host with 118 vms the host load average: 100.33, 92.45, 48.45
it can be reproduced this issue , start and shutoff vms via rhevm , libvirtd is still running .
Wenlong
(In reply to comment #11)
> libvirt-0.10.2-18.el6_4.3.x86_64
> vdsm-4.10.2-13.0.el6ev.x86_64
> qemu-kvm-rhev-0.12.1.2-2.355.el6_4.2.x86_64
>
> I use one host with 118 vms the host load average: 100.33, 92.45, 48.45
> it can be reproduced this issue , start and shutoff vms via rhevm ,
> libvirtd is still running .
>
>
> Wenlong
Sorry ! I miss a NOT ,
I can not reproduc this issue .
Do you have MALLOC_PERTURB_ set in the environment? If not, can you set it to a non-zero value, which will help glibc detect heap smashing bugs closer to the point at which they happen?
This commit mentions a crash possible for transient domains, but seems to focus on auto-destroy guests (those that go away when the virConnectPtr is closed) and might not be related to the setup you were using
commit 7ccad0b16d12d7616c7c21b1359f6a55a9677521
Author: Daniel P. Berrange <berrange>
Date: Thu Feb 28 12:18:48 2013 +0000
Fix crash in QEMU auto-destroy with transient guests
When the auto-destroy callback runs it is supposed to return
NULL if the virDomainObjPtr is no longer valid. It was not
doing this for transient guests, so we tried to virObjectUnlock
a mutex which had been freed. This often led to a crash.
Signed-off-by: Daniel P. Berrange <berrange>
(In reply to comment #15)
> was the domain being shut down transient or persistent?
VDSM (calling libvirtd here) only creates transient domains.
(In reply to comment #16)
> This commit mentions a crash possible for transient domains, but seems to
> focus on auto-destroy guests (those that go away when the virConnectPtr is
> closed) and might not be related to the setup you were using
>
> commit 7ccad0b16d12d7616c7c21b1359f6a55a9677521
> Author: Daniel P. Berrange <berrange>
> Date: Thu Feb 28 12:18:48 2013 +0000
>
> Fix crash in QEMU auto-destroy with transient guests
>
> When the auto-destroy callback runs it is supposed to return
> NULL if the virDomainObjPtr is no longer valid. It was not
> doing this for transient guests, so we tried to virObjectUnlock
> a mutex which had been freed. This often led to a crash.
>
> Signed-off-by: Daniel P. Berrange <berrange>
I'm not entirely sure how to configure auto destroy. Would the domain need to start with the VIR_DOMAIN_START_AUTODESTROY flag? AFAICT VDSM doesn't set this.
Lee
bug 915353 describes a crash on shutdown; it was fixed for libvirt-0.10.2-18.el6_4.1 - I'm starting to think that this particular fix is the one that solves the problem at hand.
Description of problem: libvirtd crashed around the same time a guest was shutdown from RHEV-M. Core was generated by `/usr/sbin/libvirtd --listen'. Program terminated with signal 6, Aborted. #0 0x00007f92202408a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) bt #0 0x00007f92202408a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007f9220242085 in abort () at abort.c:92 #2 0x00007f922027e7b7 in __libc_message (do_abort=2, fmt=0x7f9220365f80 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 #3 0x00007f92202840e6 in malloc_printerr (action=3, str=0x7f9220363f3e "corrupted double-linked list", ptr=<value optimized out>) at malloc.c:6311 #4 0x00007f92202844f0 in malloc_consolidate (av=0x7f91dc000020) at malloc.c:5181 #5 0x00007f9220286ba8 in _int_free (av=0x7f91dc000020, p=0x7f91dc126b30, have_lock=0) at malloc.c:5054 #6 0x00007f92227114b9 in virFree (ptrptr=0x7f91dc126850) at util/memory.c:309 #7 0x00007f92227295dd in virHashFree (table=0x7f91dc126850) at util/virhash.c:265 #8 0x00007f922275fd86 in virDomainSnapshotObjListFree (snapshots=0x7f91dc10f830) at conf/snapshot_conf.c:724 #9 0x00007f9222723dbb in virObjectUnref (anyobj=<value optimized out>) at util/virobject.c:139 #10 0x0000000000489ad2 in qemuMonitorDispose (obj=<value optimized out>) at qemu/qemu_monitor.c:248 #11 0x00007f9222723dbb in virObjectUnref (anyobj=<value optimized out>) at util/virobject.c:139 #12 0x00007f92227095a8 in virEventPollCleanupHandles () at util/event_poll.c:567 #13 0x00007f9222709c90 in virEventPollRunOnce () at util/event_poll.c:636 #14 0x00007f9222708b67 in virEventRunDefaultImpl () at util/event.c:247 #15 0x00007f92227f863d in virNetServerRun (srv=0x1d7cf60) at rpc/virnetserver.c:748 #16 0x00000000004235b7 in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:1228 Version-Release number of selected component (if applicable): libvirtd 0.10.2-18.el6 vdsm 4.10.2-1.6.el6 qemu-kvm 0.12.1.2-2.355.el6_4.1 qemu-img 0.12.1.2-2.355.el6_4.1 spice-server 0.12.0-12.el6 How reproducible: Unknown. Steps to Reproduce: 1. A guest was shutdown from RHEV-M: $ cat var/log/libvirt/qemu/i-web001.log | grep -B4 08\:34 inputs_connect: inputs channel client create qemu: terminating on signal 15 from pid 12005 red_channel_client_disconnect: 0x7fdf7c298f80 (channel 0x7fdf7c21d670 type 4 id 0) red_channel_client_disconnect: 0x7fdf7c2aa200 (channel 0x7fdf7c21d0b0 type 2 id 0) 2013-03-20 08:34:34.849+0000: shutting down 2. libvirtd crashed by SIGABRT: $ xzgrep error var/log/libvirtd.log.3.xz | grep -v debug 2013-03-20 08:34:34.848+0000: 12005: error : qemuMonitorIORead:513 : Unable to read from monitor: Connection reset by peer 2013-03-20 08:34:34.848+0000: 12005: error : qemuAgentIO:642 : internal error End of file from monitor 2013-03-20 08:34:34.850+0000: 12005: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet140" not in key map 2013-03-20 08:34:34.852+0000: 12005: error : virNetDevGetIndex:653 : Unable to get index for interface vnet140: No such device 2013-03-20 08:34:34.982+0000: 12005: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet141" not in key map 2013-03-20 08:34:34.984+0000: 12005: error : virNetDevGetIndex:653 : Unable to get index for interface vnet141: No such device 2013-03-20 08:34:34.848+000012005: error : qemuMonitorIORead:513 : Unable to read from monitor: Connection reset by peer 2013-03-20 08:34:34.848+000012005: error : qemuAgentIO:642 : internal error End of file from monitor 2013-03-20 08:34:34.850+000012005: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet140" not in key map 2013-03-20 08:34:34.852+000012005: error : virNetDevGetIndex:653 : Unable to get index for interface vnet140: No such device 2013-03-20 08:34:34.982+000012005: error : virNWFilterDHCPSnoopEnd:2131 : internal error ifname "vnet141" not in key map 2013-03-20 08:34:34.984+000012005: error : virNetDevGetIndex:653 : Unable to get index for interface vnet141: No such device /var/log/libvirtd.log.3.xz : 2013-03-20 08:34:35.362+0000: 12005: debug : virCgroupRemoveRecursively:727 : Removing cgroup /cgroup /freezer/libvirt/qemu/i-web001/ 2013-03-20 08:34:35.378+0000: 12005: debug : virCgroupRemove:772 : Removing cgroup /cgroup/blkio/libv irt/qemu/i-web001/ and all child cgroups 2013-03-20 08:34:35.378+0000: 12005: debug : virCgroupRemoveRecursively:727 : Removing cgroup /cgroup /blkio/libvirt/qemu/i-web001/ 2013-03-20 08:34:35.394+0000: 12005: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f91dc111420 2013-03-20 08:34:35.394+0000: 12005: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x7f91dc111420 2013-03-20 08:34:35.394+0000: 12005: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f91dc115bd0 2013-03-20 08:34:35.395+0000: 12005: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f91dc112c00 2013-03-20 08:34:35.395+0000: 12005: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x7f91dc112c00 2013-03-20 08:34:35.395+0000: 12005: debug : qemuMonitorDispose:246 : mon=0x7f91dc112c00 2013-03-20 08:34:35.395+0000: 12005: debug : virObjectUnref:135 : OBJECT_UNREF: obj=0x7f91dc115bd0 2013-03-20 08:34:35.395+0000: 12005: debug : virObjectUnref:137 : OBJECT_DISPOSE: obj=0x7f91dc115bd0 Caught abort signal dumping internal log buffer: Actual results: Expected results: Additional info: