Bug 1203030 - [abrt] libvirt-daemon: virObjectUnref(): libvirtd killed by SIGSEGV
Summary: [abrt] libvirt-daemon: virObjectUnref(): libvirtd killed by SIGSEGV
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: libvirt
Version: 21
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Krempa
QA Contact: Fedora Extras Quality Assurance
URL: https://retrace.fedoraproject.org/faf...
Whiteboard: abrt_hash:ea5cee34b05f86ae71b7a34c7b1...
: 1145269 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-17 23:59 UTC by Neil
Modified: 2015-06-05 23:02 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-05 23:02:16 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: backtrace (47.16 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: cgroup (196 bytes, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: core_backtrace (4.41 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: dso_list (10.58 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: environ (106 bytes, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: exploitable (82 bytes, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: limits (1.29 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: maps (49.92 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: open_fds (1.19 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: proc_pid_status (907 bytes, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details
File: var_log_messages (10.00 KB, text/plain)
2015-03-17 23:59 UTC, Neil
no flags Details

Description Neil 2015-03-17 23:59:37 UTC
Version-Release number of selected component:
libvirt-daemon-1.2.9.2-1.fc21

Additional info:
reporter:       libreport-2.3.0
backtrace_rating: 4
cmdline:        /usr/sbin/libvirtd
crash_function: virObjectUnref
executable:     /usr/sbin/libvirtd
kernel:         3.18.7-200.fc21.x86_64
runlevel:       N 5
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 virObjectUnref at util/virobject.c:264
 #1 remoteRelayDomainEventCheckACL at remote.c:164
 #2 remoteRelayDomainEventTrayChange at remote.c:717
 #3 virDomainEventDispatchDefaultFunc at conf/domain_event.c:1455
 #4 virObjectEventStateDispatchCallbacks at conf/object_event.c:724
 #5 virObjectEventStateQueueDispatch at conf/object_event.c:738
 #6 virObjectEventStateFlush at conf/object_event.c:816
 #7 virObjectEventTimer at conf/object_event.c:562
 #8 virEventPollDispatchTimeouts at util/vireventpoll.c:459
 #9 virEventPollRunOnce at util/vireventpoll.c:656

Comment 1 Neil 2015-03-17 23:59:41 UTC
Created attachment 1002989 [details]
File: backtrace

Comment 2 Neil 2015-03-17 23:59:42 UTC
Created attachment 1002990 [details]
File: cgroup

Comment 3 Neil 2015-03-17 23:59:43 UTC
Created attachment 1002991 [details]
File: core_backtrace

Comment 4 Neil 2015-03-17 23:59:44 UTC
Created attachment 1002992 [details]
File: dso_list

Comment 5 Neil 2015-03-17 23:59:45 UTC
Created attachment 1002993 [details]
File: environ

Comment 6 Neil 2015-03-17 23:59:46 UTC
Created attachment 1002994 [details]
File: exploitable

Comment 7 Neil 2015-03-17 23:59:46 UTC
Created attachment 1002995 [details]
File: limits

Comment 8 Neil 2015-03-17 23:59:47 UTC
Created attachment 1002996 [details]
File: maps

Comment 9 Neil 2015-03-17 23:59:48 UTC
Created attachment 1002997 [details]
File: open_fds

Comment 10 Neil 2015-03-17 23:59:49 UTC
Created attachment 1002998 [details]
File: proc_pid_status

Comment 11 Neil 2015-03-17 23:59:50 UTC
Created attachment 1002999 [details]
File: var_log_messages

Comment 12 Peter Krempa 2015-03-25 07:46:40 UTC
The backtrace shows that there is a race between threads 1 and 11. Thread 1 is trying to relay the event while thread 11 is disposing the connection and leaves the identity pointer set while the actual object is already unref'd

Comment 13 Peter Krempa 2015-03-25 11:26:35 UTC
*** Bug 1145269 has been marked as a duplicate of this bug. ***

Comment 14 Peter Krempa 2015-03-25 12:53:01 UTC
The following upstream commits should fix the problem:

commit a98129c0ee52b6a8fdd39988a6d090057f149ae9
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 25 08:56:07 2015 +0100

    rpc: Don't unref identity object while callbacks still can be executed
    
    While this thread is cleaning up the client and connection objects:
     #2  virFileReadAll (path=0x7f28780012b0 "/proc/1319/stat", maxlen=maxlen@entry=1024, buf=buf@entry=0x7f289c60fc40) at util/virfile.c:1287
     #3  0x00007f28adbb1539 in virProcessGetStartTime (pid=<optimized out>, timestamp=timestamp@entry=0x7f289c60fc98) at util/virprocess.c:838
     #4  0x00007f28adb91981 in virIdentityGetSystem () at util/viridentity.c:151
     #5  0x00007f28ae73f17c in remoteClientFreeFunc (data=<optimized out>) at remote.c:1131
     #6  0x00007f28adcb7f33 in virNetServerClientDispose (obj=0x7f28aecad180) at rpc/virnetserverclient.c:858
     #7  0x00007f28adba8eeb in virObjectUnref (anyobj=<optimized out>) at util/virobject.c:265
     #8  0x00007f28ae74ad05 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f28aec93ff0) at rpc/virnetserver.c:205
     #9  0x00007f28adbbef4e in virThreadPoolWorker (opaque=opaque@entry=0x7f28aec88030) at util/virthreadpool.c:145
    
    In stack frame #6 the client->identity object got unref'd, but the code
    that removes the event callbacks in frame #5 did not run yet as we are
    trying to obtain the system identity (frames #4, #3, #2).
    
    In other thead:
     #0  virObjectUnref (anyobj=anyobj@entry=0x7f288c162c60) at util/virobject.c:264
            klass = 0xdeadbeef
            obj = 0x7f288c162c60
     #1  0x00007f28ae71c709 in remoteRelayDomainEventCheckACL (client=<optimized out>, conn=<optimized out>, dom=dom@entry=0x7f28aecaafc0) at remote.c:164
     #2  0x00007f28ae71fc83 in remoteRelayDomainEventTrayChange (conn=<optimized out>, dom=0x7f28aecaafc0, ... ) at remote.c:717
     #3  0x00007f28adc04e53 in virDomainEventDispatchDefaultFunc (conn=0x7f287c0009a0, event=0x7f28aecab1a0, ...) at conf/domain_event.c:1455
     #4  0x00007f28adc03831 in virObjectEventStateDispatchCallbacks (callbacks=<optimized out>, ....) at conf/object_event.c:724
     #5  virObjectEventStateQueueDispatch (callbacks=0x7f288c083730, queue=0x7fff51f90030, state=0x7f288c18da20) at conf/object_event.c:738
     #6  virObjectEventStateFlush (state=0x7f288c18da20) at conf/object_event.c:816
     #7  virObjectEventTimer (timer=<optimized out>, opaque=0x7f288c18da20) at conf/object_event.c:562
     #8  0x00007f28adb859cd in virEventPollDispatchTimeouts () at util/vireventpoll.c:459
    
    Frame #0 is unrefing an invalid identity object while frame #2 hints
    that the client is still dispatching the event.
    
    For untrimmed backtrace see the bugzilla attachment.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1203030

commit 6ca857c7c8a1f7b571132d6c7fff5a06301a5e9a
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 25 08:52:26 2015 +0100

    daemon: Clear fake domain def object that is used to check ACL prior to use
    
    The fake object is used to pass the domain name and UUID to the ACL code
    for events where we don't have the full domain def when dispatching
    events. The rest of the entries would be left uninitialized. While this
    is not a problem code-wise as the used fields are initialized it looks
    ugly in the debugger.

commit ad886fa6c8ebc321a0386a75c187d315111cf1f3
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 25 08:25:45 2015 +0100

    util: identity: Harden virIdentitySetCurrent()
    
    Don't unref the old identity unless we set the new one correctly and
    unref the new one on failure to set it so that we don't leak any
    references or use invalid pointers.

v1.2.13-308-ga98129c

Comment 15 Fedora Update System 2015-04-28 16:54:54 UTC
libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21

Comment 16 Fedora Update System 2015-04-29 13:06:03 UTC
Package libvirt-1.2.9.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21
then log in and leave karma (feedback).


Note You need to log in before you can comment on or make changes to this bug.