Bug 1203030

Created attachment 1002989 [details]
File: backtrace

Created attachment 1002990 [details]
File: cgroup

Created attachment 1002991 [details]
File: core_backtrace

Created attachment 1002992 [details]
File: dso_list

Created attachment 1002993 [details]
File: environ

Created attachment 1002994 [details]
File: exploitable

Created attachment 1002995 [details]
File: limits

Created attachment 1002996 [details]
File: maps

Created attachment 1002997 [details]
File: open_fds

Created attachment 1002998 [details]
File: proc_pid_status

Created attachment 1002999 [details]
File: var_log_messages

The backtrace shows that there is a race between threads 1 and 11. Thread 1 is trying to relay the event while thread 11 is disposing the connection and leaves the identity pointer set while the actual object is already unref'd

*** Bug 1145269 has been marked as a duplicate of this bug. ***

The following upstream commits should fix the problem:

commit a98129c0ee52b6a8fdd39988a6d090057f149ae9
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 25 08:56:07 2015 +0100

    rpc: Don't unref identity object while callbacks still can be executed
    
    While this thread is cleaning up the client and connection objects:
     #2  virFileReadAll (path=0x7f28780012b0 "/proc/1319/stat", maxlen=maxlen@entry=1024, buf=buf@entry=0x7f289c60fc40) at util/virfile.c:1287
     #3  0x00007f28adbb1539 in virProcessGetStartTime (pid=<optimized out>, timestamp=timestamp@entry=0x7f289c60fc98) at util/virprocess.c:838
     #4  0x00007f28adb91981 in virIdentityGetSystem () at util/viridentity.c:151
     #5  0x00007f28ae73f17c in remoteClientFreeFunc (data=<optimized out>) at remote.c:1131
     #6  0x00007f28adcb7f33 in virNetServerClientDispose (obj=0x7f28aecad180) at rpc/virnetserverclient.c:858
     #7  0x00007f28adba8eeb in virObjectUnref (anyobj=<optimized out>) at util/virobject.c:265
     #8  0x00007f28ae74ad05 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f28aec93ff0) at rpc/virnetserver.c:205
     #9  0x00007f28adbbef4e in virThreadPoolWorker (opaque=opaque@entry=0x7f28aec88030) at util/virthreadpool.c:145
    
    In stack frame #6 the client->identity object got unref'd, but the code
    that removes the event callbacks in frame #5 did not run yet as we are
    trying to obtain the system identity (frames #4, #3, #2).
    
    In other thead:
     #0  virObjectUnref (anyobj=anyobj@entry=0x7f288c162c60) at util/virobject.c:264
            klass = 0xdeadbeef
            obj = 0x7f288c162c60
     #1  0x00007f28ae71c709 in remoteRelayDomainEventCheckACL (client=<optimized out>, conn=<optimized out>, dom=dom@entry=0x7f28aecaafc0) at remote.c:164
     #2  0x00007f28ae71fc83 in remoteRelayDomainEventTrayChange (conn=<optimized out>, dom=0x7f28aecaafc0, ... ) at remote.c:717
     #3  0x00007f28adc04e53 in virDomainEventDispatchDefaultFunc (conn=0x7f287c0009a0, event=0x7f28aecab1a0, ...) at conf/domain_event.c:1455
     #4  0x00007f28adc03831 in virObjectEventStateDispatchCallbacks (callbacks=<optimized out>, ....) at conf/object_event.c:724
     #5  virObjectEventStateQueueDispatch (callbacks=0x7f288c083730, queue=0x7fff51f90030, state=0x7f288c18da20) at conf/object_event.c:738
     #6  virObjectEventStateFlush (state=0x7f288c18da20) at conf/object_event.c:816
     #7  virObjectEventTimer (timer=<optimized out>, opaque=0x7f288c18da20) at conf/object_event.c:562
     #8  0x00007f28adb859cd in virEventPollDispatchTimeouts () at util/vireventpoll.c:459
    
    Frame #0 is unrefing an invalid identity object while frame #2 hints
    that the client is still dispatching the event.
    
    For untrimmed backtrace see the bugzilla attachment.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1203030

commit 6ca857c7c8a1f7b571132d6c7fff5a06301a5e9a
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 25 08:52:26 2015 +0100

    daemon: Clear fake domain def object that is used to check ACL prior to use
    
    The fake object is used to pass the domain name and UUID to the ACL code
    for events where we don't have the full domain def when dispatching
    events. The rest of the entries would be left uninitialized. While this
    is not a problem code-wise as the used fields are initialized it looks
    ugly in the debugger.

commit ad886fa6c8ebc321a0386a75c187d315111cf1f3
Author: Peter Krempa <pkrempa>
Date:   Wed Mar 25 08:25:45 2015 +0100

    util: identity: Harden virIdentitySetCurrent()
    
    Don't unref the old identity unless we set the new one correctly and
    unref the new one on failure to set it so that we don't leak any
    references or use invalid pointers.

v1.2.13-308-ga98129c

libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21

Package libvirt-1.2.9.3-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21
then log in and leave karma (feedback).