Version-Release number of selected component: libvirt-daemon-1.2.9.2-1.fc21 Additional info: reporter: libreport-2.3.0 backtrace_rating: 4 cmdline: /usr/sbin/libvirtd crash_function: virObjectUnref executable: /usr/sbin/libvirtd kernel: 3.18.7-200.fc21.x86_64 runlevel: N 5 type: CCpp uid: 0 Truncated backtrace: Thread no. 1 (10 frames) #0 virObjectUnref at util/virobject.c:264 #1 remoteRelayDomainEventCheckACL at remote.c:164 #2 remoteRelayDomainEventTrayChange at remote.c:717 #3 virDomainEventDispatchDefaultFunc at conf/domain_event.c:1455 #4 virObjectEventStateDispatchCallbacks at conf/object_event.c:724 #5 virObjectEventStateQueueDispatch at conf/object_event.c:738 #6 virObjectEventStateFlush at conf/object_event.c:816 #7 virObjectEventTimer at conf/object_event.c:562 #8 virEventPollDispatchTimeouts at util/vireventpoll.c:459 #9 virEventPollRunOnce at util/vireventpoll.c:656
Created attachment 1002989 [details] File: backtrace
Created attachment 1002990 [details] File: cgroup
Created attachment 1002991 [details] File: core_backtrace
Created attachment 1002992 [details] File: dso_list
Created attachment 1002993 [details] File: environ
Created attachment 1002994 [details] File: exploitable
Created attachment 1002995 [details] File: limits
Created attachment 1002996 [details] File: maps
Created attachment 1002997 [details] File: open_fds
Created attachment 1002998 [details] File: proc_pid_status
Created attachment 1002999 [details] File: var_log_messages
The backtrace shows that there is a race between threads 1 and 11. Thread 1 is trying to relay the event while thread 11 is disposing the connection and leaves the identity pointer set while the actual object is already unref'd
*** Bug 1145269 has been marked as a duplicate of this bug. ***
The following upstream commits should fix the problem: commit a98129c0ee52b6a8fdd39988a6d090057f149ae9 Author: Peter Krempa <pkrempa> Date: Wed Mar 25 08:56:07 2015 +0100 rpc: Don't unref identity object while callbacks still can be executed While this thread is cleaning up the client and connection objects: #2 virFileReadAll (path=0x7f28780012b0 "/proc/1319/stat", maxlen=maxlen@entry=1024, buf=buf@entry=0x7f289c60fc40) at util/virfile.c:1287 #3 0x00007f28adbb1539 in virProcessGetStartTime (pid=<optimized out>, timestamp=timestamp@entry=0x7f289c60fc98) at util/virprocess.c:838 #4 0x00007f28adb91981 in virIdentityGetSystem () at util/viridentity.c:151 #5 0x00007f28ae73f17c in remoteClientFreeFunc (data=<optimized out>) at remote.c:1131 #6 0x00007f28adcb7f33 in virNetServerClientDispose (obj=0x7f28aecad180) at rpc/virnetserverclient.c:858 #7 0x00007f28adba8eeb in virObjectUnref (anyobj=<optimized out>) at util/virobject.c:265 #8 0x00007f28ae74ad05 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f28aec93ff0) at rpc/virnetserver.c:205 #9 0x00007f28adbbef4e in virThreadPoolWorker (opaque=opaque@entry=0x7f28aec88030) at util/virthreadpool.c:145 In stack frame #6 the client->identity object got unref'd, but the code that removes the event callbacks in frame #5 did not run yet as we are trying to obtain the system identity (frames #4, #3, #2). In other thead: #0 virObjectUnref (anyobj=anyobj@entry=0x7f288c162c60) at util/virobject.c:264 klass = 0xdeadbeef obj = 0x7f288c162c60 #1 0x00007f28ae71c709 in remoteRelayDomainEventCheckACL (client=<optimized out>, conn=<optimized out>, dom=dom@entry=0x7f28aecaafc0) at remote.c:164 #2 0x00007f28ae71fc83 in remoteRelayDomainEventTrayChange (conn=<optimized out>, dom=0x7f28aecaafc0, ... ) at remote.c:717 #3 0x00007f28adc04e53 in virDomainEventDispatchDefaultFunc (conn=0x7f287c0009a0, event=0x7f28aecab1a0, ...) at conf/domain_event.c:1455 #4 0x00007f28adc03831 in virObjectEventStateDispatchCallbacks (callbacks=<optimized out>, ....) at conf/object_event.c:724 #5 virObjectEventStateQueueDispatch (callbacks=0x7f288c083730, queue=0x7fff51f90030, state=0x7f288c18da20) at conf/object_event.c:738 #6 virObjectEventStateFlush (state=0x7f288c18da20) at conf/object_event.c:816 #7 virObjectEventTimer (timer=<optimized out>, opaque=0x7f288c18da20) at conf/object_event.c:562 #8 0x00007f28adb859cd in virEventPollDispatchTimeouts () at util/vireventpoll.c:459 Frame #0 is unrefing an invalid identity object while frame #2 hints that the client is still dispatching the event. For untrimmed backtrace see the bugzilla attachment. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1203030 commit 6ca857c7c8a1f7b571132d6c7fff5a06301a5e9a Author: Peter Krempa <pkrempa> Date: Wed Mar 25 08:52:26 2015 +0100 daemon: Clear fake domain def object that is used to check ACL prior to use The fake object is used to pass the domain name and UUID to the ACL code for events where we don't have the full domain def when dispatching events. The rest of the entries would be left uninitialized. While this is not a problem code-wise as the used fields are initialized it looks ugly in the debugger. commit ad886fa6c8ebc321a0386a75c187d315111cf1f3 Author: Peter Krempa <pkrempa> Date: Wed Mar 25 08:25:45 2015 +0100 util: identity: Harden virIdentitySetCurrent() Don't unref the old identity unless we set the new one correctly and unref the new one on failure to set it so that we don't leak any references or use invalid pointers. v1.2.13-308-ga98129c
libvirt-1.2.9.3-1.fc21 has been submitted as an update for Fedora 21. https://admin.fedoraproject.org/updates/libvirt-1.2.9.3-1.fc21
Package libvirt-1.2.9.3-1.fc21: * should fix your issue, * was pushed to the Fedora 21 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing libvirt-1.2.9.3-1.fc21' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2015-7150/libvirt-1.2.9.3-1.fc21 then log in and leave karma (feedback).