Bug 1014933 - [abrt] libvirt-daemon-1.0.5.6-2.fc19: virQEMUCloseCallbacksGetForConn: Process /usr/sbin/libvirtd was killed by signal 11 (SIGSEGV)
[abrt] libvirt-daemon-1.0.5.6-2.fc19: virQEMUCloseCallbacksGetForConn: Proces...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
19
i686 Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Libvirt Maintainers
Fedora Extras Quality Assurance
abrt_hash:189c9c690641a1ea509dd083b91...
:
: 995720 1014644 1014842 1015292 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-03 02:50 EDT by Thomas H. Jethan
Modified: 2013-10-29 05:16 EDT (History)
16 users (show)

See Also:
Fixed In Version: libvirt-1.0.5.6-3.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1015108 (view as bug list)
Environment:
Last Closed: 2013-10-11 19:57:45 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
File: backtrace (38.73 KB, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: cgroup (163 bytes, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: core_backtrace (26.76 KB, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: dso_list (8.84 KB, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: environ (117 bytes, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: limits (1.29 KB, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: maps (26.47 KB, text/plain)
2013-10-03 02:50 EDT, Thomas H. Jethan
no flags Details
File: open_fds (764 bytes, text/plain)
2013-10-03 02:51 EDT, Thomas H. Jethan
no flags Details
File: proc_pid_status (752 bytes, text/plain)
2013-10-03 02:51 EDT, Thomas H. Jethan
no flags Details
File: var_log_messages (13.04 KB, text/plain)
2013-10-03 02:51 EDT, Thomas H. Jethan
no flags Details

  None (edit)
Description Thomas H. Jethan 2013-10-03 02:50:00 EDT
Version-Release number of selected component:
libvirt-daemon-1.0.5.6-2.fc19

Additional info:
reporter:       libreport-2.1.7
backtrace_rating: 4
cmdline:        /usr/sbin/libvirtd
crash_function: virQEMUCloseCallbacksGetForConn
executable:     /usr/sbin/libvirtd
kernel:         3.11.2-201.fc19.i686.PAE
runlevel:       N 5
type:           CCpp
uid:            0

Truncated backtrace:
Thread no. 1 (10 frames)
 #0 virQEMUCloseCallbacksGetForConn at qemu/qemu_conf.c:861
 #1 virQEMUCloseCallbacksRun at qemu/qemu_conf.c:890
 #2 qemuConnectClose at qemu/qemu_driver.c:1057
 #3 virConnectDispose at datatypes.c:159
 #4 virObjectUnref at util/virobject.c:264
 #5 virConnectClose at libvirt.c:1503
 #6 nwfilterStateReload at nwfilter/nwfilter_driver.c:301
 #7 nwfilterFirewalldDBusFilter at nwfilter/nwfilter_driver.c:90
 #9 virDBusWatchCallback at util/virdbus.c:144
 #10 virEventPollDispatchHandles at util/vireventpoll.c:500
Comment 1 Thomas H. Jethan 2013-10-03 02:50:10 EDT
Created attachment 806882 [details]
File: backtrace
Comment 2 Thomas H. Jethan 2013-10-03 02:50:17 EDT
Created attachment 806883 [details]
File: cgroup
Comment 3 Thomas H. Jethan 2013-10-03 02:50:25 EDT
Created attachment 806884 [details]
File: core_backtrace
Comment 4 Thomas H. Jethan 2013-10-03 02:50:33 EDT
Created attachment 806885 [details]
File: dso_list
Comment 5 Thomas H. Jethan 2013-10-03 02:50:40 EDT
Created attachment 806886 [details]
File: environ
Comment 6 Thomas H. Jethan 2013-10-03 02:50:47 EDT
Created attachment 806887 [details]
File: limits
Comment 7 Thomas H. Jethan 2013-10-03 02:50:56 EDT
Created attachment 806888 [details]
File: maps
Comment 8 Thomas H. Jethan 2013-10-03 02:51:03 EDT
Created attachment 806889 [details]
File: open_fds
Comment 9 Thomas H. Jethan 2013-10-03 02:51:11 EDT
Created attachment 806890 [details]
File: proc_pid_status
Comment 10 Thomas H. Jethan 2013-10-03 02:51:20 EDT
Created attachment 806891 [details]
File: var_log_messages
Comment 11 Daniel Berrange 2013-10-03 05:56:52 EDT
Oh this is a really fun bug, and by 'fun' I mean insanely obscure.

In one thread we have the libvirtd daemon startup code running, and its in the middle of QEMU state initialization.

#9  0xb00882e4 in qemuStateInitialize (privileged=true, callback=0xb77a0420 <daemonInhibitCallback>, opaque=0xb8b1fc98) at qemu/qemu_driver.c:595
        driverConf = 0xaf5afcd8 "/etc/libvirt/qemu.conf"
        conn = 0x0
        ebuf = "\000\260\025\267\024\071P\257\214\000\000\000\360\316\341\257\335\242\023\267\214\000\000\000\210\177X\257\001\000\000\000l\000\000\000\360\316\341\257\000\260\025\267\264\316\341\257\210\177X\257$\316\341\257$\316\341\257l\000\000\000\304\316\341\257\201\321LRl\000\000\000\235R\022\267\000)\233\351\260\316\341\257\000\000\000\000\253G\022\267\000\260\025\267\340\316\341\257\a\000\000\000\v\260\023\267\000\260\025\267\001\000\000\000\254\325\334\266\000\260\025\267\214\261\023\267 :P\257\037:P\257\000\000\000\000/\261\023\267\000\260\025\267uc\334\266\000\260\025\267A\262\023\267\037:P\257\000\000\000\000\001\000\000\000\000\000\000\000\340\316\341\257\334\316\341\257\001\000\000\000\001\000\000\000\033c\024\267"...
        membase = 0x0
        mempath = 0x0
        cfg = 0xaf509050
        run_uid = 4294967295
        run_gid = 4294967295
        __func__ = "qemuStateInitialize"
        __FUNCTION__ = "qemuStateInitialize"
#10 0xb74c5325 in virStateInitialize (privileged=true, callback=callback@entry=0xb77a0420 <daemonInhibitCallback>, opaque=opaque@entry=0xb8b1fc98) at libvirt.c:833
        i = 6
        __func__ = "virStateInitialize"
#11 0xb77a049e in daemonRunStateInit (opaque=opaque@entry=0xb8b1fc98) at libvirtd.c:876
        srv = 0xb8b1fc98
        __func__ = "daemonRunStateInit"



In another thread, we have a dbus event being handled by the nwfilter driver, and the nwfilter driver calls into the QEMU driver....which has not finished initializing itself yet!
 

Thread 1 (Thread 0xb6366ac0 (LWP 7041)):
#0  0xb0052861 in virQEMUCloseCallbacksGetForConn (closeCallbacks=0x0, conn=0xb8b2cc20) at qemu/qemu_conf.c:861
        list = 0xb8ac57e8
        data = {conn = 0xb8b2cc20, list = 0xb8ac57e8, oom = false}
#1  virQEMUCloseCallbacksRun (closeCallbacks=0x0, conn=conn@entry=0xb8b2cc20, driver=0xaf50b350) at qemu/qemu_conf.c:890
        list = 0xb8b2cc20
        i = <optimized out>
        __func__ = "virQEMUCloseCallbacksRun"
#2  0xb009df3b in qemuConnectClose (conn=0xb8b2cc20) at qemu/qemu_driver.c:1057
        driver = <optimized out>
#3  0xb74babc1 in virConnectDispose (obj=0xb8b2cc20) at datatypes.c:159
        conn = 0xb8b2cc20
#4  0xb742f22c in virObjectUnref (anyobj=anyobj@entry=0xb8b2cc20) at util/virobject.c:264
        klass = 0xb8b2cba0
        obj = 0xb8b2cc20
        lastRef = true
        __func__ = "virObjectUnref"
#5  0xb74c5811 in virConnectClose (conn=conn@entry=0xb8b2cc20) at libvirt.c:1503
        __func__ = "virConnectClose"
        __FUNCTION__ = "virConnectClose"
#6  0xb023424e in nwfilterStateReload () at nwfilter/nwfilter_driver.c:301
        conn = 0xb8b2cc20
#7  0xb02342fc in nwfilterFirewalldDBusFilter (connection=0xaf501038, message=0xaf503910, user_data=0x0) at nwfilter/nwfilter_driver.c:90
        __func__ = "nwfilterFirewalldDBusFilter"
#8  0xb711efb9 in dbus_connection_dispatch (connection=0xaf501038) at dbus-connection.c:4631
        filter = <optimized out>
        next = 0x0
        message = 0xaf503910
        link = <optimized out>
        filter_list_copy = 0xaf5009dc
        message_link = 0xaf500a18
        result = DBUS_HANDLER_RESULT_NOT_YET_HANDLED
        pending = <optimized out>
        reply_serial = <optimized out>
        status = <optimized out>
        found_object = 3071507249
        __FUNCTION__ = "dbus_connection_dispatch"
#9  0xb740caeb in virDBusWatchCallback (fdatch=fdatch@entry=8, fd=15, events=1, opaque=0xaf500ca8) at util/virdbus.c:144
        watch = 0xaf500ca8
        info = 0xaf500de0
        dbus_flags = 1



This DBus event is triggered when the firewalld driver is reloaded, or restarted.

Thus what I think is happening here is that both libvirtd and firewalld have just been upgraded by yum. WHile libvirtd is finishing its restart, the firewalld reload occurs, causing the crash.
Comment 12 Daniel Berrange 2013-10-03 05:57:13 EDT
*** Bug 1014842 has been marked as a duplicate of this bug. ***
Comment 13 Daniel Berrange 2013-10-03 05:57:24 EDT
*** Bug 1014644 has been marked as a duplicate of this bug. ***
Comment 14 Daniel Berrange 2013-10-03 06:06:18 EDT
Yep, there were two updates for firewalld and libvirt which hit the repo at approx the same time


https://admin.fedoraproject.org/updates/FEDORA-2013-18032/firewalld-0.3.5-1.fc19
https://admin.fedoraproject.org/updates/FEDORA-2013-17618/libvirt-1.0.5.6-2.fc19


Can someone check their yum logs to see if they happened to have firewalld & libvirt updated on the same day. eg look at

# grep -E '(libvirt|firewalld)' /var/log/yum.log  | grep Installed
Comment 15 Dave Allan 2013-10-03 08:50:02 EDT
(In reply to Daniel Berrange from comment #14)
> Can someone check their yum logs to see if they happened to have firewalld &
> libvirt updated on the same day. eg look at

I didn't happen to hit this crash, but yes the libvirt and firewalld updates hit my system in the same yum update this morning.
Comment 16 Daniel Berrange 2013-10-03 09:10:03 EDT
This series fixes the problem

https://www.redhat.com/archives/libvir-list/2013-October/msg00154.html
Comment 17 Cole Robinson 2013-10-03 14:41:41 EDT
*** Bug 995720 has been marked as a duplicate of this bug. ***
Comment 18 dq_vii 2013-10-03 15:54:15 EDT
I dont rember what caused the issue all i rember is my system crashing from it 

reporter:       libreport-2.1.7
backtrace_rating: 4
cmdline:        /usr/sbin/libvirtd
crash_function: virQEMUCloseCallbacksGetForConn
executable:     /usr/sbin/libvirtd
kernel:         3.11.2-201.fc19.x86_64
package:        libvirt-daemon-1.0.5.6-2.fc19
reason:         Process /usr/sbin/libvirtd was killed by signal 11 (SIGSEGV)
runlevel:       N 5
type:           CCpp
uid:            0
Comment 19 Eric Blake 2013-10-03 18:27:53 EDT
*** Bug 1015292 has been marked as a duplicate of this bug. ***
Comment 20 Fedora Update System 2013-10-06 15:18:48 EDT
libvirt-1.0.5.6-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/libvirt-1.0.5.6-3.fc19
Comment 21 Fedora Update System 2013-10-08 07:36:44 EDT
Package libvirt-1.0.5.6-3.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing libvirt-1.0.5.6-3.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2013-18500/libvirt-1.0.5.6-3.fc19
then log in and leave karma (feedback).
Comment 22 Fedora Update System 2013-10-11 19:57:45 EDT
libvirt-1.0.5.6-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.