Description of problem: sudo /usr/sbin/libvirtd ... segfaults The stack trace is below. Note that it's thread #1 which is segfaulting. Version-Release number of selected component (if applicable): libvirt-0.10.0-1.fc19.x86_64 How reproducible: 100% Steps to Reproduce: 1. Run /usr/sbin/libvirtd Additional info: Thread 12 (Thread 0x7f2794166700 (LWP 13517)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f2792162700 (LWP 13521)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f2793164700 (LWP 13519)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f2794967700 (LWP 13516)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f2791961700 (LWP 13522)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f279c76f840 (LWP 13512)): #0 0x00000030d0ae8e8d in poll () from /lib64/libc.so.6 #1 0x00007f279cc238fb in poll (__timeout=-1, __nfds=6, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 virEventPollRunOnce () at util/event_poll.c:615 #3 0x00007f279cc22687 in virEventRunDefaultImpl () at util/event.c:247 #4 0x00007f279cd0804d in virNetServerRun (srv=0x22f02b0) at rpc/virnetserver.c:751 #5 0x000000000040bebd in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1332 Thread 6 (Thread 0x7f2795168700 (LWP 13515)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f2792963700 (LWP 13520)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f2795969700 (LWP 13514)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f279616a700 (LWP 13513)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f2793965700 (LWP 13518)): #0 0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, m=m@entry=0x22f03f8) at util/threads-pthread.c:117 #2 0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0) at util/threadpool.c:103 #3 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #4 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #5 0x00000030d0af196d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f278c5b2700 (LWP 13523)): #0 0x00000030fdc09fef in netlink_init (ncf=0x8413ed60840086b0, ncf@entry=0x7f27840086b0) at dutil_linux.c:715 #1 0x00000030fdc05fa1 in drv_init (ncf=0x7f27840086b0) at drv_initscripts.c:372 #2 0x00000030fdc044e1 in ncf_init (ncf=<optimized out>, root=<optimized out>, root@entry=0x0) at netcf.c:75 #3 0x00007f278f1c76fa in interfaceOpenInterface (conn=0x7f278407dd30, auth=<optimized out>, flags=<optimized out>) at interface/netcf_driver.c:140 #4 0x00007f279ccaaeae in do_open ( name=name@entry=0x7f278ed7d48a "qemu:///system", auth=auth@entry=0x0, flags=flags@entry=0) at libvirt.c:1212 #5 0x00007f279ccaca14 in virConnectOpen (name=0x7f278ed7d48a "qemu:///system") at libvirt.c:1332 #6 0x00007f278ed3ea0e in qemudStartup (privileged=<optimized out>) at qemu/qemu_driver.c:825 #7 0x00007f279ccac73b in virStateInitialize (privileged=1) at libvirt.c:796 #8 0x000000000040d3c1 in daemonRunStateInit (opaque=opaque@entry=0x22f02b0) at libvirtd.c:759 #9 0x00007f279cc33869 in virThreadHelper (data=<optimized out>) at util/threads-pthread.c:161 #10 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0 #11 0x00000030d0af196d in clone () from /lib64/libc.so.6
Created attachment 608469 [details] core file (xz compressed)
That part which is crashing is inside netcf: #0 0x00000030fdc09fef in netlink_init (ncf=0x8413ed60840086b0, ncf@entry=0x7f27840086b0) at dutil_linux.c:715 #1 0x00000030fdc05fa1 in drv_init (ncf=0x7f27840086b0) at drv_initscripts.c:372 #2 0x00000030fdc044e1 in ncf_init (ncf=<optimized out>, root=<optimized out>, root@entry=0x0) at netcf.c:75 reasigning to the component, but please tell which version of the netcf package you use Daniel
This was fixed by updating to netcf-0.2.2-1.fc19.
Yes, but the intent was that libvirt-0.10.0+ would require the newer netcf. However, when I updated the libvirt specfile, there was only an explicit BuildRequires of netcf-devel (which I changed to BuildRequires: netcf-devel >= 0.2.2"). I forgot that the install-time Requires: is implied by libvirt linking to libnetcf (i.e. there is no explicit "Requires: netcf-libs"), and since the internal version number of the library didn't change (because there was no change in API), the new netcf wasn't pulled in as a prerequisite for the upgrade of libvirt. In order to avoid this situation completely, I think that means we need to add an explicit: Requires: netcf-libs >= 0.2.2 to libvirt.spec for F18+ and RHEL7+. The alternative is that the few people who update libvirt by itself (rather than updating the entire system) will encounter this same problem.
Although I appreciate Rich's leniancy in declaring this as NOTABUG, it still shouldn't happen, and can be prevented. I've pushed a patch upstream which, when applied to the F18 specfile, will prevent this crash, instead either a) forcing a simultaneous update of netcf to 0.2.2, or b) failing that, outputting an error and refusing to upgrade libvirt. Please add it to the next F18 build of libvirt: commit 89810fc42360adf0e76507e0f8051f2ab2b116a3 Author: Laine Stump <laine> Date: Tue Sep 4 13:05:54 2012 -0400 build: require netcf-0.2.2 when installing on Fedora18+ A previous patch forced libnl-3 and netcf-0.2.2 (which itself requires libnl-3) when *building* for Fedora 18+ (and RHEL 7+), but the install-time Requires: for netcf has always been implicit due to libvirtd linking with libnetcf.so. However, the since the API of netcf didn't change when it was rebuilt to use libnl-3, the internal library version didn't change either, making it possible (from rpm's point of view) to upgrade libvirt without upgrading netcf (in reality, that leads to a segfault - see https://bugzilla.redhat.com/show_bug.cgi?id=853381). The solution is to put an explicit Requires: line in libvirt's specfile for fedora >= 18 and rhel >= 7.
Another note here - the F18 repos are currently frozen due to a pending alpha release, so although netcf-0.2.2-1 has been built and is available in f18-updates-testing, it's not being pushed to stable until [some unspecified time in the future]. Fortunately, it appears that the alpha will have libvirt-0.10.0rc0 (which is built with libnl1) and netcf-0.2.1 (also built with libnl1), and hopefully both a new libvirt and a new netcf will be queued up for push to stable as soon as the alpha freeze is listed.
Pretty sure this is all resolved now in F18 and rawhide.