Bug 853381 - libvirt 0.10.0-1.fc19 daemon segfaults on startup in netlink_init
libvirt 0.10.0-1.fc19 daemon segfaults on startup in netlink_init
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: netcf (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Laine Stump
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-31 06:01 EDT by Richard W.M. Jones
Modified: 2012-09-30 21:11 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-30 21:11:58 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
core file (xz compressed) (566.90 KB, application/octet-stream)
2012-08-31 06:04 EDT, Richard W.M. Jones
no flags Details

  None (edit)
Description Richard W.M. Jones 2012-08-31 06:01:43 EDT
Description of problem:

sudo /usr/sbin/libvirtd ... segfaults

The stack trace is below.  Note that it's thread #1 which
is segfaulting.

Version-Release number of selected component (if applicable):

libvirt-0.10.0-1.fc19.x86_64

How reproducible:

100%

Steps to Reproduce:
1. Run /usr/sbin/libvirtd

Additional info:

Thread 12 (Thread 0x7f2794166700 (LWP 13517)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f2792162700 (LWP 13521)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f2793164700 (LWP 13519)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f2794967700 (LWP 13516)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f2791961700 (LWP 13522)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f279c76f840 (LWP 13512)):
#0  0x00000030d0ae8e8d in poll () from /lib64/libc.so.6
#1  0x00007f279cc238fb in poll (__timeout=-1, __nfds=6, __fds=<optimized out>)
    at /usr/include/bits/poll2.h:46
#2  virEventPollRunOnce () at util/event_poll.c:615
#3  0x00007f279cc22687 in virEventRunDefaultImpl () at util/event.c:247
#4  0x00007f279cd0804d in virNetServerRun (srv=0x22f02b0)
    at rpc/virnetserver.c:751
#5  0x000000000040bebd in main (argc=<optimized out>, argv=<optimized out>)
    at libvirtd.c:1332

Thread 6 (Thread 0x7f2795168700 (LWP 13515)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f2792963700 (LWP 13520)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f2795969700 (LWP 13514)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f279616a700 (LWP 13513)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f0420, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e7b in virThreadPoolWorker (opaque=opaque@entry=0x22e50e0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f2793965700 (LWP 13518)):
#0  0x00000030d0e0b5e5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f279cc33a36 in virCondWait (c=c@entry=0x22f04b8, 
    m=m@entry=0x22f03f8) at util/threads-pthread.c:117
#2  0x00007f279cc33e9b in virThreadPoolWorker (opaque=opaque@entry=0x22e4fe0)
    at util/threadpool.c:103
#3  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#4  0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030d0af196d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f278c5b2700 (LWP 13523)):
#0  0x00000030fdc09fef in netlink_init (ncf=0x8413ed60840086b0, 
    ncf@entry=0x7f27840086b0) at dutil_linux.c:715
#1  0x00000030fdc05fa1 in drv_init (ncf=0x7f27840086b0)
    at drv_initscripts.c:372
#2  0x00000030fdc044e1 in ncf_init (ncf=<optimized out>, root=<optimized out>, 
    root@entry=0x0) at netcf.c:75
#3  0x00007f278f1c76fa in interfaceOpenInterface (conn=0x7f278407dd30, 
    auth=<optimized out>, flags=<optimized out>)
    at interface/netcf_driver.c:140
#4  0x00007f279ccaaeae in do_open (
    name=name@entry=0x7f278ed7d48a "qemu:///system", auth=auth@entry=0x0, 
    flags=flags@entry=0) at libvirt.c:1212
#5  0x00007f279ccaca14 in virConnectOpen (name=0x7f278ed7d48a "qemu:///system")
    at libvirt.c:1332
#6  0x00007f278ed3ea0e in qemudStartup (privileged=<optimized out>)
    at qemu/qemu_driver.c:825
#7  0x00007f279ccac73b in virStateInitialize (privileged=1) at libvirt.c:796
#8  0x000000000040d3c1 in daemonRunStateInit (opaque=opaque@entry=0x22f02b0)
    at libvirtd.c:759
#9  0x00007f279cc33869 in virThreadHelper (data=<optimized out>)
    at util/threads-pthread.c:161
#10 0x00000030d0e07d15 in start_thread () from /lib64/libpthread.so.0
#11 0x00000030d0af196d in clone () from /lib64/libc.so.6
Comment 1 Richard W.M. Jones 2012-08-31 06:04:48 EDT
Created attachment 608469 [details]
core file (xz compressed)
Comment 2 Daniel Veillard 2012-08-31 06:19:35 EDT
That part which is crashing is inside netcf:

#0  0x00000030fdc09fef in netlink_init (ncf=0x8413ed60840086b0, 
    ncf@entry=0x7f27840086b0) at dutil_linux.c:715
#1  0x00000030fdc05fa1 in drv_init (ncf=0x7f27840086b0)
    at drv_initscripts.c:372
#2  0x00000030fdc044e1 in ncf_init (ncf=<optimized out>, root=<optimized out>, 
    root@entry=0x0) at netcf.c:75

reasigning to the component, but please tell which version of the netcf package
you use

Daniel
Comment 3 Richard W.M. Jones 2012-08-31 06:21:04 EDT
This was fixed by updating to netcf-0.2.2-1.fc19.
Comment 4 Laine Stump 2012-09-04 10:12:10 EDT
Yes, but the intent was that libvirt-0.10.0+ would require the newer netcf. However, when I updated the libvirt specfile, there was only an explicit BuildRequires of netcf-devel (which I changed to BuildRequires: netcf-devel >= 0.2.2"). I forgot that the install-time Requires: is implied by libvirt linking to libnetcf (i.e. there is no explicit "Requires: netcf-libs"), and since the internal version number of the library didn't change (because there was no change in API), the new netcf wasn't pulled in as a prerequisite for the upgrade of libvirt.

In order to avoid this situation completely, I think that means we need to add an explicit:

   Requires: netcf-libs >= 0.2.2

to libvirt.spec for F18+ and RHEL7+. The alternative is that the few people who update libvirt by itself (rather than updating the entire system) will encounter this same problem.
Comment 5 Laine Stump 2012-09-06 15:08:04 EDT
Although I appreciate Rich's leniancy in declaring this as NOTABUG, it still shouldn't happen, and can be prevented.

I've pushed a patch upstream which, when applied to the F18 specfile, will prevent this crash, instead either a) forcing a simultaneous update of netcf to 0.2.2, or b) failing that, outputting an error and refusing to upgrade libvirt. Please add it to the next F18 build of libvirt:

commit 89810fc42360adf0e76507e0f8051f2ab2b116a3
Author: Laine Stump <laine@laine.org>
Date:   Tue Sep 4 13:05:54 2012 -0400

    build: require netcf-0.2.2 when installing on Fedora18+
    
    A previous patch forced libnl-3 and netcf-0.2.2 (which itself requires
    libnl-3) when *building* for Fedora 18+ (and RHEL 7+), but the
    install-time Requires: for netcf has always been implicit due to
    libvirtd linking with libnetcf.so. However, the since the API of netcf
    didn't change when it was rebuilt to use libnl-3, the internal library
    version didn't change either, making it possible (from rpm's point of
    view) to upgrade libvirt without upgrading netcf (in reality, that
    leads to a segfault - see
    https://bugzilla.redhat.com/show_bug.cgi?id=853381).
    
    The solution is to put an explicit Requires: line in libvirt's
    specfile for fedora >= 18 and rhel >= 7.
Comment 6 Laine Stump 2012-09-06 16:43:05 EDT
Another note here - the F18 repos are currently frozen due to a pending alpha release, so although netcf-0.2.2-1 has been built and is available in f18-updates-testing, it's not being pushed to stable until [some unspecified time in the future]. Fortunately, it appears that the alpha will have libvirt-0.10.0rc0 (which is built with libnl1) and netcf-0.2.1 (also built with libnl1), and hopefully both a new libvirt and a new netcf will be queued up for push to stable as soon as the alpha freeze is listed.
Comment 7 Cole Robinson 2012-09-30 21:11:58 EDT
Pretty sure this is all resolved now in F18 and rawhide.

Note You need to log in before you can comment on or make changes to this bug.