Bug 1464601 - libvirtd is not accepting connections
libvirtd is not accepting connections
Status: CLOSED DEFERRED
Product: Fedora
Classification: Fedora
Component: libvirt (Show other bugs)
26
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Libvirt Maintainers
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-23 17:32 EDT by Haroldo Gamal
Modified: 2017-08-09 09:21 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-08-09 09:21:18 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Haroldo Gamal 2017-06-23 17:32:06 EDT
Description of problem:

Since I've upgraded to Fedora25, I can not connect to libvirtd.

How reproducible:

Just update from Fedora 23 to Fedora 25 and virt-manager can not connect to libvirtd anymore.

Additional info:

I know there is a thread here detailing the exactly the same issue:

    https://www.redhat.com/archives/libvirt-users/2017-June/msg00001.html

As I need my virtual machine running, and I did not want to install older Fedora versions, I've tried to delete:

    qemu-system-x86-2.7.1-6.fc25.x86_64

and installed over an older release:

    qemu-system-x86-2.4.1-11.fc23.x86_64.rpm

It have worked, but I have to create some symbolic links to comply older libraries dependencies.
Comment 1 Haroldo Gamal 2017-07-16 22:38:30 EDT
I've tried to update to Fedora 26 and I've got the same bug:
# rpm -qf /usr/sbin/libvirtd
libvirt-daemon-3.2.1-3.fc26.x86_64

# rpm -qf /usr/bin/qemu-system-x86_64
qemu-system-x86-core-2.9.0-1.fc26.1.x86_64

# systemctl status libvirtd
 libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-07-16 23:34:59 -03; 19s ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 2991 (libvirtd)
    Tasks: 23 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─2965 /usr/bin/qemu-system-x86_64 -machine accel=kvm -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qem
           ├─2969 /usr/bin/qemu-system-x86_64 -machine accel=kvm -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qem
           ├─2991 /usr/sbin/libvirtd
           ├─3017 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfil
           └─3019 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfil

Jul 16 23:34:59 ygamal.silexonline.org systemd[1]: Starting Virtualization daemon...
Jul 16 23:34:59 ygamal.silexonline.org libvirtd[2991]: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory
Jul 16 23:34:59 ygamal.silexonline.org libvirtd[2991]: gssapiv2_client_plug_init() failed in sasl_server_add_plugin(): generic failure
Jul 16 23:34:59 ygamal.silexonline.org libvirtd[2991]: _sasl_plugin_load failed on sasl_server_plug_init for plugin: gssapiv2
Jul 16 23:34:59 ygamal.silexonline.org systemd[1]: Started Virtualization daemon.
Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: info : libvirt version: 3.2.1, package: 3.fc26 (Fedora Project, 2017-05-31-21:32:40, buildhw-04.phx2.fedoraproject.org)
Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: info : hostname: ygamal
Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: warning : virStorageBackendRefreshLocal:3501 : Ignoring file with control characters under '/'
Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: warning : virStorageBackendRefreshLocal:3501 : Ignoring file with control characters under '/'
Comment 2 Haroldo Gamal 2017-07-29 21:04:11 EDT
I've found a way to make everything work again. I've compiled a vanilla qemu 2.9.0 and I've overwritten the system's ones, as follows:

# git clone git://git.qemu.org/qemu.git
# cd qemu
# git checkout 359c41abe32638adad503e386969fa428cecff52
# ./configure --target-list=i386-softmmu,x86_64-softmmu,i386-linux-user,x86_64-linux-user --prefix=/opt/qemu --disable-werror
# make install
# mv /usr/bin/qemu-system-i386 /usr/bin/qemu-system-i386.org
# mv /usr/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64.org
# ln -sf /opt/qemu/bin/qemu-system-i386 /usr/bin/qemu-system-i386
# ln -sf /opt/qemu/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64

I think some patch applied to the original source code is causing this error.

Is there anybody reading this bug? Is there any value to report some bug here?
Comment 3 Cole Robinson 2017-08-06 16:19:38 EDT
Maybe when we probe qemu on libvirtd startup, qemu is hanging somehow, causing libvirtd to hang and not accept connections. Can you reproduce the hang and provide:

sudo virsh list   (I just wanna see the error)
sudo systemctl status libvirtd
ps axwww | grep -e libvirt -e qemu
Comment 4 Haroldo Gamal 2017-08-06 16:48:50 EDT
Here come some results

#virsh list

it does not produce errors. It freezes.

# systemctl status libvirtd

● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-08-06 17:44:20 -03; 4s ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 6766 (libvirtd)
    Tasks: 22 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─1371 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ├─1372 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           ├─6766 /usr/sbin/libvirtd
           ├─6824 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
           └─6826 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize

Aug 06 17:44:20 ygamal.silexonline.org systemd[1]: Starting Virtualization daemon...
Aug 06 17:44:20 ygamal.silexonline.org libvirtd[6766]: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory
Aug 06 17:44:20 ygamal.silexonline.org libvirtd[6766]: gssapiv2_client_plug_init() failed in sasl_server_add_plugin(): generic failure
Aug 06 17:44:20 ygamal.silexonline.org libvirtd[6766]: _sasl_plugin_load failed on sasl_server_plug_init for plugin: gssapiv2
Aug 06 17:44:20 ygamal.silexonline.org systemd[1]: Started Virtualization daemon.
Aug 06 17:44:21 ygamal.silexonline.org dnsmasq[1371]: read /etc/hosts - 2 addresses
Aug 06 17:44:21 ygamal.silexonline.org dnsmasq[1371]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Aug 06 17:44:21 ygamal.silexonline.org dnsmasq-dhcp[1371]: read /var/lib/libvirt/dnsmasq/default.hostsfile

# ps axwww | grep -e libvirt -e qemu

 1371 ?        S      0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
 1372 ?        S      0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
 6766 ?        Ssl    0:00 /usr/sbin/libvirtd
 6824 ?        Sl     0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
 6826 ?        S      0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
Comment 5 Cole Robinson 2017-08-06 16:55:39 EDT
>  1371 ?        S      0:00 /usr/sbin/dnsmasq
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> --dhcp-script=/usr/libexec/libvirt_leaseshelper
>  1372 ?        S      0:00 /usr/sbin/dnsmasq
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
> --dhcp-script=/usr/libexec/libvirt_leaseshelper
>  6766 ?        Ssl    0:00 /usr/sbin/libvirtd
>  6824 ?        Sl     0:00 /usr/bin/qemu-system-i386 -S -no-user-config
> -nodefaults -nographic -machine none,accel=kvm:tcg -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
> /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
>  6826 ?        S      0:00 /usr/bin/qemu-system-i386 -S -no-user-config
> -nodefaults -nographic -machine none,accel=kvm:tcg -qmp
> unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile
> /var/lib/libvirt/qemu/capabilities.pidfile -daemonize

Okay thanks. So maybe those qemu processes are stuck. Can you do 'sudo pstack $PID' for eacho qemu process, and the libvirtd process, and post the output
Comment 6 Haroldo Gamal 2017-08-07 19:28:03 EDT
[root@ygamal ~]# systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-08-07 20:16:28 -03; 7min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 967 (libvirtd)
    Tasks: 20 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─ 967 /usr/sbin/libvirtd
           ├─1087 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile
           └─1098 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile

Aug 07 20:16:27 ygamal.silexonline.org systemd[1]: Starting Virtualization daemon...
Aug 07 20:16:28 ygamal.silexonline.org libvirtd[967]: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory
Aug 07 20:16:28 ygamal.silexonline.org libvirtd[967]: gssapiv2_client_plug_init() failed in sasl_server_add_plugin(): generic failure
Aug 07 20:16:28 ygamal.silexonline.org libvirtd[967]: _sasl_plugin_load failed on sasl_server_plug_init for plugin: gssapiv2
Aug 07 20:16:28 ygamal.silexonline.org systemd[1]: Started Virtualization daemon.
[root@ygamal ~]# pstack 967
Thread 17 (Thread 0x7f862a0c7700 (LWP 1053)):
#0  0x00007f865a7ffa9d in poll () from /lib64/libc.so.6
#1  0x00007f865e86d553 in poll (__timeout=-1, __nfds=<optimized out>, __fds=0x7f862a0c62b0) at /usr/include/bits/poll2.h:41
#2  virCommandProcessIO (cmd=cmd@entry=0x7f8624145a70) at util/vircommand.c:2063
#3  0x00007f865e87186f in virCommandRun (cmd=0x7f8624145a70, exitstatus=exitstatus@entry=0x7f862a0c6824) at util/vircommand.c:2288
#4  0x00007f862e82c00a in virQEMUCapsInitQMPCommandRun (cmd=cmd@entry=0x7f8624185f80, forceTCG=forceTCG@entry=false) at qemu/qemu_capabilities.c:4914
#5  0x00007f862e83242b in virQEMUCapsInitQMP (qmperr=0x7f862a0c68c0, runGid=107, runUid=107, libDir=0x7f86241a3c40 "/var/lib/libvirt/qemu", qemuCaps=0x7f86240f0130) at qemu/qemu_capabilities.c:4970
#6  virQEMUCapsNewForBinaryInternal (caps=caps@entry=0x7f86241a2bb0, binary=binary@entry=0x7f862400ed80 "/usr/bin/qemu-system-i386", libDir=0x7f86241a3c40 "/var/lib/libvirt/qemu", cacheDir=0x7f86241b1130 "/var/cache/libvirt/qemu", runUid=107, runGid=107, qmpOnly=false) at qemu/qemu_capabilities.c:5065
#7  0x00007f862e83356f in virQEMUCapsNewForBinary (runGid=<optimized out>, runUid=<optimized out>, cacheDir=<optimized out>, libDir=<optimized out>, binary=0x7f862400ed80 "/usr/bin/qemu-system-i386", caps=0x7f86241a2bb0) at qemu/qemu_capabilities.c:5110
#8  virQEMUCapsCacheValidate (cache=0x7f86241a4590, binary=0x7f862400ed80 "/usr/bin/qemu-system-i386", caps=0x7f86241a2bb0, qemuCaps=0x7f862a0c6aa0) at qemu/qemu_capabilities.c:5261
#9  0x00007f862e833690 in virQEMUCapsCacheLookup (caps=caps@entry=0x7f86241a2bb0, cache=cache@entry=0x7f86241a4590, binary=0x7f862400ed80 "/usr/bin/qemu-system-i386") at qemu/qemu_capabilities.c:5291
#10 0x00007f862e833947 in virQEMUCapsInitGuest (guestarch=VIR_ARCH_I686, hostarch=VIR_ARCH_X86_64, cache=0x7f86241a4590, caps=0x7f86241a2bb0) at qemu/qemu_capabilities.c:906
#11 virQEMUCapsInit (cache=0x7f86241a4590) at qemu/qemu_capabilities.c:1140
#12 0x00007f862e877fff in virQEMUDriverCreateCapabilities (driver=driver@entry=0x7f862415c070) at qemu/qemu_conf.c:927
#13 0x00007f862e8bc2c1 in qemuStateInitialize (privileged=true, callback=<optimized out>, opaque=<optimized out>) at qemu/qemu_driver.c:844
#14 0x00007f865e977005 in virStateInitialize (privileged=true, callback=0x55e95d85a320 <daemonInhibitCallback>, opaque=0x55e95f63d7e0) at libvirt.c:770
#15 0x000055e95d85a37b in daemonRunStateInit (opaque=0x55e95f63d7e0) at libvirtd.c:881
#16 0x00007f865e8d32b2 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#17 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#18 0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f862beca700 (LWP 1051)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671650) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f862c6cb700 (LWP 1050)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671890) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f862cecc700 (LWP 1049)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671650) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f862d6cd700 (LWP 1048)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671890) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f862dece700 (LWP 1047)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671650) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f864785f700 (LWP 1022)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f8648060700 (LWP 1021)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f8648861700 (LWP 1020)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f8649062700 (LWP 1019)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f8649863700 (LWP 1018)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f864a064700 (LWP 1017)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f6488a0) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f864a865700 (LWP 1016)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f864b066700 (LWP 1015)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f864b867700 (LWP 1014)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f864c068700 (LWP 1013)):
#0  0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154
#2  0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124
#3  0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#4  0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0
#5  0x00007f865a80bb8f in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f865f5041c0 (LWP 967)):
#0  0x00007f865a7ffa9d in poll () from /lib64/libc.so.6
#1  0x00007f865e8805f1 in poll (__timeout=-1, __nfds=6, __fds=<optimized out>) at /usr/include/bits/poll2.h:46
#2  virEventPollRunOnce () at util/vireventpoll.c:641
#3  0x00007f865e87f171 in virEventRunDefaultImpl () at util/virevent.c:314
#4  0x00007f865e9eaeb5 in virNetDaemonRun (dmn=0x55e95f63d7e0) at rpc/virnetdaemon.c:819
#5  0x000055e95d85916d in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1541

root@ygamal ~]# pstack 1087
Thread 2 (Thread 0x7f442bcf4700 (LWP 1097)):
#0  0x00007f443a9c6529 in syscall () at /lib64/libc.so.6
#1  0x000055ec8e409f05 in qemu_event_wait ()
#2  0x000055ec8e419d1e in call_rcu_thread ()
#3  0x00007f443ac9336d in start_thread () at /lib64/libpthread.so.0
#4  0x00007f443a9cbb8f in clone () at /lib64/libc.so.6
Thread 1 (Thread 0x7f4444117dc0 (LWP 1087)):
#0  0x00007f443ac9d19d in read () at /lib64/libpthread.so.0
#1  0x000055ec8e1b20b5 in os_daemonize ()
#2  0x000055ec8e08d11d in main ()

[root@ygamal ~]# pstack 1098
#0  0x00007f443bda077c in base::internal::SpinLockDelay(int volatile*, int, int) () at /lib64/libtcmalloc.so.4
#1  0x00007f443bda0683 in SpinLock::SlowLock() () at /lib64/libtcmalloc.so.4
#2  0x00007f443bd91642 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () at /lib64/libtcmalloc.so.4
#3  0x00007f443bd943da in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int) () at /lib64/libtcmalloc.so.4
#4  0x00007f44358880a9 in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#5  0x00007f442ed29874 in  () at /usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.135
#6  0x00007f442eeeaf30 in  () at /usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.135
#7  0x00007f4435868581 in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#8  0x00007f44358686c4 in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#9  0x00007f4435868e4f in  () at /usr/lib64/nvidia-304xx/libGL.so.1
#10 0x00007f443a98f5c2 in fork () at /lib64/libc.so.6
#11 0x000055ec8e1b2008 in os_daemonize ()
#12 0x000055ec8e08d11d in main ()
Comment 7 Cole Robinson 2017-08-07 19:31:15 EDT
the prescence of nvidia drivers in the qemu stack trace is concerning. can you uninstall the drivers, reboot, and see if things still hang? if so, post the pstack of qemu again without the drivers installed
Comment 8 Haroldo Gamal 2017-08-08 22:24:58 EDT
Something very strange happened.

I have removed nvidia drivers. Everything worked ok.

I installed nvidia drivers again. Now all is working well again! I can not reproduce the problem anymore.

Definitively, nvidia drivers is the problem key.
Comment 9 Cole Robinson 2017-08-09 09:21:18 EDT
Interesting. Well if you hit it again please reopen

Note You need to log in before you can comment on or make changes to this bug.