Description of problem: Since I've upgraded to Fedora25, I can not connect to libvirtd. How reproducible: Just update from Fedora 23 to Fedora 25 and virt-manager can not connect to libvirtd anymore. Additional info: I know there is a thread here detailing the exactly the same issue: https://www.redhat.com/archives/libvirt-users/2017-June/msg00001.html As I need my virtual machine running, and I did not want to install older Fedora versions, I've tried to delete: qemu-system-x86-2.7.1-6.fc25.x86_64 and installed over an older release: qemu-system-x86-2.4.1-11.fc23.x86_64.rpm It have worked, but I have to create some symbolic links to comply older libraries dependencies.
I've tried to update to Fedora 26 and I've got the same bug: # rpm -qf /usr/sbin/libvirtd libvirt-daemon-3.2.1-3.fc26.x86_64 # rpm -qf /usr/bin/qemu-system-x86_64 qemu-system-x86-core-2.9.0-1.fc26.1.x86_64 # systemctl status libvirtd libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2017-07-16 23:34:59 -03; 19s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 2991 (libvirtd) Tasks: 23 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─2965 /usr/bin/qemu-system-x86_64 -machine accel=kvm -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qem ├─2969 /usr/bin/qemu-system-x86_64 -machine accel=kvm -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qem ├─2991 /usr/sbin/libvirtd ├─3017 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfil └─3019 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfil Jul 16 23:34:59 ygamal.silexonline.org systemd[1]: Starting Virtualization daemon... Jul 16 23:34:59 ygamal.silexonline.org libvirtd[2991]: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory Jul 16 23:34:59 ygamal.silexonline.org libvirtd[2991]: gssapiv2_client_plug_init() failed in sasl_server_add_plugin(): generic failure Jul 16 23:34:59 ygamal.silexonline.org libvirtd[2991]: _sasl_plugin_load failed on sasl_server_plug_init for plugin: gssapiv2 Jul 16 23:34:59 ygamal.silexonline.org systemd[1]: Started Virtualization daemon. Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: info : libvirt version: 3.2.1, package: 3.fc26 (Fedora Project, 2017-05-31-21:32:40, buildhw-04.phx2.fedoraproject.org) Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: info : hostname: ygamal Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: warning : virStorageBackendRefreshLocal:3501 : Ignoring file with control characters under '/' Jul 16 23:35:00 ygamal.silexonline.org libvirtd[2991]: 2017-07-17 02:35:00.360+0000: 3007: warning : virStorageBackendRefreshLocal:3501 : Ignoring file with control characters under '/'
I've found a way to make everything work again. I've compiled a vanilla qemu 2.9.0 and I've overwritten the system's ones, as follows: # git clone git://git.qemu.org/qemu.git # cd qemu # git checkout 359c41abe32638adad503e386969fa428cecff52 # ./configure --target-list=i386-softmmu,x86_64-softmmu,i386-linux-user,x86_64-linux-user --prefix=/opt/qemu --disable-werror # make install # mv /usr/bin/qemu-system-i386 /usr/bin/qemu-system-i386.org # mv /usr/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64.org # ln -sf /opt/qemu/bin/qemu-system-i386 /usr/bin/qemu-system-i386 # ln -sf /opt/qemu/bin/qemu-system-x86_64 /usr/bin/qemu-system-x86_64 I think some patch applied to the original source code is causing this error. Is there anybody reading this bug? Is there any value to report some bug here?
Maybe when we probe qemu on libvirtd startup, qemu is hanging somehow, causing libvirtd to hang and not accept connections. Can you reproduce the hang and provide: sudo virsh list (I just wanna see the error) sudo systemctl status libvirtd ps axwww | grep -e libvirt -e qemu
Here come some results #virsh list it does not produce errors. It freezes. # systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2017-08-06 17:44:20 -03; 4s ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 6766 (libvirtd) Tasks: 22 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─1371 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper ├─1372 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper ├─6766 /usr/sbin/libvirtd ├─6824 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize └─6826 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize Aug 06 17:44:20 ygamal.silexonline.org systemd[1]: Starting Virtualization daemon... Aug 06 17:44:20 ygamal.silexonline.org libvirtd[6766]: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory Aug 06 17:44:20 ygamal.silexonline.org libvirtd[6766]: gssapiv2_client_plug_init() failed in sasl_server_add_plugin(): generic failure Aug 06 17:44:20 ygamal.silexonline.org libvirtd[6766]: _sasl_plugin_load failed on sasl_server_plug_init for plugin: gssapiv2 Aug 06 17:44:20 ygamal.silexonline.org systemd[1]: Started Virtualization daemon. Aug 06 17:44:21 ygamal.silexonline.org dnsmasq[1371]: read /etc/hosts - 2 addresses Aug 06 17:44:21 ygamal.silexonline.org dnsmasq[1371]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Aug 06 17:44:21 ygamal.silexonline.org dnsmasq-dhcp[1371]: read /var/lib/libvirt/dnsmasq/default.hostsfile # ps axwww | grep -e libvirt -e qemu 1371 ? S 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper 1372 ? S 0:00 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper 6766 ? Ssl 0:00 /usr/sbin/libvirtd 6824 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize 6826 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile -daemonize
> 1371 ? S 0:00 /usr/sbin/dnsmasq > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > --dhcp-script=/usr/libexec/libvirt_leaseshelper > 1372 ? S 0:00 /usr/sbin/dnsmasq > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > --dhcp-script=/usr/libexec/libvirt_leaseshelper > 6766 ? Ssl 0:00 /usr/sbin/libvirtd > 6824 ? Sl 0:00 /usr/bin/qemu-system-i386 -S -no-user-config > -nodefaults -nographic -machine none,accel=kvm:tcg -qmp > unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile > /var/lib/libvirt/qemu/capabilities.pidfile -daemonize > 6826 ? S 0:00 /usr/bin/qemu-system-i386 -S -no-user-config > -nodefaults -nographic -machine none,accel=kvm:tcg -qmp > unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile > /var/lib/libvirt/qemu/capabilities.pidfile -daemonize Okay thanks. So maybe those qemu processes are stuck. Can you do 'sudo pstack $PID' for eacho qemu process, and the libvirtd process, and post the output
[root@ygamal ~]# systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2017-08-07 20:16:28 -03; 7min ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 967 (libvirtd) Tasks: 20 (limit: 32768) CGroup: /system.slice/libvirtd.service ├─ 967 /usr/sbin/libvirtd ├─1087 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile └─1098 /usr/bin/qemu-system-i386 -S -no-user-config -nodefaults -nographic -machine none,accel=kvm:tcg -qmp unix:/var/lib/libvirt/qemu/capabilities.monitor.sock,server,nowait -pidfile /var/lib/libvirt/qemu/capabilities.pidfile Aug 07 20:16:27 ygamal.silexonline.org systemd[1]: Starting Virtualization daemon... Aug 07 20:16:28 ygamal.silexonline.org libvirtd[967]: Could not find keytab file: /etc/libvirt/krb5.tab: No such file or directory Aug 07 20:16:28 ygamal.silexonline.org libvirtd[967]: gssapiv2_client_plug_init() failed in sasl_server_add_plugin(): generic failure Aug 07 20:16:28 ygamal.silexonline.org libvirtd[967]: _sasl_plugin_load failed on sasl_server_plug_init for plugin: gssapiv2 Aug 07 20:16:28 ygamal.silexonline.org systemd[1]: Started Virtualization daemon. [root@ygamal ~]# pstack 967 Thread 17 (Thread 0x7f862a0c7700 (LWP 1053)): #0 0x00007f865a7ffa9d in poll () from /lib64/libc.so.6 #1 0x00007f865e86d553 in poll (__timeout=-1, __nfds=<optimized out>, __fds=0x7f862a0c62b0) at /usr/include/bits/poll2.h:41 #2 virCommandProcessIO (cmd=cmd@entry=0x7f8624145a70) at util/vircommand.c:2063 #3 0x00007f865e87186f in virCommandRun (cmd=0x7f8624145a70, exitstatus=exitstatus@entry=0x7f862a0c6824) at util/vircommand.c:2288 #4 0x00007f862e82c00a in virQEMUCapsInitQMPCommandRun (cmd=cmd@entry=0x7f8624185f80, forceTCG=forceTCG@entry=false) at qemu/qemu_capabilities.c:4914 #5 0x00007f862e83242b in virQEMUCapsInitQMP (qmperr=0x7f862a0c68c0, runGid=107, runUid=107, libDir=0x7f86241a3c40 "/var/lib/libvirt/qemu", qemuCaps=0x7f86240f0130) at qemu/qemu_capabilities.c:4970 #6 virQEMUCapsNewForBinaryInternal (caps=caps@entry=0x7f86241a2bb0, binary=binary@entry=0x7f862400ed80 "/usr/bin/qemu-system-i386", libDir=0x7f86241a3c40 "/var/lib/libvirt/qemu", cacheDir=0x7f86241b1130 "/var/cache/libvirt/qemu", runUid=107, runGid=107, qmpOnly=false) at qemu/qemu_capabilities.c:5065 #7 0x00007f862e83356f in virQEMUCapsNewForBinary (runGid=<optimized out>, runUid=<optimized out>, cacheDir=<optimized out>, libDir=<optimized out>, binary=0x7f862400ed80 "/usr/bin/qemu-system-i386", caps=0x7f86241a2bb0) at qemu/qemu_capabilities.c:5110 #8 virQEMUCapsCacheValidate (cache=0x7f86241a4590, binary=0x7f862400ed80 "/usr/bin/qemu-system-i386", caps=0x7f86241a2bb0, qemuCaps=0x7f862a0c6aa0) at qemu/qemu_capabilities.c:5261 #9 0x00007f862e833690 in virQEMUCapsCacheLookup (caps=caps@entry=0x7f86241a2bb0, cache=cache@entry=0x7f86241a4590, binary=0x7f862400ed80 "/usr/bin/qemu-system-i386") at qemu/qemu_capabilities.c:5291 #10 0x00007f862e833947 in virQEMUCapsInitGuest (guestarch=VIR_ARCH_I686, hostarch=VIR_ARCH_X86_64, cache=0x7f86241a4590, caps=0x7f86241a2bb0) at qemu/qemu_capabilities.c:906 #11 virQEMUCapsInit (cache=0x7f86241a4590) at qemu/qemu_capabilities.c:1140 #12 0x00007f862e877fff in virQEMUDriverCreateCapabilities (driver=driver@entry=0x7f862415c070) at qemu/qemu_conf.c:927 #13 0x00007f862e8bc2c1 in qemuStateInitialize (privileged=true, callback=<optimized out>, opaque=<optimized out>) at qemu/qemu_driver.c:844 #14 0x00007f865e977005 in virStateInitialize (privileged=true, callback=0x55e95d85a320 <daemonInhibitCallback>, opaque=0x55e95f63d7e0) at libvirt.c:770 #15 0x000055e95d85a37b in daemonRunStateInit (opaque=0x55e95f63d7e0) at libvirtd.c:881 #16 0x00007f865e8d32b2 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #17 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #18 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f862beca700 (LWP 1051)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671650) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f862c6cb700 (LWP 1050)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671890) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f862cecc700 (LWP 1049)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671650) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f862d6cd700 (LWP 1048)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671890) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f862dece700 (LWP 1047)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f671548, m=m@entry=0x55e95f671520) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f671650) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f864785f700 (LWP 1022)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f8648060700 (LWP 1021)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f8648861700 (LWP 1020)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f8649062700 (LWP 1019)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f8649863700 (LWP 1018)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c768, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d3f58 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f864a064700 (LWP 1017)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f6488a0) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f864a865700 (LWP 1016)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f864b066700 (LWP 1015)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f864b867700 (LWP 1014)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648980) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f864c068700 (LWP 1013)): #0 0x00007f865aad981b in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f865e8d34e6 in virCondWait (c=c@entry=0x55e95f63c6c8, m=m@entry=0x55e95f63c6a0) at util/virthread.c:154 #2 0x00007f865e8d4023 in virThreadPoolWorker (opaque=opaque@entry=0x55e95f648ab0) at util/virthreadpool.c:124 #3 0x00007f865e8d3288 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f865aad336d in start_thread () from /lib64/libpthread.so.0 #5 0x00007f865a80bb8f in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f865f5041c0 (LWP 967)): #0 0x00007f865a7ffa9d in poll () from /lib64/libc.so.6 #1 0x00007f865e8805f1 in poll (__timeout=-1, __nfds=6, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 virEventPollRunOnce () at util/vireventpoll.c:641 #3 0x00007f865e87f171 in virEventRunDefaultImpl () at util/virevent.c:314 #4 0x00007f865e9eaeb5 in virNetDaemonRun (dmn=0x55e95f63d7e0) at rpc/virnetdaemon.c:819 #5 0x000055e95d85916d in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1541 root@ygamal ~]# pstack 1087 Thread 2 (Thread 0x7f442bcf4700 (LWP 1097)): #0 0x00007f443a9c6529 in syscall () at /lib64/libc.so.6 #1 0x000055ec8e409f05 in qemu_event_wait () #2 0x000055ec8e419d1e in call_rcu_thread () #3 0x00007f443ac9336d in start_thread () at /lib64/libpthread.so.0 #4 0x00007f443a9cbb8f in clone () at /lib64/libc.so.6 Thread 1 (Thread 0x7f4444117dc0 (LWP 1087)): #0 0x00007f443ac9d19d in read () at /lib64/libpthread.so.0 #1 0x000055ec8e1b20b5 in os_daemonize () #2 0x000055ec8e08d11d in main () [root@ygamal ~]# pstack 1098 #0 0x00007f443bda077c in base::internal::SpinLockDelay(int volatile*, int, int) () at /lib64/libtcmalloc.so.4 #1 0x00007f443bda0683 in SpinLock::SlowLock() () at /lib64/libtcmalloc.so.4 #2 0x00007f443bd91642 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () at /lib64/libtcmalloc.so.4 #3 0x00007f443bd943da in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int) () at /lib64/libtcmalloc.so.4 #4 0x00007f44358880a9 in () at /usr/lib64/nvidia-304xx/libGL.so.1 #5 0x00007f442ed29874 in () at /usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.135 #6 0x00007f442eeeaf30 in () at /usr/lib64/nvidia-304xx/libnvidia-glcore.so.304.135 #7 0x00007f4435868581 in () at /usr/lib64/nvidia-304xx/libGL.so.1 #8 0x00007f44358686c4 in () at /usr/lib64/nvidia-304xx/libGL.so.1 #9 0x00007f4435868e4f in () at /usr/lib64/nvidia-304xx/libGL.so.1 #10 0x00007f443a98f5c2 in fork () at /lib64/libc.so.6 #11 0x000055ec8e1b2008 in os_daemonize () #12 0x000055ec8e08d11d in main ()
the prescence of nvidia drivers in the qemu stack trace is concerning. can you uninstall the drivers, reboot, and see if things still hang? if so, post the pstack of qemu again without the drivers installed
Something very strange happened. I have removed nvidia drivers. Everything worked ok. I installed nvidia drivers again. Now all is working well again! I can not reproduce the problem anymore. Definitively, nvidia drivers is the problem key.
Interesting. Well if you hit it again please reopen