Bug 1320084 - Libvirtd hangs/crashes when guest is set to have host-passthrough CPU
Summary: Libvirtd hangs/crashes when guest is set to have host-passthrough CPU
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Luyao Huang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-03-22 09:54 UTC by Asim
Modified: 2017-02-13 15:10 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-02-13 15:10:40 UTC
Target Upstream Version:

Attachments (Terms of Use)
virshlistdebuglogs (10.61 KB, text/plain)
2016-05-26 06:07 UTC, Asim 		no flags Details
libvirtdlogs (31.98 KB, text/plain)
2016-05-26 06:08 UTC, Asim 		no flags Details
gdblog (31.66 KB, text/plain)
2016-05-26 06:08 UTC, Asim 		no flags Details
logs shown in gdb (27.60 KB, text/plain)
2016-06-28 03:52 UTC, Asim 		no flags Details
virsh List logs (10.64 KB, text/plain)
2016-06-28 03:53 UTC, Asim 		no flags Details
GDB Logs (50.08 KB, text/plain)
2016-09-27 07:35 UTC, Asim 		no flags Details
Logs as per : LIBVIRT_DEBUG=debug LIBVIRT_LOG_FILTERS="1:remote" LIBVIRT_LOG_OUTPUTS="3:syslog:libvirtd 1:file:/tmp/libvirt.log" virsh list (18.51 KB, text/plain)
2016-09-27 07:36 UTC, Asim 		no flags Details
View All Add an attachment (proposed patch, testcase, etc.)

Description Asim 2016-03-22 09:54:37 UTC 
Description of problem:

For qemu-kvm guests setting CPU mode as host-passthrough on a Centos 7.2 host node crashes libvirt service.

On checking libvirt logs "/var/log/libvirt/qemu/guestfs-zzzqomm5duk949a2.log" found Domain id=218657 is tainted: host-cpu
qemu: terminating on signal 15 from pid 15710

Need to restart libvirt to fix the issue.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.Create a KVM guest with CPU mode=host-passthrough
2.Randomly libvirt hangs
Comment 2 Jaroslav Suchanek 2016-03-23 16:48:35 UTC 
Asim, would you mind providing following info?
1) debug logs, how to turn on the debug logging is described for example here
   http://libvirt.org/guide/html/Application_Development_Guide-Connections-Debug.html
2) description of installed libvirt packages
3) configuration of the guest
4) steps how to reliably reproduce the problem
5) in case of crash, please provide the crash dump or full backtrace

Thanks.
Comment 3 Asim 2016-03-28 12:12:24 UTC 
Hi Jaroslav Suchanek,

>>1) debug logs, how to turn on the debug logging is described for example here

I will do it once libvirt fails once again as I have already started it by running "service libvirtd restart" which fixes the issue temporarily.



>>2) description of installed libvirt packages

-CentOS Linux release 7.2.1511 (Core)
-libvirt-1.2.17-13.el7_2.3.x86_64
-qemu-kvm-1.5.3-105.el7_2.3.x86_64



>>3) configuration of the guest

Specs of one of my guests 
- 256 MB RAM 
- 1 vcpu
- CPU mode host-passthrough
- 20 GB (File based QCOW2 Storage)
- Running Ubuntu 14.04-x86_64



>>4) steps how to reliably reproduce the problem

I dont how to reproduce it, just happens randomly.



>>5) in case of crash, please provide the crash dump or full backtrace


guestfs logs :

Domain id=218657 is tainted: custom-argv
Domain id=218657 is tainted: host-cpu
qemu: terminating on signal 15 from pid 15710
2016-02-12 11:03:14.410+0000: shutting down

v1334.log:

[root@qemu]# tail -f v1334.log
2016-03-08 04:06:25.500+0000: shutting down
2016-03-08 04:06:35.888+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name v1334 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu host -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1b956fce-bc90-44be-bf24-8586d403eeb9 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/v1334.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot order=c,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vps/v1334-0x5o-1tulxrlbiyzcbyhl.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:3e:db:ea:4b,bus=pci.0,addr=0x3 -device usb-tablet,id=input0 -vnc 107.191.112.131:15,password -k en-us -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
Domain id=320207 is tainted: host-cpu

Hope this helps you to find out the cause.
Comment 4 Jaroslav Suchanek 2016-04-29 12:28:22 UTC 
Hi Asim, have you experienced any further crashes? If there are no crash dumps/backtraces available or debug logs from the time when crash occurred I would close this.

Thanks.
Comment 5 Jaroslav Suchanek 2016-05-23 11:29:49 UTC 
Since there is no response for almost a month I am closing this for insufficient data. Please reopen this if you the problem reappear. Thanks.
Comment 6 Asim 2016-05-24 03:42:49 UTC 
Hi,

Apologies for replying late.
Yes, I was able to reproduce it couple of times.

Here is the debug info log.


[root@CentOS-71-64-minimal ~]# LIBVIRT_DEBUG=debug virsh list
2016-05-24 03:34:13.235+0000: 9361: info : libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org)
2016-05-24 03:34:13.235+0000: 9361: debug : virGlobalInit:389 : register drivers
2016-05-24 03:34:13.235+0000: 9361: debug : virRegisterConnectDriver:691 : driver=0x7f7390b49640 name=Test
2016-05-24 03:34:13.235+0000: 9361: debug : virRegisterConnectDriver:702 : registering Test as driver 0
2016-05-24 03:34:13.235+0000: 9361: debug : virRegisterConnectDriver:691 : driver=0x7f7390b4b260 name=ESX
2016-05-24 03:34:13.235+0000: 9361: debug : virRegisterConnectDriver:702 : registering ESX as driver 1
2016-05-24 03:34:13.235+0000: 9361: debug : virRegisterConnectDriver:691 : driver=0x7f7390b4a0c0 name=remote
2016-05-24 03:34:13.235+0000: 9361: debug : virRegisterConnectDriver:702 : registering remote as driver 2
2016-05-24 03:34:13.235+0000: 9361: debug : virEventRegisterDefaultImpl:261 : registering default event implementation
2016-05-24 03:34:13.235+0000: 9361: debug : virEventPollAddHandle:115 : Used 0 handle slots, adding at least 10 more
2016-05-24 03:34:13.235+0000: 9361: debug : virEventPollInterruptLocked:723 : Skip interrupt, 0 0
2016-05-24 03:34:13.235+0000: 9361: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f7390688810 opaque=(nil) ff=(nil)
2016-05-24 03:34:13.235+0000: 9361: debug : virEventRegisterImpl:231 : addHandle=0x7f7390688f70 updateHandle=0x7f73906892d0 removeHandle=0x7f7390688870 addTimeout=0x7f73906889f0 updateTimeout=0x7f7390688c10 removeTimeout=0x7f7390688de0
2016-05-24 03:34:13.235+0000: 9361: debug : virEventPollAddTimeout:230 : Used 0 timeout slots, adding at least 10 more
2016-05-24 03:34:13.235+0000: 9361: debug : virEventPollInterruptLocked:723 : Skip interrupt, 0 0
2016-05-24 03:34:13.235+0000: 9361: info : virEventPollAddTimeout:253 : EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x7f7390d96c10 opaque=0x7ffdbd71d2d0 ff=(nil)
2016-05-24 03:34:13.235+0000: 9361: debug : virConnectOpenAuth:1339 : name=<null>, auth=0x7f7390b49400, flags=0
2016-05-24 03:34:13.235+0000: 9362: debug : virThreadJobSet:99 : Thread 9362 is now running job vshEventLoop
2016-05-24 03:34:13.235+0000: 9361: info : virObjectNew:202 : OBJECT_NEW: obj=0x7f7392df5820 classname=virConnect
2016-05-24 03:34:13.235+0000: 9362: debug : virEventRunDefaultImpl:305 : running default event implementation
2016-05-24 03:34:13.235+0000: 9361: info : virObjectNew:202 : OBJECT_NEW: obj=0x7f7392df6f20 classname=virConnectCloseCallbackData
2016-05-24 03:34:13.235+0000: 9362: debug : virEventPollCleanupTimeouts:526 : Cleanup 1
2016-05-24 03:34:13.235+0000: 9362: debug : virEventPollCleanupHandles:575 : Cleanup 1
2016-05-24 03:34:13.235+0000: 9361: debug : virConnectGetConfigFile:953 : Loading config file '/etc/libvirt/libvirt.conf'
2016-05-24 03:34:13.235+0000: 9362: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0
2016-05-24 03:34:13.235+0000: 9362: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2016-05-24 03:34:13.235+0000: 9361: debug : virConfReadFile:769 : filename=/etc/libvirt/libvirt.conf
2016-05-24 03:34:13.235+0000: 9362: debug : virEventPollCalculateTimeout:371 : No timeout is pending
2016-05-24 03:34:13.235+0000: 9362: info : virEventPollRunOnce:641 : EVENT_POLL_RUN: nhandles=1 timeout=-1
2016-05-24 03:34:13.235+0000: 9361: debug : virFileClose:102 : Closed fd 6
2016-05-24 03:34:13.235+0000: 9361: debug : do_open:1136 : no name, allowing driver auto-select
2016-05-24 03:34:13.235+0000: 9361: debug : do_open:1179 : trying driver 0 (Test) ...
2016-05-24 03:34:13.235+0000: 9361: debug : do_open:1194 : driver 0 Test returned DECLINED
2016-05-24 03:34:13.235+0000: 9361: debug : do_open:1179 : trying driver 1 (ESX) ...
2016-05-24 03:34:13.235+0000: 9361: debug : do_open:1194 : driver 1 ESX returned DECLINED
2016-05-24 03:34:13.235+0000: 9361: debug : do_open:1179 : trying driver 2 (remote) ...
2016-05-24 03:34:13.235+0000: 9361: debug : remoteConnectOpen:1191 : Auto-probe remote URI
2016-05-24 03:34:13.235+0000: 9361: debug : doRemoteOpen:792 : proceeding with name =
2016-05-24 03:34:13.235+0000: 9361: debug : doRemoteOpen:801 : Connecting with transport 1
2016-05-24 03:34:13.235+0000: 9361: debug : doRemoteOpen:893 : Proceeding with sockname /var/run/libvirt/libvirt-sock


It gets stuck here and does not proceed further.

Also there are more then 25827 entries in /var/log/libvirt/qemu/ directory.

One of the latest guestfs log output is given below :

[root@CentOS-71-64-minimal ~]# tail -n 1000 /var/log/libvirt/qemu/guestfs-zzz2bku55a3mnxb6.log
2015-11-19 05:01:02.952+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none TMPDIR=/var/tmp /usr/libexec/qemu-kvm -name guestfs-zzz2bku55a3mnxb6 -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu host -m 500 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 1ff59628-aa21-4344-9cd8-39bf4791e554 -nographic -no-user-config -nodefaults -device sga -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/guestfs-zzz2bku55a3mnxb6.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/.guestfs-0/appliance.d/kernel -initrd /var/tmp/.guestfs-0/appliance.d/initrd -append panic=1 console=ttyS0 udevtimeout=6000 udev.event-timeout=6000 no_timer_check acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 TERM=linux -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x3 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/tmp/libguestfsZzyB5d/overlay1,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/tmp/libguestfsZzyB5d/overlay2,if=none,id=drive-scsi0-0-1-0,format=qcow2,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 -chardev socket,id=charserial0,path=/tmp/libguestfsZzyB5d/console.sock -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/tmp/libguestfsZzyB5d/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
Domain id=12195 is tainted: custom-argv
Domain id=12195 is tainted: host-cpu
qemu: terminating on signal 15 from pid 6040
2015-11-19 05:01:11.414+0000: shutting down
Comment 7 Jiri Denemark 2016-05-24 13:58:17 UTC 
There's still no sign of any crash, which makes it quite unclear what the problem is.

It seems it could be that virsh list hangs when trying to connect to libvirtd, is that right?

If so, we'd need debug logs from libvirtd (see http://wiki.libvirt.org/page/DebugLogs) in addition to log from virsh.

Since libvirtd might be stuck somewhere, we'd also need a backtrace of all libvirtd threads at the point virsh gets stuck connecting to the daemon. To get the backtrace follow the steps below:

1) gdb -p `pidof libvirtd`
2) thread apply all backtrace
3) quit

If prints something about missing debug symbols and shows lots of question marks instead of function names, please follow the advice from gdb and install missing debuginfo packages. Then repeat the steps above.

Once you have the data, please add them to this bug as attachments.
Comment 8 Asim 2016-05-25 04:37:56 UTC 
Hi,

>>It seems it could be that virsh list hangs when trying to connect to libvirtd, is that right?

Yes, exactly.


>>http://wiki.libvirt.org/page/DebugLogs

The link is not working.


>>Since libvirtd might be stuck somewhere, we'd also need a backtrace of all libvirtd threads at the point virsh gets stuck connecting to the daemon. To get the backtrace follow the steps below:

1) gdb -p `pidof libvirtd`
2) thread apply all backtrace
3) quit

Here is the output :


[root@CentOS-71-64-minimal ~]# gdb -p 26584
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 26584
Reading symbols from /usr/sbin/libvirtd...Reading symbols from /usr/sbin/libvirtd...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Reading symbols from /lib64/libvirt-admin.so.0...Reading symbols from /lib64/libvirt-admin.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libvirt-admin.so.0
Reading symbols from /lib64/libvirt-lxc.so.0...Reading symbols from /lib64/libvirt-lxc.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libvirt-lxc.so.0
Reading symbols from /lib64/libvirt-qemu.so.0...Reading symbols from /lib64/libvirt-qemu.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libvirt-qemu.so.0
Reading symbols from /lib64/libvirt.so.0...Reading symbols from /lib64/libvirt.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libvirt.so.0
Reading symbols from /lib64/libcap-ng.so.0...Reading symbols from /lib64/libcap-ng.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap-ng.so.0
Reading symbols from /lib64/libyajl.so.2...Reading symbols from /lib64/libyajl.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libyajl.so.2
Reading symbols from /lib64/libnl-route-3.so.200...Reading symbols from /lib64/libnl-route-3.so.200...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnl-route-3.so.200
Reading symbols from /lib64/libnl-3.so.200...Reading symbols from /lib64/libnl-3.so.200...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnl-3.so.200
Reading symbols from /lib64/libaudit.so.1...Reading symbols from /lib64/libaudit.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libaudit.so.1
Reading symbols from /lib64/libdevmapper.so.1.02...Reading symbols from /lib64/libdevmapper.so.1.02...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdevmapper.so.1.02
Reading symbols from /lib64/libnuma.so.1...Reading symbols from /lib64/libnuma.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnuma.so.1
Reading symbols from /lib64/libsystemd.so.0...Reading symbols from /lib64/libsystemd.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsystemd.so.0
Reading symbols from /lib64/libxml2.so.2...Reading symbols from /lib64/libxml2.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libxml2.so.2
Reading symbols from /lib64/libavahi-common.so.3...Reading symbols from /lib64/libavahi-common.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libavahi-common.so.3
Reading symbols from /lib64/libavahi-client.so.3...Reading symbols from /lib64/libavahi-client.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libavahi-client.so.3
Reading symbols from /lib64/libdbus-1.so.3...Reading symbols from /lib64/libdbus-1.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdbus-1.so.3
Reading symbols from /lib64/libgnutls.so.28...Reading symbols from /lib64/libgnutls.so.28...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgnutls.so.28
Reading symbols from /lib64/libsasl2.so.3...Reading symbols from /lib64/libsasl2.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsasl2.so.3
Reading symbols from /lib64/libcurl.so.4...Reading symbols from /lib64/libcurl.so.4...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcurl.so.4
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[New LWP 26626]
[New LWP 26624]
[New LWP 26623]
[New LWP 26622]
[New LWP 26621]
[New LWP 26620]
[New LWP 26595]
[New LWP 26594]
[New LWP 26593]
[New LWP 26592]
[New LWP 26591]
[New LWP 26590]
[New LWP 26589]
[New LWP 26588]
[New LWP 26587]
[New LWP 26586]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libsepol.so.1...Reading symbols from /lib64/libsepol.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsepol.so.1
Reading symbols from /lib64/libudev.so.1...Reading symbols from /lib64/libudev.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libudev.so.1
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libcap.so.2...Reading symbols from /lib64/libcap.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcap.so.2
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /lib64/libgcrypt.so.11...Reading symbols from /lib64/libgcrypt.so.11...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcrypt.so.11
Reading symbols from /lib64/libgpg-error.so.0...Reading symbols from /lib64/libgpg-error.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgpg-error.so.0
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libdw.so.1...Reading symbols from /lib64/libdw.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdw.so.1
Reading symbols from /lib64/libz.so.1...Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libp11-kit.so.0...Reading symbols from /lib64/libp11-kit.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libp11-kit.so.0
Reading symbols from /lib64/libtspi.so.1...Reading symbols from /lib64/libtspi.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtspi.so.1
Reading symbols from /lib64/libtasn1.so.6...Reading symbols from /lib64/libtasn1.so.6...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtasn1.so.6
Reading symbols from /lib64/libnettle.so.4...Reading symbols from /lib64/libnettle.so.4...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnettle.so.4
Reading symbols from /lib64/libhogweed.so.2...Reading symbols from /lib64/libhogweed.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libhogweed.so.2
Reading symbols from /lib64/libgmp.so.10...Reading symbols from /lib64/libgmp.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgmp.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libidn.so.11...Reading symbols from /lib64/libidn.so.11...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libidn.so.11
Reading symbols from /lib64/libssh2.so.1...Reading symbols from /lib64/libssh2.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssh2.so.1
Reading symbols from /lib64/libssl3.so...Reading symbols from /lib64/libssl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl3.so
Reading symbols from /lib64/libsmime3.so...Reading symbols from /lib64/libsmime3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libsmime3.so
Reading symbols from /lib64/libnss3.so...Reading symbols from /lib64/libnss3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss3.so
Reading symbols from /lib64/libnssutil3.so...Reading symbols from /lib64/libnssutil3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/liblber-2.4.so.2...Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libldap-2.4.so.2...Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libattr.so.1...Reading symbols from /lib64/libattr.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libattr.so.1
Reading symbols from /lib64/libelf.so.1...Reading symbols from /lib64/libelf.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libelf.so.1
Reading symbols from /lib64/libbz2.so.1...Reading symbols from /lib64/libbz2.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libbz2.so.1
Reading symbols from /lib64/libffi.so.6...Reading symbols from /lib64/libffi.so.6...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libffi.so.6
Reading symbols from /lib64/libcrypto.so.10...Reading symbols from /lib64/libcrypto.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypto.so.10
Reading symbols from /lib64/libssl.so.10...Reading symbols from /lib64/libssl.so.10...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libssl.so.10
Reading symbols from /lib64/libfreebl3.so...Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_interface.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_interface.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_interface.so
Reading symbols from /lib64/libnetcf.so.1...Reading symbols from /lib64/libnetcf.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libnetcf.so.1
Reading symbols from /lib64/libaugeas.so.0...Reading symbols from /lib64/libaugeas.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libaugeas.so.0
Reading symbols from /lib64/libexslt.so.0...Reading symbols from /lib64/libexslt.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libexslt.so.0
Reading symbols from /lib64/libxslt.so.1...Reading symbols from /lib64/libxslt.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libxslt.so.1
Reading symbols from /lib64/libfa.so.1...Reading symbols from /lib64/libfa.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfa.so.1
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_storage.so
Reading symbols from /lib64/libblkid.so.1...Reading symbols from /lib64/libblkid.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libblkid.so.1
Reading symbols from /lib64/librbd.so.1...Reading symbols from /lib64/librbd.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/librbd.so.1
Reading symbols from /lib64/librados.so.2...Reading symbols from /lib64/librados.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/librados.so.2
Reading symbols from /lib64/libacl.so.1...Reading symbols from /lib64/libacl.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libacl.so.1
Reading symbols from /lib64/libgfapi.so.0...Reading symbols from /lib64/libgfapi.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgfapi.so.0
Reading symbols from /lib64/libglusterfs.so.0...Reading symbols from /lib64/libglusterfs.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libglusterfs.so.0
Reading symbols from /lib64/libgfrpc.so.0...Reading symbols from /lib64/libgfrpc.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgfrpc.so.0
Reading symbols from /lib64/libgfxdr.so.0...Reading symbols from /lib64/libgfxdr.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libgfxdr.so.0
Reading symbols from /lib64/libuuid.so.1...Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libboost_thread-mt.so.1.53.0...Reading symbols from /lib64/libboost_thread-mt.so.1.53.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libboost_thread-mt.so.1.53.0
Reading symbols from /lib64/libboost_system-mt.so.1.53.0...Reading symbols from /lib64/libboost_system-mt.so.1.53.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libboost_system-mt.so.1.53.0
Reading symbols from /lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libstdc++.so.6
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so
Reading symbols from /lib64/libpciaccess.so.0...Reading symbols from /lib64/libpciaccess.so.0...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpciaccess.so.0
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_secret.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_secret.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_secret.so
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so
Reading symbols from /lib64/libpcap.so.1...Reading symbols from /lib64/libpcap.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcap.so.1
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so...Reading symbols from /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so
Reading symbols from /lib64/libfuse.so.2...Reading symbols from /lib64/libfuse.so.2...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libfuse.so.2
Reading symbols from /usr/lib64/sasl2/libsasldb.so...Reading symbols from /usr/lib64/sasl2/libsasldb.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libsasldb.so
Reading symbols from /lib64/libdb-5.3.so...Reading symbols from /lib64/libdb-5.3.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libdb-5.3.so
Reading symbols from /usr/lib64/sasl2/libanonymous.so...Reading symbols from /usr/lib64/sasl2/libanonymous.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libanonymous.so
Reading symbols from /usr/lib64/sasl2/libcrammd5.so...Reading symbols from /usr/lib64/sasl2/libcrammd5.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libcrammd5.so
Reading symbols from /usr/lib64/sasl2/libdigestmd5.so...Reading symbols from /usr/lib64/sasl2/libdigestmd5.so...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/sasl2/libdigestmd5.so
0x00007f68f806669d in poll () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install libvirt-daemon-1.2.17-13.el7_2.4.x86_64
(gdb) thread apply all backtrace

Thread 17 (Thread 0x7f68eb6ba700 (LWP 26586)):
#0  0x00007f68f83476d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f68facd9186 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f68facd9a8b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f68facd8f18 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f68f8343dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f68f8070ced in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f68eaeb9700 (LWP 26587)):
#0  0x00007f68f83476d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f68facd9186 in virCondWait () from /lib64/libvirt.so.0
#2  0x00007f68facd9a8b in virThreadPoolWorker () from /lib64/libvirt.so.0
#3  0x00007f68facd8f18 in virThreadHelper () from /lib64/libvirt.so.0
#4  0x00007f68f8343dc5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f68f8070ced in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f68ea6b8700 (LWP 26588)):
#0  0x00007f68f83476d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00007f68facd9186 in virCondWait () from /lib64/libv
Comment 9 Jiri Denemark 2016-05-25 14:21:37 UTC 
(In reply to Asim from comment #8)
> > http://wiki.libvirt.org/page/DebugLogs
> 
> The link is not working.

Hmm, it might have been a temporary issue. It worked for me at that time and
it works now too.

> Here is the output :
> 
> 
> [root@CentOS-71-64-minimal ~]# gdb -p 26584
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7

Attachment please, pasting this directly in a comment makes this bug very hard
to follow.

> Thread 15 (Thread 0x7f68ea6b8700 (LWP 26588)):
> #0  0x00007f68f83476d5 in pthread_cond_wait@@GLIBC_2.3.2 ()
>    from /lib64/libpthread.so.0
> #1  0x00007f68facd9186 in virCondWait () from /lib64/libv

It seems the backtrace is even truncated.

And please, provide all data (backtrace, libvirtd log, and virsh log) at once
showing the same instance of the issue.
Comment 10 Asim 2016-05-26 06:07:32 UTC 
Created attachment 1161729 [details]
virshlistdebuglogs
Comment 11 Asim 2016-05-26 06:08:21 UTC 
Created attachment 1161730 [details]
libvirtdlogs
Comment 12 Asim 2016-05-26 06:08:50 UTC 
Created attachment 1161731 [details]
gdblog
Comment 13 Asim 2016-05-26 06:10:10 UTC 
Hi,

> > http://wiki.libvirt.org/page/DebugLogs
> 
> The link is not working.

yes the link is working now.

Please let me know if you need anything else.
Comment 14 Jiri Denemark 2016-05-26 08:12:03 UTC 
So the reason why virsh cannot connect to libvirtd is that the daemon has not fully start yet.

It is stuck in Thread 2 (Thread 0x7f0b8f1e5700 (LWP 4988)):
#0  0x00007f0bcce2669d in poll () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#1  0x00007f0bcfa45562 in virCommandProcessIO () from /lib64/libvirt.so.0
#2  0x00007f0bcfa49b32 in virCommandRun () from /lib64/libvirt.so.0
#3  0x00007f0bcfa5d7b6 in virFirewallCheckUpdateLock ()
   from /lib64/libvirt.so.0
#4  0x00007f0bcfa5dbb3 in virFirewallValidateBackend ()
   from /lib64/libvirt.so.0
#5  0x00007f0bcfa5dd0f in virFirewallOnce () from /lib64/libvirt.so.0
#6  0x00007f0bcd108bb0 in pthread_once () from /lib64/libpthread.so.0
#7  0x00007f0bcfa5d747 in virFirewallInitialize () from /lib64/libvirt.so.0
#8  0x00007f0bcfa5e949 in virFirewallNew () from /lib64/libvirt.so.0
#9  0x00007f0bbc260ffe in networkRemoveFirewallRules ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so
#10 0x00007f0bbc256ebd in networkReloadFirewallRulesHelper ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so
#11 0x00007f0bcfaeb24a in virNetworkObjListForEachHelper ()
   from /lib64/libvirt.so.0
#12 0x00007f0bcfa6024c in virHashForEach () from /lib64/libvirt.so.0
#13 0x00007f0bcfaf4f71 in virNetworkObjListForEach () from /lib64/libvirt.so.0
#14 0x00007f0bbc2596a4 in networkStateInitialize ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_network.so
#15 0x00007f0bcfb26b6f in virStateInitialize () from /lib64/libvirt.so.0
#16 0x00007f0bd0764f1b in daemonRunStateInit ()
#17 0x00007f0bcfa98f42 in virThreadHelper () from /lib64/libvirt.so.0
---Type <return> to continue, or q <return> to quit---
#18 0x00007f0bcd103dc5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f0bcce30ced in clone () from /lib64/libc.so.6

According to logs libvirtd wanted to check if update locks are usable by running "ebtables --concurrent -L", but the ebtables command never returns:

2016-05-26 05:50:50.516+0000: 4988: info : virFirewallCheckUpdateLock:127 :
    using locking for /usr/sbin/ip6tables
2016-05-26 05:50:50.516+0000: 4988: debug : virCommandRunAsync:2428 :
    About to run /usr/sbin/ebtables --concurrent -L
2016-05-26 05:50:50.516+0000: 4988: debug : virFileClose:102 : Closed fd 16
2016-05-26 05:50:50.516+0000: 4988: debug : virFileClose:102 : Closed fd 18
2016-05-26 05:50:50.516+0000: 4988: debug : virFileClose:102 : Closed fd 20
2016-05-26 05:50:50.516+0000: 4988: debug : virCommandRunAsync:2431 :
    Command result 0, with PID 5028

It's worth mentioning that firewalld is not running (in contrast to the default RHEL-7 setup):

2016-05-26 05:50:50.501+0000: 4988: debug : virDBusIsServiceRegistered:1754 :
    Service org.fedoraproject.FirewallD1 is not registered
2016-05-26 05:50:50.501+0000: 4988: debug : virFirewallValidateBackend:163 :
    Firewalld is registered ? -2
2016-05-26 05:50:50.501+0000: 4988: debug : virFirewallValidateBackend:171 :
    firewalld service not running, trying direct backend
2016-05-26 05:50:50.501+0000: 4988: debug : virFirewallValidateBackend:197 :
    found iptables/ip6tables/ebtables, using direct backend
Comment 15 Asim 2016-05-26 09:18:08 UTC 
Hi,

Deleting the /var/lib/ebtables/lock  file fixed the issue.

I will debug it in a similar way on my other machines which require libvirtd restart to make virsh commands work.
Comment 16 Jiri Denemark 2016-06-07 15:07:56 UTC 
OK, closing as not a bug since it was apparently caused by a stale ebtables lock file. Feel free to reopen in case you have exact steps how this can be reproduced.
Comment 17 Asim 2016-06-28 03:52:50 UTC 
Created attachment 1173229 [details]
logs shown in gdb
Comment 18 Asim 2016-06-28 03:53:31 UTC 
Created attachment 1173230 [details]
virsh List logs
Comment 19 Asim 2016-06-28 03:54:32 UTC 
Hi,

I am still facing the "virsh list" hanging issue on one of my nodes.

I am attaching the Logs above.
Comment 20 Jiri Denemark 2016-06-28 06:05:11 UTC 
From the gdb backtrace it looks like all the threads are blocked by

Thread 13 (Thread 0x7ff3a931c700 (LWP 8369)):
#0  0x00007ff3c502022d in read () from /lib64/libpthread.so.0
#1  0x00007ff3c796ee4e in saferead () from /lib64/libvirt.so.0
#2  0x00007ff3c795ff43 in virCommandHandshakeWait () from /lib64/libvirt.so.0
#3  0x00007ff3aee1d757 in qemuProcessStart ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#4  0x00007ff3aee6a8ef in qemuDomainCreateXML ()
   from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#5  0x00007ff3c7a3e151 in virDomainCreateXML () from /lib64/libvirt.so.0
#6  0x00007ff3c86a211a in remoteDispatchDomainCreateXMLHelper ()
#7  0x00007ff3c7ab9b02 in virNetServerProgramDispatch ()
#8  0x00007ff3c7ab4d7d in virNetServerHandleJob () from /lib64/libvirt.so.0
#9  0x00007ff3c79af9f5 in virThreadPoolWorker () from /lib64/libvirt.so.0
#10 0x00007ff3c79aef18 in virThreadHelper () from /lib64/libvirt.so.0
#11 0x00007ff3c5019dc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007ff3c4d46ced in clone () from /lib64/libc.so.6


Starting a new QEMU process is really not expected to take such a long time. Could you attach corresponding qemu and libvirtd logs?
Comment 21 Jaroslav Suchanek 2016-07-15 11:11:54 UTC 
Ping Asim, please attach qemu and libvirtd logs as Jiri requested in comment 20. Thanks.
Comment 22 Asim 2016-07-15 14:31:37 UTC 
Hi,

I just restarted the libvirt service as it was a production server to fix the issue temporarily.

Will provide you the logs when it happens next time.
Comment 23 Asim 2016-09-27 07:35:37 UTC 
Created attachment 1205077 [details]
GDB Logs
Comment 24 Asim 2016-09-27 07:36:23 UTC 
Created attachment 1205079 [details]
Logs as per : LIBVIRT_DEBUG=debug LIBVIRT_LOG_FILTERS="1:remote" LIBVIRT_LOG_OUTPUTS="3:syslog:libvirtd 1:file:/tmp/libvirt.log" virsh list
Comment 25 Asim 2016-09-27 07:41:26 UTC 
Hi,

I just faced the same issue on my another node the virsh list simply hangs.

I have attached the GDB and "LIBVIRT_DEBUG=debug LIBVIRT_LOG_FILTERS="1:remote" LIBVIRT_LOG_OUTPUTS="3:syslog:libvirtd 1:file:/tmp/libvirt.log" virsh list" command logs.

I have also enabled debugging as per the guide http://wiki.libvirt.org/page/DebugLogs but as it requires libvirtd restart it has not taken any effect as I did not restart libvirtd (since it usually gets fixed when we restart libvirtd service).

Also there are tons of  files similar to guestfs-hsa7kv9b4e6qti4v.log in /var/log/libvirt/qemu/ folder ( I assume it is virt-df which is generating these logs), anyway to purge it ?

Any other logs which you need from me ?
Comment 26 Asim 2016-12-11 07:22:10 UTC 
Hi,

We were able to debug the issue, it was due to multiple virt-df instances running making virh list unresponsive.

We had a cron set to run virt-df command regularly to capture the data usage.

But it only affected centos 7 nodes, it worked flawless on other centos versions.

You can close this bug report.
Note You need to log in before you can comment on or make changes to this bug.