Bug 1302965

(In reply to yafu from comment #0)
> Description of problem:
> 
> 
> Version-Release number of selected component (if applicable):
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1.
> 2.
> 3.
> 
> Actual results:
> 
> 
> Expected results:
> 
> 
> Additional info:

Sorry for wrong clicking. The description is as follows:

Description of problem:
Using test-virt-alignment-scan-guests.sh in the libguestfs (https://github.com/libguestfs/libguestfs/blob/master/align/test-virt-alignment-scan-guests.sh, the script will start up lots of parallel libvirt instances), and the libvirtd daemon hang because of deadlock caused by parallel domain event handle.

Version-Release number of selected component (if applicable):
libvirt-1.2.17-13.el7_2.3.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.4.x86_64

How reproducible:
sometimes

Steps to Reproduce:
1.Run test-virt-alignment-scan-guests.sh in the libguestfs:
# while true ; do ./test-virt-alignment-scan-guests.sh ; done

2.Check the output of virsh list at the same time:
#watch virsh list

3.After about 20 hours, the libvirtd daemon hang and the output of virsh list did not change any more.

4.Using gdb to print the libvirtd backtrace:
(gdb)bt
#0  0x00007f80b0deaf4d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f80b0de6d02 in _L_lock_791 () from /lib64/libpthread.so.0
#2  0x00007f80b0de6c08 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f80b3779ba5 in virMutexLock (m=<optimized out>) at util/virthread.c:89
#4  0x00007f80b376122e in virObjectLock (anyobj=<optimized out>) at util/virobject.c:323
#5  0x00007f809abe223c in qemuProcessHandleEvent (mon=<optimized out>, vm=0x7f805c003aa0, eventName=0x7f80b61f6e10 "SHUTDOWN", seconds=1454045255, micros=634477, details=0x0, 
    opaque=0x7f8090101fc0) at qemu/qemu_process.c:645
#6  0x00007f809abfd81e in qemuMonitorEmitEvent (mon=mon@entry=0x7f805c00cb20, event=event@entry=0x7f80b61f6e10 "SHUTDOWN", seconds=1454045255, micros=634477, details=0x0)
    at qemu/qemu_monitor.c:1255
#7  0x00007f809ac11e4c in qemuMonitorJSONIOProcessEvent (obj=0x7f80b62151a0, mon=0x7f805c00cb20) at qemu/qemu_monitor_json.c:165
#8  qemuMonitorJSONIOProcessLine (msg=0x0, line=<optimized out>, mon=0x7f805c00cb20) at qemu/qemu_monitor_json.c:202
#9  qemuMonitorJSONIOProcess (mon=mon@entry=0x7f805c00cb20, data=0x7f80b621f620 "{\"timestamp\": {\"seconds\": 1454045255, \"microseconds\": 634477}, \"event\": \"SHUTDOWN\"}\r\n", 
    len=85, msg=msg@entry=0x0) at qemu/qemu_monitor_json.c:244
#10 0x00007f809abfbff3 in qemuMonitorIOProcess (mon=0x7f805c00cb20) at qemu/qemu_monitor.c:455
#11 qemuMonitorIO (watch=watch@entry=154073, fd=<optimized out>, events=0, events@entry=1, opaque=opaque@entry=0x7f805c00cb20) at qemu/qemu_monitor.c:709
#12 0x00007f80b3738967 in virEventPollDispatchHandles (fds=<optimized out>, nfds=<optimized out>) at util/vireventpoll.c:509
#13 virEventPollRunOnce () at util/vireventpoll.c:658
#14 0x00007f80b3737032 in virEventRunDefaultImpl () at util/virevent.c:308
#15 0x00007f80b387f035 in virNetDaemonRun (dmn=0x7f80b6208860) at rpc/virnetdaemon.c:701
#16 0x00007f80b4444524 in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1577


Actual results:


Expected results:
The libvirtd process should not hang while starting up lots of parallel libvirt instances.

Additional info:

Please see the libvirtd process's backtrace of all stack frames in the attachment.

Created attachment 1119375 [details]
backtrace of all stack frames

Could you provide debug logs from libvirtd and ideally also full core dump of the stuck libvirtd?

(In reply to Jiri Denemark from comment #5)
> Could you provide debug logs from libvirtd and ideally also full core dump
> of the stuck libvirtd?

Since the log_level is 3 while libvirtd hang, I can not provide debug logs from libvirtd.
Please see the full core dump of the libvirtd in the attachment.

*** This bug has been marked as a duplicate of bug 1348936 ***