Bug 1290647

It would be helpful if you wouldn't truncate threads that are doing actual work and left behind idle worker threads. There are two relevant threads in my reproduction attempt:

Thread 11 (Thread 0x7f6e9afbe700 (LWP 23726)):
#0  virDomainSnapshotDropParent (snapshot=0x7f6e7800a0b0) at conf/snapshot_conf.c:1007
#1  0x000000000047135f in qemuDomainSnapshotCreateXML (domain=0x7f6e6c0c5ee0, xmlDesc=<value optimized out>, 
    flags=<value optimized out>) at qemu/qemu_driver.c:12541
#2  0x00007f6ea4216997 in virDomainSnapshotCreateXML (domain=0x7f6e6c0c5ee0, 
    xmlDesc=0x7f6e6c0c92e0 "<domainsnapshot>\n  <name>test</name>\n  <state>running</state>\n  <creationTime>1449849610</creationTime>\n  <memory snapshot='internal'/>\n  <disks>\n    <disk name='vda' snapshot='internal'/>\n  </disks>\n"..., flags=3)
    at libvirt.c:18016
#3  0x0000000000431e3f in remoteDispatchDomainSnapshotCreateXML (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e9afbdb80, args=0x7f6e6c0c9270, ret=0x7f6e6c0c92b0) at remote_dispatch.h:5894
#4  remoteDispatchDomainSnapshotCreateXMLHelper (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e9afbdb80, args=0x7f6e6c0c9270, ret=0x7f6e6c0c92b0) at remote_dispatch.h:5870
#5  0x00007f6ea425df62 in virNetServerProgramDispatchCall (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d15350, msg=0x1d170c0)
    at rpc/virnetserverprogram.c:431
#6  virNetServerProgramDispatch (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d15350, msg=0x1d170c0)
    at rpc/virnetserverprogram.c:304
#7  0x00007f6ea425aeee in virNetServerProcessMsg (srv=<value optimized out>, client=0x1d15350, prog=<value optimized out>, 
    msg=0x1d170c0) at rpc/virnetserver.c:170
#8  0x00007f6ea425b58c in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1d0f4b0) at rpc/virnetserver.c:191
#9  0x00007f6ea417ad7c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#10 0x00007f6ea417a669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#11 0x00007f6ea375ba51 in start_thread (arg=0x7f6e9afbe700) at pthread_create.c:301
#12 0x00007f6ea30a193d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115


Thread 8 (Thread 0x7f6e991bb700 (LWP 23729)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f6ea417a846 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x000000000049b4bc in qemuMonitorSend (mon=0x7f6e7400bc90, msg=<value optimized out>) at qemu/qemu_monitor.c:914
#3  0x000000000049ff3e in qemuMonitorJSONCommandWithFd (mon=0x7f6e7400bc90, cmd=0x7f6e78000910, scm_fd=-1, reply=0x7f6e991ba778)
    at qemu/qemu_monitor_json.c:271
#4  0x00000000004a86d3 in qemuMonitorJSONHumanCommandWithFd (mon=0x7f6e7400bc90, cmd_str=0x7f6e7800d2f0 "savevm \"test\"", 
    scm_fd=-1, reply_str=0x7f6e991ba820) at qemu/qemu_monitor_json.c:1024
#5  0x000000000049c98d in qemuMonitorHMPCommandWithFd (mon=0x7f6e7400bc90, cmd=0x7f6e78000cc0 "savevm \"test\"", 
    scm_fd=<value optimized out>, reply=<value optimized out>) at qemu/qemu_monitor.c:955
#6  0x00000000004ab454 in qemuMonitorTextCreateSnapshot (mon=<value optimized out>, name=<value optimized out>)
    at qemu/qemu_monitor_text.c:2838
#7  0x00000000004a5f49 in qemuMonitorJSONCreateSnapshot (mon=0x7f6e7400bc90, name=0x7f6e78000bd0 "test")
    at qemu/qemu_monitor_json.c:3399
#8  0x0000000000471447 in qemuDomainSnapshotCreateActiveInternal (domain=<value optimized out>, xmlDesc=<value optimized out>, 
    flags=136) at qemu/qemu_driver.c:11713
#9  qemuDomainSnapshotCreateXML (domain=<value optimized out>, xmlDesc=<value optimized out>, flags=136)
    at qemu/qemu_driver.c:12631
#10 0x00007f6ea4216997 in virDomainSnapshotCreateXML (domain=0x7f6e78000ce0, 
    xmlDesc=0x7f6e78000c30 "<domainsnapshot>\n  <name>test</name>\n</domainsnapshot>\n", flags=8) at libvirt.c:18016
#11 0x0000000000431e3f in remoteDispatchDomainSnapshotCreateXML (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e991bab80, args=0x7f6e78000bf0, ret=0x7f6e780008c0) at remote_dispatch.h:5894
#12 remoteDispatchDomainSnapshotCreateXMLHelper (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e991bab80, args=0x7f6e78000bf0, ret=0x7f6e780008c0) at remote_dispatch.h:5870
#13 0x00007f6ea425df62 in virNetServerProgramDispatchCall (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d16d20, msg=0x1d16e00)
    at rpc/virnetserverprogram.c:431
#14 virNetServerProgramDispatch (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d16d20, msg=0x1d16e00)
    at rpc/virnetserverprogram.c:304
#15 0x00007f6ea425aeee in virNetServerProcessMsg (srv=<value optimized out>, client=0x1d16d20, prog=<value optimized out>, 
    msg=0x1d16e00) at rpc/virnetserver.c:170
#16 0x00007f6ea425b58c in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1d0f4b0) at rpc/virnetserver.c:191
#17 0x00007f6ea417ad7c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#18 0x00007f6ea417a669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#19 0x00007f6ea375ba51 in start_thread (arg=0x7f6e991bb700) at pthread_create.c:301
#20 0x00007f6ea30a193d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

All other worker threads are idle.

The issue cannot be reproduced with scratch build

# rpm -q libvirt
libvirt-0.10.2-56.el6_rc.33ae2f56.x86_64

Reproducer:
# virsh list
 Id    Name                           State
----------------------------------------------------
 3     vm1                            running

# virsh snapshot-create vm1 rh6-snapshot.xml 
Domain snapshot rh6-snapshot created from 'rh6-snapshot.xml'

# virsh snapshot-dumpxml vm1 rh6-snapshot > snap1.xml; virsh snapshot-create vm1 --redefine snap1.xml 
error: Timed out during operation: cannot acquire state change lock

I can reproduce it on libvirt-libvirt-0.10.2-55.el6.x86_64.
Verify it on libvirt-0.10.2-56.el6.x86_64:
1. Prepare a running guest:
# virsh list 
 Id    Name                           State
----------------------------------------------------
 9     nn                             running

2. Do snapshot redefine while create snapshot with --halt:
./reproduce.sh:
```
#!/bin/bash -x
virsh snapshot-create-as nn --halt --name "nn" &
virsh snapshot-dumpxml nn nn|virsh snapshot-create nn --redefine --current /dev/stdin
```
# ./reproduce.sh
+ set -o nounset
+ virsh snapshot-create-as nn --halt --name nn
+ virsh snapshot-dumpxml nn nn
+ virsh snapshot-create nn --redefine --current /dev/stdin
error: Domain snapshot not found: no domain snapshot with matching name 'nn'
error: (domain_snapshot):2: Start tag expected, '<' not found
(null)
^
Domain snapshot nn created

3. Add --disk-only option in the script and run it:
# ./reproduce.sh
+ virsh snapshot-create-as nn --halt --disk-only --name n1
+ virsh snapshot-dumpxml nn n1
+ virsh snapshot-create nn --redefine --current /dev/stdin
error: Domain snapshot not found: no domain snapshot with matching name 'n1'
error: (domain_snapshot):2: Start tag expected, '<' not found
(null)
^
Domain snapshot n1 created

4. Without --halt option and run the script:
+ virsh snapshot-create-as nn --name n2
+ virsh snapshot-dumpxml nn n2
+ virsh snapshot-create nn --redefine --current /dev/stdin
error: Domain snapshot not found: no domain snapshot with matching name 'n2'
error: (domain_snapshot):2: Start tag expected, '<' not found
(null)
^
Domain snapshot n2 created


Libvirtd doesn't crash when running the script. And finally the snapshot created.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0738.html