RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1290647 - libvirtd will crash when try to redefine a snapshot for the snapshot is creating
Summary: libvirtd will crash when try to redefine a snapshot for the snapshot is creating
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Peter Krempa
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-12-11 03:41 UTC by Jingjing Shao
Modified: 2016-05-10 19:25 UTC (History)
6 users (show)

Fixed In Version: libvirt-0.10.2-56.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-05-10 19:25:39 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:0738 0 normal SHIPPED_LIVE libvirt bug fix update 2016-05-12 18:40:37 UTC

Description Jingjing Shao 2015-12-11 03:41:16 UTC 
Description of problem:
libvirtd will crash when try to redefine a snapshot for the snapshot is creating

PKG
OS-rhel6.7
libvirt-libvirt-0.10.2-55.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.481.el6.x86_64

How reproducible:
100%

Setup
A running healthy guest rh6


Steps to Reproduce:
1.# vim rh6-snapshot.xml
<domainsnapshot>
<name>rh6-snapshot</name>
<description>snapshot API test - create snapshot</description>
</domainsnapshot>

2.# virsh snapshot-create rh6 rh6-snapshot.xml --halt

When step2 is running, execute step3 and step4

3.# virsh snapshot-dumpxml rh6 rh6-snapshot > snap1.xml

4.# virsh snapshot-create rh6 --redefine --current snap1.xml

Then the libvirtd crash


Expected results:
The libvirtd should not crash


Additional info:
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fed9bb84700 (LWP 14910)]
virDomainSnapshotDropParent (snapshot=0x7fed880080a0) at conf/snapshot_conf.c:1007
1007	    curr = snapshot->parent->first_child;
(gdb) t a a bt 

Thread 11 (Thread 0x7fed9c585700 (LWP 14909)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x0000003f98865846 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x0000003f98865e13 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
#3  0x0000003f98865669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00000031c0e07a51 in start_thread (arg=0x7fed9c585700) at pthread_create.c:301
#5  0x00000031c0ae896d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 10 (Thread 0x7fed9bb84700 (LWP 14910)):
#0  virDomainSnapshotDropParent (snapshot=0x7fed880080a0) at conf/snapshot_conf.c:1007
#1  0x000000000047135f in qemuDomainSnapshotCreateXML (domain=0x7fed7c000930, xmlDesc=<value optimized out>, flags=<value optimized out>) at qemu/qemu_driver.c:12541
#2  0x0000003f98901997 in virDomainSnapshotCreateXML (domain=0x7fed7c000930, 
    xmlDesc=0x7fed7c0011c0 "<domainsnapshot>\n  <name>generic-snapshot</name>\n  <description>snapshot API test - create snapshot</description>\n  <state>running</state>\n  <creationTime>1449804555</creationTime>\n  <memory snapshot="..., flags=3) at libvirt.c:18016
#3  0x0000000000431e3f in remoteDispatchDomainSnapshotCreateXML (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, 
    rerr=0x7fed9bb83b80, args=0x7fed7c02f9a0, ret=0x7fed7c02f9e0) at remote_dispatch.h:5894
#4  remoteDispatchDomainSnapshotCreateXMLHelper (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7fed9bb83b80, 
    args=0x7fed7c02f9a0, ret=0x7fed7c02f9e0) at remote_dispatch.h:5870
#5  0x0000003f98948f62 in virNetServerProgramDispatchCall (prog=0x1e26ea0, server=0x1e1e450, client=0x1e22290, msg=0x1e220d0) at rpc/virnetserverprogram.c:431
#6  virNetServerProgramDispatch (prog=0x1e26ea0, server=0x1e1e450, client=0x1e22290, msg=0x1e220d0) at rpc/virnetserverprogram.c:304
#7  0x0000003f98945eee in virNetServerProcessMsg (srv=<value optimized out>, client=0x1e22290, prog=<value optimized out>, msg=0x1e220d0) at rpc/virnetserver.c:170
#8  0x0000003f9894658c in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1e1e450) at rpc/virnetserver.c:191
#9  0x0000003f98865d7c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#10 0x0000003f98865669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#11 0x00000031c0e07a51 in start_thread (arg=0x7fed9bb84700) at pthread_create.c:301
#12 0x00000031c0ae896d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

Thread 9 (Thread 0x7fed9b183700 (LWP 14911)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x0000003f98865846 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x0000003f98865e13 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103
---Type <return> to continue, or q <return> to quit---
#3  0x0000003f98865669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#4  0x00000031c0e07a51 in start_thread (arg=0x7fed9b183700) at pthread_create.c:301
#5  0x00000031c0ae896d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115
Comment 1 Peter Krempa 2015-12-11 16:04:15 UTC 
It would be helpful if you wouldn't truncate threads that are doing actual work and left behind idle worker threads. There are two relevant threads in my reproduction attempt:

Thread 11 (Thread 0x7f6e9afbe700 (LWP 23726)):
#0  virDomainSnapshotDropParent (snapshot=0x7f6e7800a0b0) at conf/snapshot_conf.c:1007
#1  0x000000000047135f in qemuDomainSnapshotCreateXML (domain=0x7f6e6c0c5ee0, xmlDesc=<value optimized out>, 
    flags=<value optimized out>) at qemu/qemu_driver.c:12541
#2  0x00007f6ea4216997 in virDomainSnapshotCreateXML (domain=0x7f6e6c0c5ee0, 
    xmlDesc=0x7f6e6c0c92e0 "<domainsnapshot>\n  <name>test</name>\n  <state>running</state>\n  <creationTime>1449849610</creationTime>\n  <memory snapshot='internal'/>\n  <disks>\n    <disk name='vda' snapshot='internal'/>\n  </disks>\n"..., flags=3)
    at libvirt.c:18016
#3  0x0000000000431e3f in remoteDispatchDomainSnapshotCreateXML (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e9afbdb80, args=0x7f6e6c0c9270, ret=0x7f6e6c0c92b0) at remote_dispatch.h:5894
#4  remoteDispatchDomainSnapshotCreateXMLHelper (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e9afbdb80, args=0x7f6e6c0c9270, ret=0x7f6e6c0c92b0) at remote_dispatch.h:5870
#5  0x00007f6ea425df62 in virNetServerProgramDispatchCall (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d15350, msg=0x1d170c0)
    at rpc/virnetserverprogram.c:431
#6  virNetServerProgramDispatch (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d15350, msg=0x1d170c0)
    at rpc/virnetserverprogram.c:304
#7  0x00007f6ea425aeee in virNetServerProcessMsg (srv=<value optimized out>, client=0x1d15350, prog=<value optimized out>, 
    msg=0x1d170c0) at rpc/virnetserver.c:170
#8  0x00007f6ea425b58c in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1d0f4b0) at rpc/virnetserver.c:191
#9  0x00007f6ea417ad7c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#10 0x00007f6ea417a669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#11 0x00007f6ea375ba51 in start_thread (arg=0x7f6e9afbe700) at pthread_create.c:301
#12 0x00007f6ea30a193d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115


Thread 8 (Thread 0x7f6e991bb700 (LWP 23729)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:183
#1  0x00007f6ea417a846 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117
#2  0x000000000049b4bc in qemuMonitorSend (mon=0x7f6e7400bc90, msg=<value optimized out>) at qemu/qemu_monitor.c:914
#3  0x000000000049ff3e in qemuMonitorJSONCommandWithFd (mon=0x7f6e7400bc90, cmd=0x7f6e78000910, scm_fd=-1, reply=0x7f6e991ba778)
    at qemu/qemu_monitor_json.c:271
#4  0x00000000004a86d3 in qemuMonitorJSONHumanCommandWithFd (mon=0x7f6e7400bc90, cmd_str=0x7f6e7800d2f0 "savevm \"test\"", 
    scm_fd=-1, reply_str=0x7f6e991ba820) at qemu/qemu_monitor_json.c:1024
#5  0x000000000049c98d in qemuMonitorHMPCommandWithFd (mon=0x7f6e7400bc90, cmd=0x7f6e78000cc0 "savevm \"test\"", 
    scm_fd=<value optimized out>, reply=<value optimized out>) at qemu/qemu_monitor.c:955
#6  0x00000000004ab454 in qemuMonitorTextCreateSnapshot (mon=<value optimized out>, name=<value optimized out>)
    at qemu/qemu_monitor_text.c:2838
#7  0x00000000004a5f49 in qemuMonitorJSONCreateSnapshot (mon=0x7f6e7400bc90, name=0x7f6e78000bd0 "test")
    at qemu/qemu_monitor_json.c:3399
#8  0x0000000000471447 in qemuDomainSnapshotCreateActiveInternal (domain=<value optimized out>, xmlDesc=<value optimized out>, 
    flags=136) at qemu/qemu_driver.c:11713
#9  qemuDomainSnapshotCreateXML (domain=<value optimized out>, xmlDesc=<value optimized out>, flags=136)
    at qemu/qemu_driver.c:12631
#10 0x00007f6ea4216997 in virDomainSnapshotCreateXML (domain=0x7f6e78000ce0, 
    xmlDesc=0x7f6e78000c30 "<domainsnapshot>\n  <name>test</name>\n</domainsnapshot>\n", flags=8) at libvirt.c:18016
#11 0x0000000000431e3f in remoteDispatchDomainSnapshotCreateXML (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e991bab80, args=0x7f6e78000bf0, ret=0x7f6e780008c0) at remote_dispatch.h:5894
#12 remoteDispatchDomainSnapshotCreateXMLHelper (server=<value optimized out>, client=<value optimized out>, 
    msg=<value optimized out>, rerr=0x7f6e991bab80, args=0x7f6e78000bf0, ret=0x7f6e780008c0) at remote_dispatch.h:5870
#13 0x00007f6ea425df62 in virNetServerProgramDispatchCall (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d16d20, msg=0x1d16e00)
    at rpc/virnetserverprogram.c:431
#14 virNetServerProgramDispatch (prog=0x1d191f0, server=0x1d0f4b0, client=0x1d16d20, msg=0x1d16e00)
    at rpc/virnetserverprogram.c:304
#15 0x00007f6ea425aeee in virNetServerProcessMsg (srv=<value optimized out>, client=0x1d16d20, prog=<value optimized out>, 
    msg=0x1d16e00) at rpc/virnetserver.c:170
#16 0x00007f6ea425b58c in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x1d0f4b0) at rpc/virnetserver.c:191
#17 0x00007f6ea417ad7c in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144
#18 0x00007f6ea417a669 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161
#19 0x00007f6ea375ba51 in start_thread (arg=0x7f6e991bb700) at pthread_create.c:301
#20 0x00007f6ea30a193d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115

All other worker threads are idle.
Comment 3 Yang Yang 2015-12-12 01:37:43 UTC 
The issue cannot be reproduced with scratch build

# rpm -q libvirt
libvirt-0.10.2-56.el6_rc.33ae2f56.x86_64

Reproducer:
# virsh list
 Id    Name                           State
----------------------------------------------------
 3     vm1                            running

# virsh snapshot-create vm1 rh6-snapshot.xml 
Domain snapshot rh6-snapshot created from 'rh6-snapshot.xml'

# virsh snapshot-dumpxml vm1 rh6-snapshot > snap1.xml; virsh snapshot-create vm1 --redefine snap1.xml 
error: Timed out during operation: cannot acquire state change lock
Comment 5 Han Han 2016-02-01 09:19:21 UTC 
I can reproduce it on libvirt-libvirt-0.10.2-55.el6.x86_64.
Verify it on libvirt-0.10.2-56.el6.x86_64:
1. Prepare a running guest:
# virsh list 
 Id    Name                           State
----------------------------------------------------
 9     nn                             running

2. Do snapshot redefine while create snapshot with --halt:
./reproduce.sh:
```
#!/bin/bash -x
virsh snapshot-create-as nn --halt --name "nn" &
virsh snapshot-dumpxml nn nn|virsh snapshot-create nn --redefine --current /dev/stdin
```
# ./reproduce.sh
+ set -o nounset
+ virsh snapshot-create-as nn --halt --name nn
+ virsh snapshot-dumpxml nn nn
+ virsh snapshot-create nn --redefine --current /dev/stdin
error: Domain snapshot not found: no domain snapshot with matching name 'nn'
error: (domain_snapshot):2: Start tag expected, '<' not found
(null)
^
Domain snapshot nn created

3. Add --disk-only option in the script and run it:
# ./reproduce.sh
+ virsh snapshot-create-as nn --halt --disk-only --name n1
+ virsh snapshot-dumpxml nn n1
+ virsh snapshot-create nn --redefine --current /dev/stdin
error: Domain snapshot not found: no domain snapshot with matching name 'n1'
error: (domain_snapshot):2: Start tag expected, '<' not found
(null)
^
Domain snapshot n1 created

4. Without --halt option and run the script:
+ virsh snapshot-create-as nn --name n2
+ virsh snapshot-dumpxml nn n2
+ virsh snapshot-create nn --redefine --current /dev/stdin
error: Domain snapshot not found: no domain snapshot with matching name 'n2'
error: (domain_snapshot):2: Start tag expected, '<' not found
(null)
^
Domain snapshot n2 created


Libvirtd doesn't crash when running the script. And finally the snapshot created.
Comment 7 errata-xmlrpc 2016-05-10 19:25:39 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0738.html
Note You need to log in before you can comment on or make changes to this bug.