Bug 1610207 - libvirtd crash when do snapshot-delete while doing snapshot revert
Summary: libvirtd crash when do snapshot-delete while doing snapshot revert
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux Advanced Virtualization
Classification: Red Hat
Component: libvirt
Version: 8.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 8.1
Assignee: Pavel Mores
QA Contact: yisun
URL:
Whiteboard:
: 1643810 (view as bug list)
Depends On:
Blocks: 1643810
TreeView+ depends on / blocked
 
Reported: 2018-07-31 08:19 UTC by yafu
Modified: 2020-11-06 04:04 UTC (History)
9 users (show)

Fixed In Version: libvirt-6.0.0-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1643810 (view as bug list)
Environment:
Last Closed: 2020-05-05 09:43:16 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:2017 0 None None None 2020-05-05 09:44:40 UTC

Description yafu 2018-07-31 08:19:46 UTC
Description of problem:
libvirtd crash when do snapshot-delete while doing snapshot revert.

Version-Release number of selected component (if applicable):
libvirt-4.5.0-4.el7.x86_64
qemu-kvm-rhev-2.12.0-8.el7.x86_64

How reproducible:
50%

Steps to reproduce:
1.Start a guest:
#virsh start avocado-vt-vm1

2.Do internal snapshot:
#virsh snapshot-create-as avocado-vt-vm1 s1

3.Do some change to the guest:
#cat rng.xml
  <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
  </rng>

#virsh attach-device avocado rng.xml

4.Destroy and start the guest again:
#virsh destroy avocado-vt-vm1
#virsh start avocado-vt-vm1

5.Do internal snapshot:
#virsh snapshot-create-as avocado-vt-vm1 s2

6.Do snapshot-delete while doing snapshot revert:
#virsh snapshot-revert avocado-vt-vm1 s1 --force & sleep 3; virsh snapshot-delete avocado-vt-vm1 s2
error: Disconnected from qemu:///system due to end of file
error: Failed to delete snapshot s2
error: End of file while reading data: Input/output error

Actual resutls:
libvirtd crash when do snapshot-delete while doing snapshot revert.

Expected results:
libvirtd should not crash

Additional info:
1.Can reproduce with following commands:
#virsh start avocado-vt-vm1; sleep 15; virsh snapshot-create-as avocado-vt-vm1 s1; virsh destroy avocado-vt-vm1; virsh attach-device avocado-vt-vm1 /tmp/rng.xml --config;virsh start avocado-vt-vm1; sleep 28; virsh snapshot-create-as avocado-vt-vm1 s2; virsh snapshot-revert avocado-vt-vm1 s1 --force & sleep 3; virsh snapshot-delete avocado-vt-vm1 s2

2.The backtrace of libvirtd:
(gdb) bt
#0  qemuMonitorSend (mon=mon@entry=0x7f0970053070, msg=msg@entry=0x7f09898a7380)
    at qemu/qemu_monitor.c:1085
#1  0x00007f0946679ed8 in qemuMonitorJSONCommandWithFd (mon=mon@entry=0x7f0970053070, 
    cmd=cmd@entry=0x7f0970052f20, scm_fd=scm_fd@entry=-1, 
    reply=reply@entry=0x7f09898a7400) at qemu/qemu_monitor_json.c:307
#2  0x00007f094667c0d8 in qemuMonitorJSONCommand (reply=0x7f09898a7400, 
    cmd=0x7f0970052f20, mon=0x7f0970053070) at qemu/qemu_monitor_json.c:337
#3  qemuMonitorJSONSetCapabilities (mon=mon@entry=0x7f0970053070)
    at qemu/qemu_monitor_json.c:1385
---Type <return> to continue, or q <return> to quit---
#4  0x00007f094666585e in qemuMonitorSetCapabilities (mon=0x7f0970053070)
    at qemu/qemu_monitor.c:1695
#5  0x00007f094663748b in qemuProcessInitMonitor (asyncJob=QEMU_ASYNC_JOB_NONE, 
    vm=0x7f097c000a60, driver=0x7f092c0fa080) at qemu/qemu_process.c:1727
#6  qemuConnectMonitor (driver=driver@entry=0x7f092c0fa080, vm=vm@entry=0x7f097c000a60, 
    asyncJob=asyncJob@entry=0, retry=retry@entry=false, 
    logCtxt=logCtxt@entry=0x7f09700187a0) at qemu/qemu_process.c:1804
#7  0x00007f0946638a0d in qemuProcessWaitForMonitor (driver=driver@entry=0x7f092c0fa080, 
    vm=vm@entry=0x7f097c000a60, asyncJob=asyncJob@entry=0, 
---Type <return> to continue, or q <return> to quit---
    logCtxt=logCtxt@entry=0x7f09700187a0) at qemu/qemu_process.c:2214
#8  0x00007f0946641131 in qemuProcessLaunch (conn=conn@entry=0x7f097c0090e0, 
    driver=driver@entry=0x7f092c0fa080, vm=vm@entry=0x7f097c000a60, 
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, incoming=incoming@entry=0x0, 
    snapshot=snapshot@entry=0x7f09700365d0, 
    vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=flags@entry=34)
    at qemu/qemu_process.c:6464
#9  0x00007f094664860d in qemuProcessStart (conn=0x7f097c0090e0, 
    driver=driver@entry=0x7f092c0fa080, vm=0x7f097c000a60, updatedCPU=0x0, 
---Type <return> to continue, or q <return> to quit---
    asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, migrateFrom=migrateFrom@entry=0x0, 
    migrateFd=migrateFd@entry=-1, migratePath=migratePath@entry=0x0, 
    snapshot=snapshot@entry=0x7f09700365d0, 
    vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=flags@entry=34)
    at qemu/qemu_process.c:6690
#10 0x00007f09466bf6c5 in qemuDomainRevertToSnapshot (snapshot=0x7f097003bbd0, flags=4)
    at qemu/qemu_driver.c:16274
#11 0x00007f099b237795 in virDomainRevertToSnapshot (
    snapshot=snapshot@entry=0x7f097003bbd0, flags=4) at libvirt-domain-snapshot.c:1071
---Type <return> to continue, or q <return> to quit---
#12 0x000055dec2fdb66a in remoteDispatchDomainRevertToSnapshot (server=<optimized out>, 
    msg=0x55dec4746ad0, args=0x7f097003b590, rerr=0x7f09898a7bc0, client=<optimized out>)
    at remote/remote_daemon_dispatch_stubs.h:9670
#13 remoteDispatchDomainRevertToSnapshotHelper (server=<optimized out>, 
    client=<optimized out>, msg=0x55dec4746ad0, rerr=0x7f09898a7bc0, args=0x7f097003b590, 
    ret=0x7f097003b8f0) at remote/remote_daemon_dispatch_stubs.h:9642
#14 0x00007f099b103ed5 in virNetServerProgramDispatchCall (msg=0x55dec4746ad0, 
    client=0x55dec4746590, server=0x55dec4723fd0, prog=0x55dec4743650)
    at rpc/virnetserverprogram.c:437
---Type <return> to continue, or q <return> to quit---
#15 virNetServerProgramDispatch (prog=0x55dec4743650, server=server@entry=0x55dec4723fd0, 
    client=client@entry=0x55dec4746590, msg=msg@entry=0x55dec4746ad0)
    at rpc/virnetserverprogram.c:304
#16 0x00007f099b10cc3a in virNetServerProcessMsg (srv=srv@entry=0x55dec4723fd0, 
    client=0x55dec4746590, prog=<optimized out>, msg=0x55dec4746ad0)
    at rpc/virnetserver.c:143
#17 0x00007f099b10d088 in virNetServerHandleJob (jobOpaque=<optimized out>, 
    opaque=0x55dec4723fd0) at rpc/virnetserver.c:164
#18 0x00007f099afeada1 in virThreadPoolWorker (opaque=opaque@entry=0x55dec47185c0)
---Type <return> to continue, or q <return> to quit---
    at util/virthreadpool.c:167
#19 0x00007f099afe9b70 in virThreadHelper (data=<optimized out>) at util/virthread.c:206
#20 0x00007f0998301dd5 in start_thread (arg=0x7f09898a8700) at pthread_create.c:307
#21 0x00007f099802aead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Comment 2 Jaroslav Suchanek 2019-04-24 12:40:34 UTC
This bug is going to be addressed in the next major release.

Comment 3 Jaroslav Suchanek 2019-07-22 19:06:39 UTC
*** Bug 1643810 has been marked as a duplicate of this bug. ***

Comment 4 Pavel Mores 2019-12-12 10:53:19 UTC
Fixed by:

d75f865fb9 qemu: fix concurrency crash bug in snapshot revert

v5.10.0-205-gd75f865fb9

Comment 6 yisun 2020-01-19 13:21:12 UTC
Verified with libvirt-6.0.0-1, no crash happened
[root@dell-per730-67 ~]# rpm -qa | grep libvirt-6
libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64

[root@dell-per730-67 ~]# virsh define vm1.xml ; virsh start vm1
Domain vm1 defined from vm1.xml

Domain vm1 started


[root@dell-per730-67 ~]# virsh snapshot-create-as vm1 s1
Domain snapshot s1 created
[root@dell-per730-67 ~]# cat rng.xml
<rng model='virtio'>
      <backend model='random'>/dev/random</backend>
  </rng>

[root@dell-per730-67 ~]# virsh attach-device vm1 rng.xml
Device attached successfully

[root@dell-per730-67 ~]# virsh destroy vm1; virsh start vm1
Domain vm1 destroyed

Domain vm1 started

[root@dell-per730-67 ~]# virsh snapshot-create-as vm1 s2
Domain snapshot s2 created


[root@dell-per730-67 ~]# virsh snapshot-revert vm1 s1 --force & sleep 3; virsh snapshot-delete vm1 s2
[1] 3150

[1]+  Done                    virsh snapshot-revert vm1 s1 --force
Domain snapshot s2 deleted


[root@dell-per730-67 ~]# virsh snapshot-list vm1
 Name   Creation Time               State
---------------------------------------------
 s1     2020-01-19 08:18:28 -0500   running

Comment 8 errata-xmlrpc 2020-05-05 09:43:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017


Note You need to log in before you can comment on or make changes to this bug.