Description of problem: libvirtd crash when do snapshot-delete while doing snapshot revert. Version-Release number of selected component (if applicable): libvirt-4.5.0-4.el7.x86_64 qemu-kvm-rhev-2.12.0-8.el7.x86_64 How reproducible: 50% Steps to reproduce: 1.Start a guest: #virsh start avocado-vt-vm1 2.Do internal snapshot: #virsh snapshot-create-as avocado-vt-vm1 s1 3.Do some change to the guest: #cat rng.xml <rng model='virtio'> <backend model='random'>/dev/random</backend> </rng> #virsh attach-device avocado rng.xml 4.Destroy and start the guest again: #virsh destroy avocado-vt-vm1 #virsh start avocado-vt-vm1 5.Do internal snapshot: #virsh snapshot-create-as avocado-vt-vm1 s2 6.Do snapshot-delete while doing snapshot revert: #virsh snapshot-revert avocado-vt-vm1 s1 --force & sleep 3; virsh snapshot-delete avocado-vt-vm1 s2 error: Disconnected from qemu:///system due to end of file error: Failed to delete snapshot s2 error: End of file while reading data: Input/output error Actual resutls: libvirtd crash when do snapshot-delete while doing snapshot revert. Expected results: libvirtd should not crash Additional info: 1.Can reproduce with following commands: #virsh start avocado-vt-vm1; sleep 15; virsh snapshot-create-as avocado-vt-vm1 s1; virsh destroy avocado-vt-vm1; virsh attach-device avocado-vt-vm1 /tmp/rng.xml --config;virsh start avocado-vt-vm1; sleep 28; virsh snapshot-create-as avocado-vt-vm1 s2; virsh snapshot-revert avocado-vt-vm1 s1 --force & sleep 3; virsh snapshot-delete avocado-vt-vm1 s2 2.The backtrace of libvirtd: (gdb) bt #0 qemuMonitorSend (mon=mon@entry=0x7f0970053070, msg=msg@entry=0x7f09898a7380) at qemu/qemu_monitor.c:1085 #1 0x00007f0946679ed8 in qemuMonitorJSONCommandWithFd (mon=mon@entry=0x7f0970053070, cmd=cmd@entry=0x7f0970052f20, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7f09898a7400) at qemu/qemu_monitor_json.c:307 #2 0x00007f094667c0d8 in qemuMonitorJSONCommand (reply=0x7f09898a7400, cmd=0x7f0970052f20, mon=0x7f0970053070) at qemu/qemu_monitor_json.c:337 #3 qemuMonitorJSONSetCapabilities (mon=mon@entry=0x7f0970053070) at qemu/qemu_monitor_json.c:1385 ---Type <return> to continue, or q <return> to quit--- #4 0x00007f094666585e in qemuMonitorSetCapabilities (mon=0x7f0970053070) at qemu/qemu_monitor.c:1695 #5 0x00007f094663748b in qemuProcessInitMonitor (asyncJob=QEMU_ASYNC_JOB_NONE, vm=0x7f097c000a60, driver=0x7f092c0fa080) at qemu/qemu_process.c:1727 #6 qemuConnectMonitor (driver=driver@entry=0x7f092c0fa080, vm=vm@entry=0x7f097c000a60, asyncJob=asyncJob@entry=0, retry=retry@entry=false, logCtxt=logCtxt@entry=0x7f09700187a0) at qemu/qemu_process.c:1804 #7 0x00007f0946638a0d in qemuProcessWaitForMonitor (driver=driver@entry=0x7f092c0fa080, vm=vm@entry=0x7f097c000a60, asyncJob=asyncJob@entry=0, ---Type <return> to continue, or q <return> to quit--- logCtxt=logCtxt@entry=0x7f09700187a0) at qemu/qemu_process.c:2214 #8 0x00007f0946641131 in qemuProcessLaunch (conn=conn@entry=0x7f097c0090e0, driver=driver@entry=0x7f092c0fa080, vm=vm@entry=0x7f097c000a60, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, incoming=incoming@entry=0x0, snapshot=snapshot@entry=0x7f09700365d0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=flags@entry=34) at qemu/qemu_process.c:6464 #9 0x00007f094664860d in qemuProcessStart (conn=0x7f097c0090e0, driver=driver@entry=0x7f092c0fa080, vm=0x7f097c000a60, updatedCPU=0x0, ---Type <return> to continue, or q <return> to quit--- asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_NONE, migrateFrom=migrateFrom@entry=0x0, migrateFd=migrateFd@entry=-1, migratePath=migratePath@entry=0x0, snapshot=snapshot@entry=0x7f09700365d0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=flags@entry=34) at qemu/qemu_process.c:6690 #10 0x00007f09466bf6c5 in qemuDomainRevertToSnapshot (snapshot=0x7f097003bbd0, flags=4) at qemu/qemu_driver.c:16274 #11 0x00007f099b237795 in virDomainRevertToSnapshot ( snapshot=snapshot@entry=0x7f097003bbd0, flags=4) at libvirt-domain-snapshot.c:1071 ---Type <return> to continue, or q <return> to quit--- #12 0x000055dec2fdb66a in remoteDispatchDomainRevertToSnapshot (server=<optimized out>, msg=0x55dec4746ad0, args=0x7f097003b590, rerr=0x7f09898a7bc0, client=<optimized out>) at remote/remote_daemon_dispatch_stubs.h:9670 #13 remoteDispatchDomainRevertToSnapshotHelper (server=<optimized out>, client=<optimized out>, msg=0x55dec4746ad0, rerr=0x7f09898a7bc0, args=0x7f097003b590, ret=0x7f097003b8f0) at remote/remote_daemon_dispatch_stubs.h:9642 #14 0x00007f099b103ed5 in virNetServerProgramDispatchCall (msg=0x55dec4746ad0, client=0x55dec4746590, server=0x55dec4723fd0, prog=0x55dec4743650) at rpc/virnetserverprogram.c:437 ---Type <return> to continue, or q <return> to quit--- #15 virNetServerProgramDispatch (prog=0x55dec4743650, server=server@entry=0x55dec4723fd0, client=client@entry=0x55dec4746590, msg=msg@entry=0x55dec4746ad0) at rpc/virnetserverprogram.c:304 #16 0x00007f099b10cc3a in virNetServerProcessMsg (srv=srv@entry=0x55dec4723fd0, client=0x55dec4746590, prog=<optimized out>, msg=0x55dec4746ad0) at rpc/virnetserver.c:143 #17 0x00007f099b10d088 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x55dec4723fd0) at rpc/virnetserver.c:164 #18 0x00007f099afeada1 in virThreadPoolWorker (opaque=opaque@entry=0x55dec47185c0) ---Type <return> to continue, or q <return> to quit--- at util/virthreadpool.c:167 #19 0x00007f099afe9b70 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #20 0x00007f0998301dd5 in start_thread (arg=0x7f09898a8700) at pthread_create.c:307 #21 0x00007f099802aead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
This bug is going to be addressed in the next major release.
*** Bug 1643810 has been marked as a duplicate of this bug. ***
Fixed by: d75f865fb9 qemu: fix concurrency crash bug in snapshot revert v5.10.0-205-gd75f865fb9
Verified with libvirt-6.0.0-1, no crash happened [root@dell-per730-67 ~]# rpm -qa | grep libvirt-6 libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64 [root@dell-per730-67 ~]# virsh define vm1.xml ; virsh start vm1 Domain vm1 defined from vm1.xml Domain vm1 started [root@dell-per730-67 ~]# virsh snapshot-create-as vm1 s1 Domain snapshot s1 created [root@dell-per730-67 ~]# cat rng.xml <rng model='virtio'> <backend model='random'>/dev/random</backend> </rng> [root@dell-per730-67 ~]# virsh attach-device vm1 rng.xml Device attached successfully [root@dell-per730-67 ~]# virsh destroy vm1; virsh start vm1 Domain vm1 destroyed Domain vm1 started [root@dell-per730-67 ~]# virsh snapshot-create-as vm1 s2 Domain snapshot s2 created [root@dell-per730-67 ~]# virsh snapshot-revert vm1 s1 --force & sleep 3; virsh snapshot-delete vm1 s2 [1] 3150 [1]+ Done virsh snapshot-revert vm1 s1 --force Domain snapshot s2 deleted [root@dell-per730-67 ~]# virsh snapshot-list vm1 Name Creation Time State --------------------------------------------- s1 2020-01-19 08:18:28 -0500 running
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017