Description of problem: libvirtd crashes when undefine vm with --snapshots-metadata option Version-Release number of selected component (if applicable): libvirt-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64 qemu-kvm-3.1.0-25.module+el8.1.0+3164+94495c71.x86_64 kernel-4.18.0-80.23.el8.x86_64 How reproducible: 100% Steps to Reproduce: # ps -C libvirtd PID TTY TIME CMD 2636 ? 00:00:00 libvirtd # virsh snapshot-list tck Name Creation Time State -------------------------------------------------- snapshot1 2019-05-08 03:17:40 -0400 shutoff snapshot2 2019-05-08 03:17:40 -0400 shutoff snapshot3 2019-05-08 03:17:40 -0400 shutoff # virsh list --all Id Name State --------------------------------- - tck shut off # virsh undefine tck error: Failed to undefine domain tck error: Requested operation is not valid: cannot delete inactive domain with 3 snapshots # ps -C libvirtd PID TTY TIME CMD 2636 ? 00:00:00 libvirtd # virsh undefine tck --snapshots-metadata error: Disconnected from qemu:///system due to end of file error: Failed to undefine domain tck error: End of file while reading data: Input/output error # ps -C libvirtd PID TTY TIME CMD 2962 ? 00:00:00 libvirtd Actual results: libvirtd crashes. Expected results: undefine works. Additional info: GDB log: Thread 3 "libvirtd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7f4b3d0b2700 (LWP 1410)] 0x0000000000000000 in ?? () (gdb) thread apply all bt Thread 17 (Thread 0x7f4af19da700 (LWP 1469)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4afe99da34 in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so #3 0x00007f4b47254b7a in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7f4af37fe700 (LWP 1423)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7f4af3fff700 (LWP 1422)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7f4afcf4d700 (LWP 1421)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 --Type <RET> for more, q to quit, c to continue without paging--c #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7f4afd74e700 (LWP 1420)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7f4afdf4f700 (LWP 1419)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7f4b357fa700 (LWP 1418)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b472558d4 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7f4b35ffb700 (LWP 1417)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b472558d4 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7f4b367fc700 (LWP 1416)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b472558d4 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f4b36ffd700 (LWP 1415)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b472558d4 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f4b377fe700 (LWP 1414)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b472558d4 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f4b37fff700 (LWP 1413)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f4b2ffff700 (LWP 1412)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f4b3c8b1700 (LWP 1411)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f4b3d0b2700 (LWP 1410)): #0 0x0000000000000000 in ?? () #1 0x00007f4afdfd3518 in qemuDomainMomentDiscardAll () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #2 0x00007f4b471fef68 in virHashForEach () from /lib64/libvirt.so.0 #3 0x00007f4afdfdd67e in qemuDomainSnapshotDiscardAllMetadata () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f4afe0697b3 in qemuDomainUndefineFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007f4b473f3179 in virDomainUndefineFlags () from /lib64/libvirt.so.0 #6 0x00005583094efee1 in remoteDispatchDomainUndefineFlagsHelper () #7 0x00007f4b4732a7c4 in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #8 0x00007f4b47330cdc in virNetServerHandleJob () from /lib64/libvirt.so.0 #9 0x00007f4b47255840 in virThreadPoolWorker () from /lib64/libvirt.so.0 #10 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #11 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #12 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f4b3d8b3700 (LWP 1409)): #0 0x00007f4b446f84dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f4b47254dea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007f4b47255923 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007f4b47254b4c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007f4b446f22de in start_thread () from /lib64/libpthread.so.0 #5 0x00007f4b43de8653 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f4b47d76300 (LWP 1408)): #0 0x00007f4b43ddd3d1 in poll () from /lib64/libc.so.6 #1 0x00007f4b471f1b8b in virEventPollRunOnce () from /lib64/libvirt.so.0 #2 0x00007f4b471f06c5 in virEventRunDefaultImpl () from /lib64/libvirt.so.0 #3 0x00007f4b4733050d in virNetDaemonRun () from /lib64/libvirt.so.0 #4 0x00005583094d7986 in main ()
No symbols in backtrace. Please install the libvirt-debuginfo-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64 and libvirt-debugsource-5.3.0-1.module+el8.1.0+3164+94495c71.x86_64, then try to reproduce it again
Created attachment 1565541 [details] gdb log
virQEMUMomentRemovePtr->momentDiscard is passed in as NULL from qemuDomainSnapshotDiscardAllMetadata to qemuDomainMomentDiscardAll and unconditionally dereferenced. The code was introduced in commit a487890d371b8cc3662c1717dfe07eea3f1ef1c0 Author: Eric Blake <eblake> Date: Wed Mar 27 02:12:37 2019 -0500 snapshot: Refactor qemu to utilize virDomainMoment more Use the common base class virDomainMoment for iterator callbacks related to snapshots from the qemu code, so that when checkpoint operations are introduced, they can share the same callbacks. Simplify the code for qemuDomainSnapshotCurrent by better utilizing virDomainMoment helpers.
Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2019-May/msg00249.html
This is already fixed upstream by commit 9dd5bc151c51980807a29d59220290173f260b5a Refs: v5.3.0-62-g9dd5bc151c Author: Eric Blake <eblake> AuthorDate: Fri May 10 09:38:31 2019 -0500 Commit: Eric Blake <eblake> CommitDate: Fri May 10 10:50:16 2019 -0500 qemu: Fix regression with undefine --snapshots-metadata In refactoring the snapshot code to prepare for checkpoints, I changed qemuDomainMomentDiscardAll to take a callback that would handle the cleanup of either a snapshot or a checkpoint, but failed to set the callback on one of the two snapshot callers. As a result, 'virsh undefine $dom --snapshots-metadata' crashed on a NULL function dereference. Fixes: a487890d371b8cc3662c1717dfe07eea3f1ef1c0 Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1707708 Signed-off-by: Eric Blake <eblake> Acked-by: Peter Krempa <pkrempa>
Verified and PASSED [root@hp-dl320eg8-13 ~]# rpm -qa | egrep "qemu-kvm-4|libvirt-5" libvirt-5.4.0-1.module+el8.1.0+3304+7eb41d5f.x86_64 python3-libvirt-5.4.0-1.module+el8.1.0+3305+28419a35.x86_64 qemu-kvm-4.0.0-4.module+el8.1.0+3356+cda7f1ee.x86_64 [root@hp-dl320eg8-13 ~]# virsh start avocado-vt-vm1 Domain avocado-vt-vm1 started [root@hp-dl320eg8-13 ~]# for i in s1 s2 s3 s4; do virsh snapshot-create-as avocado-vt-vm1 $i --disk-only ;done Domain snapshot s1 created Domain snapshot s2 created Domain snapshot s3 created Domain snapshot s4 created [root@hp-dl320eg8-13 ~]# virsh snapshot-list avocado-vt-vm1 Name Creation Time State --------------------------------------------------- s1 2019-06-14 02:09:19 -0400 disk-snapshot s2 2019-06-14 02:09:19 -0400 disk-snapshot s3 2019-06-14 02:09:19 -0400 disk-snapshot s4 2019-06-14 02:09:19 -0400 disk-snapshot [root@hp-dl320eg8-13 ~]# virsh destroy avocado-vt-vm1 Domain avocado-vt-vm1 destroyed [root@hp-dl320eg8-13 ~]# virsh domstate avocado-vt-vm1 shut off [root@hp-dl320eg8-13 ~]# ps -C libvirtd PID TTY TIME CMD 16097 ? 00:00:00 libvirtd [root@hp-dl320eg8-13 ~]# virsh undefine avocado-vt-vm1 error: Failed to undefine domain avocado-vt-vm1 error: Requested operation is not valid: cannot delete inactive domain with 4 snapshots [root@hp-dl320eg8-13 ~]# ps -C libvirtd PID TTY TIME CMD 16097 ? 00:00:00 libvirtd [root@hp-dl320eg8-13 ~]# virsh undefine avocado-vt-vm1 --snapshots-metadata Domain avocado-vt-vm1 has been undefined [root@hp-dl320eg8-13 ~]# ps -C libvirtd PID TTY TIME CMD 16097 ? 00:00:00 libvirtd [root@hp-dl320eg8-13 ~]# virsh domstate avocado-vt-vm1 error: failed to get domain 'avocado-vt-vm1'
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3723