description: libvirtd crashed when undefine vm after blockpull job finished Version: libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7.x86_64 How reproducible: 100% steps: 1. have a running vm (.libvirt-ci-venv-ci-runtest-y6VZ9O) [root@dell-per730-66 images]# virsh domstate avocado-vt-vm1 running 2. create a external snapshot for it (.libvirt-ci-venv-ci-runtest-y6VZ9O) [root@dell-per730-66 images]# virsh snapshot-create-as avocado-vt-vm1 snap1 --disk-only Domain snapshot snap1 created 3. do blockpull (.libvirt-ci-venv-ci-runtest-y6VZ9O) [root@dell-per730-66 images]# virsh blockpull avocado-vt-vm1 vda --wait Pull complete 4. remove the snapshot (.libvirt-ci-venv-ci-runtest-y6VZ9O) [root@dell-per730-66 images]# virsh snapshot-delete avocado-vt-vm1 snap1 --metadata Domain snapshot snap1 deleted 5. destroy the vm (.libvirt-ci-venv-ci-runtest-y6VZ9O) [root@dell-per730-66 images]# virsh destroy avocado-vt-vm1 Domain avocado-vt-vm1 destroyed 6. undefine the vm, crash happens (.libvirt-ci-venv-ci-runtest-y6VZ9O) [root@dell-per730-66 images]# virsh undefine avocado-vt-vm1 error: Disconnected from qemu:///system due to end of file error: Failed to undefine domain avocado-vt-vm1 error: End of file while reading data: Input/output error Additional info: GDB backtrace info as follow: Thread 2 "libvirtd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fcfe0325700 (LWP 6012)] 0x00007fcfa0a84bcb in qemuDomainLookupByName () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so (gdb) t a a bt Thread 18 (Thread 0x7fcf877fe700 (LWP 6216)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fcf87fff700 (LWP 6079)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfa13beb34 in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so #3 0x00007fcfea1b879a in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fcf9e960700 (LWP 6026)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fcf9f161700 (LWP 6025)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fcf9f962700 (LWP 6024)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fcfa0163700 (LWP 6023)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fcfa0964700 (LWP 6022)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fcfd7fff700 (LWP 6021)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b9474 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fcfdcb1e700 (LWP 6020)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b9474 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fcfdd31f700 (LWP 6019)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b9474 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fcfddb20700 (LWP 6018)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b9474 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fcfde321700 (LWP 6017)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b9474 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fcfdeb22700 (LWP 6016)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fcfdf323700 (LWP 6015)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fcfd7323700 (LWP 6014)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fcfdfb24700 (LWP 6013)): #0 0x00007fcfe692348c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fcfea1b89ea in virCondWait () from /lib64/libvirt.so.0 #2 0x00007fcfea1b94c3 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #4 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #5 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fcfe0325700 (LWP 6012)): #0 0x00007fcfa0a84bcb in qemuDomainLookupByName () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #1 0x00007fcfea34c0ff in virDomainLookupByName () from /lib64/libvirt.so.0 #2 0x000055d50884174c in remoteDispatchDomainLookupByNameHelper () #3 0x00007fcfea295839 in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #4 0x00007fcfea29a9dc in virNetServerHandleJob () from /lib64/libvirt.so.0 #5 0x00007fcfea1b93e0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #6 0x00007fcfea1b876c in virThreadHelper () from /lib64/libvirt.so.0 #7 0x00007fcfe691d2de in start_thread () from /lib64/libpthread.so.0 #8 0x00007fcfe664ee93 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fcfeacdcc00 (LWP 6009)): #0 0x00007fcfe6643f31 in poll () from /lib64/libc.so.6 #1 0x00007fcfea15c77a in virEventPollRunOnce () from /lib64/libvirt.so.0 #2 0x00007fcfea15b325 in virEventRunDefaultImpl () from /lib64/libvirt.so.0 #3 0x00007fcfea29a215 in virNetDaemonRun () from /lib64/libvirt.so.0 #4 0x000055d50881bfbb in main ()
*** Bug 1775488 has been marked as a duplicate of this bug. ***
The issue is a double-unref in qemuDomainBlockPull caused by commit commit 421c9550f5446729b513ee50f5c44e6f6969b5a2 Author: Peter Krempa <pkrempa> Date: Thu Sep 26 13:50:16 2019 +0200 qemu: Don't repeat virDomainObjEndAPI in qemuDomainBlockPull Add a 'cleanup' label and use jumps as we do in other places. Any other operation following the removal of the last reference then accesses freed data.
Fixed upstream: commit a10eb613404fedb4ce44ec47848710fdb3d7a91d (HEAD -> master, origin/master, origin/HEAD) Author: Peter Krempa <pkrempa> Date: Wed Nov 27 13:40:14 2019 +0100 Revert "qemu: Don't repeat virDomainObjEndAPI in qemuDomainBlockPull" This reverts commit 421c9550f5446729b513ee50f5c44e6f6969b5a2 qemuDomainBlockPullCommon calls virDomainObjEndAPI internally so the original commit made us shed two references of @vm instead of one getting us into a premature free of @vm. This is not a straight revert as qemuDomainBlockPull was modified meanwhile. I've also added a warning comment that @vm is consumed. v5.10.0-rc1-1-ga10eb61340
Verified on: libvirt-5.10.0-1.module+el8.2.0+5040+bd433686.x86_64 [root@libvirt-rhel-8 ~]# virsh start avocado-vt-vm1 Domain avocado-vt-vm1 started [root@libvirt-rhel-8 ~]# virsh snapshot-create-as avocado-vt-vm1 snap1 --disk-only Domain snapshot snap1 created [root@libvirt-rhel-8 ~]# virsh blockpull avocado-vt-vm1 vda --wait Pull complete [root@libvirt-rhel-8 ~]# virsh snapshot-delete avocado-vt-vm1 snap1 --metadata Domain snapshot snap1 deleted [root@libvirt-rhel-8 ~]# virsh destroy avocado-vt-vm1 Domain avocado-vt-vm1 destroyed [root@libvirt-rhel-8 ~]# virsh undefine avocado-vt-vm1 Domain avocado-vt-vm1 has been undefined [root@libvirt-rhel-8 ~]# virsh list --all Id Name State ---------------------- 1 gls running
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2017