Description of problem: libvirtd crash when trying to save more than one numa cell without 'cpus' setting Version-Release number of selected component (if applicable): libvirt-6.6.0-6.module+el8.3.0+8125+aefcf088.x86_64 qemu-kvm-5.1.0-8.module+el8.3.0+8141+3cd9cd43.x86_64 How reproducible: 100% Steps to Reproduce: 1.Edit guest xml: #virsh edit vm1 ... <cpu> <numa> <cell id='0' memory='2097152' unit='KiB' memAccess='shared'/> <cell id='1' memory='2097152' unit='KiB' memAccess='shared'/> </numa> </cpu> save xml: error: Disconnected from qemu:///system due to end of file error: End of file while reading data: Input/output error Failed. Try again? [y,n,i,f,?]: Actual results: libvirtd crash when trying to save more than one numa cell without 'cpus' setting Expected results: libvirtd should not crash. Additional info: 1.Backtrace of libvirtd: Core was generated by `/usr/sbin/libvirtd --timeout 120'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f0044876318 in virBitmapSubtract () from /lib64/libvirt.so.0 [Current thread is 1 (Thread 0x7f003a479700 (LWP 105527))] (gdb) t a a bt Thread 17 (Thread 0x7f00337fe700 (LWP 105532)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc79174570) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc79174548) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc79174548, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f63d8 in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 16 (Thread 0x7f0038c76700 (LWP 105531)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791744d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc791744a8) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791744a8, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 15 (Thread 0x7efff95f0700 (LWP 105593)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc7917f794) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc7917f728, cond=0x55bc7917f768) --Type <RET> for more, q to quit, c to continue without paging-- at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc7917f768, mutex=0x55bc7917f728) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00234563fc in udevEventHandleThread () at /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 14 (Thread 0x7f0032ffd700 (LWP 105533)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc79174570) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc79174548) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc79174548, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f63d8 in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 13 (Thread 0x7f0045420c00 (LWP 105526)): #0 0x00007f00409a5ca1 in __GI___poll (fds=0x55bc791dfa60, nfds=10, timeout=5000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007f00415049b6 in g_main_context_poll (priority=<optimized out>, n_fds=10, fds=0x55bc791dfa60, timeout=<optimized out>, context=0x55bc791df400) at gmain.c:4203 #2 0x00007f00415049b6 in g_main_context_iterate (context=context@entry=0x55bc791df400, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at gmain.c:3897 #3 0x00007f0041504ae0 in g_main_context_iteration (context=0x55bc791df400, may_block=1) at gmain.c:3963 #4 0x00007f004489af44 in virEventGLibRunOnce () at /lib64/libvirt.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #5 0x00007f00449e64a5 in virNetDaemonRun () at /lib64/libvirt.so.0 #6 0x000055bc77d804df in main () Thread 12 (Thread 0x7f0033fff700 (LWP 105530)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791744d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc791744a8) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791744a8, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 11 (Thread 0x7f002120a700 (LWP 105540)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791f8190) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc791f8140, cond=0x55bc791f8168) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791f8168, mutex=0x55bc791f8140) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 10 (Thread 0x7f0022a0d700 (LWP 105537)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791f8190) --Type <RET> for more, q to quit, c to continue without paging-- at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc791f8140, cond=0x55bc791f8168) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791f8168, mutex=0x55bc791f8140) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 9 (Thread 0x7f0039477700 (LWP 105529)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791744d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc791744a8) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791744a8, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 8 (Thread 0x7f002220c700 (LWP 105538)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791f8190) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc791f8140, cond=0x55bc791f8168) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791f8168, mutex=0x55bc791f8140) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 7 (Thread 0x7f0020a09700 (LWP 105541)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791f8190) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc791f8140, cond=0x55bc791f8168) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791f8168, mutex=0x55bc791f8140) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 6 (Thread 0x7f0031ffb700 (LWP 105535)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc79174574) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc79174548) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc79174548, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f63d8 in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 --Type <RET> for more, q to quit, c to continue without paging-- Thread 5 (Thread 0x7f0039c78700 (LWP 105528)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791744d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc791744a8) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791744a8, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 4 (Thread 0x7f00327fc700 (LWP 105534)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc79174570) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc79174548) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc79174548, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f63d8 in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 3 (Thread 0x7f0021a0b700 (LWP 105539)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc791f8190) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc791f8140, cond=0x55bc791f8168) at pthread_cond_wait.c:502 --Type <RET> for more, q to quit, c to continue without paging-- #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc791f8168, mutex=0x55bc791f8140) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f641b in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 2 (Thread 0x7f00317fa700 (LWP 105536)): #0 0x00007f00410a12fc in futex_wait_cancelable (private=0, expected=0, futex_word=0x55bc79174574) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 0x00007f00410a12fc in __pthread_cond_wait_common (abstime=0x0, mutex=0x55bc79174480, cond=0x55bc79174548) at pthread_cond_wait.c:502 #2 0x00007f00410a12fc in __pthread_cond_wait (cond=0x55bc79174548, mutex=0x55bc79174480) at pthread_cond_wait.c:655 #3 0x00007f00448f57fa in virCondWait () at /lib64/libvirt.so.0 #4 0x00007f00448f63d8 in virThreadPoolWorker () at /lib64/libvirt.so.0 #5 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #6 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #7 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Thread 1 (Thread 0x7f003a479700 (LWP 105527)): #0 0x00007f0044876318 in virBitmapSubtract () at /lib64/libvirt.so.0 #1 0x00007f004496b023 in virDomainNumaFillCPUsInNode () at /lib64/libvirt.so.0 #2 0x00007f0022aa4afb in qemuDomainDefNumaCPUsRectify () at /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007f0022ab4812 in qemuDomainDefPostParse () at /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007f0044960b6a in virDomainDefPostParse () at /lib64/libvirt.so.0 #5 0x00007f0044961a8c in virDomainDefParseNode () at /lib64/libvirt.so.0 #6 0x00007f0044961b9f in virDomainDefParse () at /lib64/libvirt.so.0 #7 0x00007f0022b35aac in qemuDomainDefineXMLFlags () at /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007f0044ab3084 in virDomainDefineXMLFlags () at /lib64/libvirt.so.0 --Type <RET> for more, q to quit, c to continue without paging-- #9 0x000055bc77d8c83e in remoteDispatchDomainDefineXMLFlagsHelper () #10 0x00007f00449e1aa9 in virNetServerProgramDispatch () at /lib64/libvirt.so.0 #11 0x00007f00449e6ca6 in virNetServerHandleJob () at /lib64/libvirt.so.0 #12 0x00007f00448f634f in virThreadPoolWorker () at /lib64/libvirt.so.0 #13 0x00007f00448f594b in virThreadHelper () at /lib64/libvirt.so.0 #14 0x00007f004109b14a in start_thread (arg=<optimized out>) at pthread_create.c:479 #15 0x00007f00409b0f23 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2020-September/msg01550.html
Pushed upstream as: 2752a67826 virDomainNumaFillCPUsInNode: Skip over NUMA nodes without vCPUs v6.8.0-19-g2752a67826
Moving to POST per comment 3.
Tested with libvirt-daemon-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 & qemu-kvm-5.2.0-0.module+el8.4.0+8855+a9e237a9.x86_64 s1: step 1.Edit guest xml: #virsh edit vm1 <vcpu placement='static'>4</vcpu> <cpu> <numa> <cell id='0' memory='1' unit='GiB' memAccess='shared'/> <cell id='1' memory='1' unit='GiB' memAccess='shared'/> </numa> </cpu> 2. Save & dump domain xml: <numa> <cell id='0' cpus='0-3' memory='1048576' unit='KiB' memAccess='shared'/> <cell id='1' memory='1048576' unit='KiB' memAccess='shared'/> </numa> Result: No libvirtd crash.
Verified in versions: libvirt-daemon-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64 qemu-kvm-5.2.0-2.module+el8.4.0+9186+ec44380f.x86_64 s1: Save domain without setting vcpus in numa nodes and domain still have non-assigned vcpus. step 1.Edit guest xml: #virsh edit vm1 <vcpu placement='static'>4</vcpu> <cpu> <numa> <cell id='0' memory='1' unit='GiB' memAccess='shared'/> <cell id='1' memory='1' unit='GiB' memAccess='shared'/> </numa> </cpu> 2. Save & dump domain xml: <numa> <cell id='0' cpus='0-3' memory='1048576' unit='KiB' memAccess='shared'/> <cell id='1' memory='1048576' unit='KiB' memAccess='shared'/> </numa> 3. Start domain. # virsh start vm1 Domain 'vm1' started Result: No libvirtd crash.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098