Description of problem: libvirtd crashed when starting guest after setting the max_threads_per_process to non default value Version-Release number of selected component (if applicable): libvirt-daemon-7.0.0-8.module+el8.4.0+10233+8b7fd9eb.x86_64 qemu-kvm-5.2.0-11.module+el8.4.0+10268+62bcbbed.x86_64 How reproducible: 100% Steps to Reproduce: Description of problem: libvirtd crashed when starting guest after setting the max_threads_per_process to a small value Version-Release number of selected component (if applicable): libvirt-daemon-7.0.0-8.module+el8.4.0+10233+8b7fd9eb.x86_64 qemu-kvm-5.2.0-11.module+el8.4.0+10268+62bcbbed.x86_64 How reproducible: 100% 1. define a guest # virt-install --connect qemu:///system -n lizhu --hvm --accelerate -r 1536 --vcpus=2 --os-variant rhel8.4 --disk path=/var/lib/libvirt/images/RHEL-8.4-x86_64-latest.qcow2,bus=virtio,format=qcow2 --network source=default,model=virtio --import --noreboot --noautoconsole --serial pty --memballoon model=virtio --graphics vnc --video cirrus --machine q35 Starting install... Domain creation completed. You can restart your domain by running: virsh --connect qemu:///system start lizhu 2. set the max_threads_per_process to a small value # grep -i max_threads /etc/libvirt/qemu.conf # If max_threads_per_process is set to a positive integer, libvirt # 16k. max_threads_per_process can be used to override default #max_threads_per_process = 0 max_threads_per_process = 1 3. restart libvirtd # systemctl restart libvirtd 4. check the libvirtd daemon status # systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-03-10 02:04:45 EST; 1min 22s ago Docs: man:libvirtd(8) https://libvirt.org Main PID: 94413 (libvirtd) Tasks: 23 (limit: 32768) Memory: 32.7M CGroup: /system.slice/libvirtd.service ├─ 7825 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper ├─ 7826 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─94413 /usr/sbin/libvirtd --timeout 120 5. try to start the above guest # virsh start lizhu error: Disconnected from qemu:///system due to end of file error: Failed to start domain 'lizhu' error: End of file while reading data: Input/output error 6. check the libvirtd daemon status again # systemctl status libvirtd ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2021-03-10 02:06:42 EST; 31s ago Docs: man:libvirtd(8) https://libvirt.org Main PID: 94723 (libvirtd) Tasks: 23 (limit: 32768) Memory: 37.6M CGroup: /system.slice/libvirtd.service ├─ 7825 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper ├─ 7826 /usr/sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper └─94723 /usr/sbin/libvirtd --timeout 120 Actual results: libvirtd crashed Expected results: libvirtd should not crash
Created attachment 1762239 [details] backtrace
The problem is in the format string used to construct a numeric gvariant. Unfortunately the code path isn't covered by tests. Glib also reports: (process:3561615): GLib-CRITICAL **: 14:32:08.663: 'llu' does not have a valid GVariant format string as a prefix (process:3561615): GLib-CRITICAL **: 14:32:08.663: g_variant_new_va: assertion 'valid_format_string (format_string, !endptr, NULL)' failed
Fixed upstream by: commit 7b8f78a3afcb51f74cc81b69226a3c49a49f0b9b Author: Peter Krempa <pkrempa> Date: Wed Mar 10 14:37:56 2021 +0100 virsystemdtest: Call at least one virSystemdCreateMachine with 'maxthreads' > 0 There was a bug in the code adding TasksMax property. It remained undetected because all tests used '0' for @maxthreads. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Pavel Hrdina <phrdina> commit 07c6e493b2466c47401b6eb8a3f9b7d2d20df83e Author: Peter Krempa <pkrempa> Date: Wed Mar 10 14:33:06 2021 +0100 virSystemdCreateMachine: Use proper format string for uint64_t when constructing gvariant g_variant_new_parsed uses '%t' for a uint64_t rather than printf-like %llu. Additionally ensure that the passed value is a uint64_t since the argument used is a 'unsigned int'. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1937287 Fixes: bf5f2ed09c2 Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Pavel Hrdina <phrdina>
Exception approved in review meeting on 12 Mar 2021.
Verified on: libvirt-7.0.0-9.module+el8.4.0+10326+5e50a3b6.x86_64 Result: PASS Cgroup1 ENV: [root@lenovo-sr630-10 ~]# grep -i ^max_threads /etc/libvirt/qemu.conf max_threads_per_process = 2 [root@lenovo-sr630-10 ~]# systemctl restart libvirtd [root@lenovo-sr630-10 ~]# virsh start gls error: Failed to start domain 'gls' error: internal error: process exited while connecting to monitor: qemu: qemu_thread_create: Resource temporarily unavailable [root@lenovo-sr630-10 ~]# grep -i ^max_threads /etc/libvirt/qemu.conf max_threads_per_process = 128 [root@lenovo-sr630-10 ~]# systemctl restart libvirtd [root@lenovo-sr630-10 ~]# virsh start gls Domain 'gls' started [root@lenovo-sr630-10 ~]# cat /sys/fs/cgroup/pids/machine.slice/machine-qemu\\x2d1\\x2dgls.scope/pids.max 128 [root@lenovo-sr630-10 ~]# grep -i ^max_threads /etc/libvirt/qemu.conf max_threads_per_process = 99999 [root@lenovo-sr630-10 ~]# systemctl restart libvirtd [root@lenovo-sr630-10 ~]# virsh start gls Domain 'gls' started [root@lenovo-sr630-10 ~]# cat /sys/fs/cgroup/pids/machine.slice/machine-qemu\\x2d1\\x2dgls.scope/pids.max 99999 Cgroup2 ENV: [root@dell-per740-01 ~]# grep -i ^max_threads /etc/libvirt/qemu.conf max_threads_per_process = 100 [root@dell-per740-01 ~]# systemctl restart libvirtd [root@dell-per740-01 ~]# virsh start vm1 Domain 'vm1' started [root@dell-per740-01 ~]# grep -i ^max_threads /etc/libvirt/qemu.conf max_threads_per_process = 2 [root@dell-per740-01 ~]# systemctl restart libvirtd [root@dell-per740-01 ~]# virsh start vm1 error: Failed to start domain 'vm1' error: internal error: process exited while connecting to monitor: qemu: qemu_thread_create: Resource temporarily unavailable [root@dell-per740-01 ~]# grep -i ^max_threads /etc/libvirt/qemu.conf max_threads_per_process = 99999 [root@dell-per740-01 ~]# systemctl restart libvirtd [root@dell-per740-01 ~]# virsh start vm1 Domain 'vm1' started [root@dell-per740-01 ~]# cat /sys/fs/cgroup/machine.slice/machine-qemu\\x2d1\\x2dvm1.scope/pids.max 99999
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098