Bug 1808293

Summary: libvirtd crashed after setting blkio weight
Product: Red Hat Enterprise Linux Advanced Virtualization Reporter: Lili Zhu <lizhu>
Component: libvirtAssignee: Pavel Hrdina <phrdina>
Status: CLOSED ERRATA QA Contact: yisun
Severity: low Docs Contact:
Priority: medium    
Version: 8.2CC: chhu, jdenemar, jsuchane, lcheng, mkletzan, phrdina, pkrempa, virt-maint, xuzhang, yisun
Target Milestone: rcKeywords: Automation, Regression, Triaged
Target Release: 8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-6.6.0-4.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 17:46:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lili Zhu 2020-02-28 08:22:25 UTC 
Description of problem:
libvirtd crashed after setting blkio weight

Version-Release number of selected component (if applicable):
libvirt-daemon-6.0.0-7.el8.x86_64
qemu-kvm-4.2.0-12.module+el8.2.0+5858+afd073bc.x86_64
kernel-4.18.0-184.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. check whether the blkio cgroup is mounted
# mount | grep cgroup | grep blkio
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)

2. check libvirtd pid
# pidof libvirtd
56249

3. set blkio weight 
# virsh blkiotune rhel8.2 --weight 300
error: Disconnected from qemu:///system due to end of file
error: Unable to change blkio parameters
error: End of file while reading data: Input/output error

4. check libvirtd pid again
# pidof libvirtd
57792

Actual results:
libvirtd crashed

Additional info:
1. get blkio parameters
# virsh blkiotune rhel8.2 
error: Unable to get blkio parameters
error: Requested operation is not valid: blkio cgroup isn't mounted
Comment 1 Peter Krempa 2020-02-28 08:38:08 UTC 
*** Bug 1808297 has been marked as a duplicate of this bug. ***
Comment 2 Peter Krempa 2020-02-28 08:38:11 UTC 
*** Bug 1808295 has been marked as a duplicate of this bug. ***
Comment 3 Peter Krempa 2020-02-28 08:38:15 UTC 
*** Bug 1808294 has been marked as a duplicate of this bug. ***
Comment 4 Peter Krempa 2020-02-28 08:38:55 UTC 
Could you please attach the backtrace?
Comment 5 Lili Zhu 2020-02-28 08:45:50 UTC 
(gdb) t a a bt

Thread 18 (Thread 0x7f42427fc700 (LWP 4081)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc1b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc188) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc188, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7f4201fc9700 (LWP 4331)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f41e0333754) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f41e0333700, cond=0x7f41e0333728) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f41e0333728, mutex=0x7f41e0333700) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7f42003ed700 (LWP 4088)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec2863a30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec28639e0, cond=0x556ec2863a08) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec2863a08, mutex=0x556ec28639e0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7f4208bee700 (LWP 4087)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec2863a30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec28639e0, cond=0x556ec2863a08) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec2863a08, mutex=0x556ec28639e0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

--Type <RET> for more, q to quit, c to continue without paging--c
Thread 14 (Thread 0x7f4209bf0700 (LWP 4085)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec2863a30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec28639e0, cond=0x556ec2863a08) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec2863a08, mutex=0x556ec28639e0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f42027ca700 (LWP 4139)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f41e00fa86c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f41e00fa800, cond=0x7f41e00fa840) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f41e00fa840, mutex=0x7f41e00fa800) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f420afb21da in udevEventHandleThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nodedev.so
#5  0x00007f425425db9a in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f4248d15700 (LWP 4076)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc114) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc0e8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc0e8, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f420a3f1700 (LWP 4084)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec2863a30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec28639e0, cond=0x556ec2863a08) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec2863a08, mutex=0x556ec28639e0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f4249516700 (LWP 4075)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc110) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc0e8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc0e8, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f4240514700 (LWP 4077)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc110) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc0e8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc0e8, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f4249d17700 (LWP 4074)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc110) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc0e8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc0e8, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f42093ef700 (LWP 4086)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec2863a30) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec28639e0, cond=0x556ec2863a08) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec2863a08, mutex=0x556ec28639e0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f4242ffd700 (LWP 4080)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc1b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc188) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc188, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f4255021c00 (LWP 4073)):
#0  0x00007f4250238f21 in __GI___poll (fds=0x556ec283f2e0, nfds=12, timeout=4999) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f42541cceb8 in virEventPollRunOnce () from /lib64/libvirt.so.0
#2  0x00007f42541cb041 in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#3  0x00007f42543c90c5 in virNetDaemonRun () from /lib64/libvirt.so.0
#4  0x0000556ec0d4871a in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1430

Thread 4 (Thread 0x7f42417fa700 (LWP 4083)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc1b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc188) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc188, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f4241ffb700 (LWP 4082)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc1b0) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc188) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc188, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f42437fe700 (LWP 4079)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x556ec27fc1b4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x556ec27fc0c0, cond=0x556ec27fc188) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x556ec27fc188, mutex=0x556ec27fc0c0) at pthread_cond_wait.c:655
#3  0x00007f425425df1e in virCondWait () from /lib64/libvirt.so.0
#4  0x00007f425425ee34 in virThreadPoolWorker () from /lib64/libvirt.so.0
#5  0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#6  0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f4243fff700 (LWP 4078)):
#0  0x00007f42541a1bf9 in virCgroupBackendForController () from /lib64/libvirt.so.0
#1  0x00007f425419a38d in virCgroupSetBlkioWeight () from /lib64/libvirt.so.0
#2  0x00007f420a5bf75a in qemuDomainSetBlkioParameters () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#3  0x00007f42544d3852 in virDomainSetBlkioParameters () from /lib64/libvirt.so.0
#4  0x0000556ec0d613f9 in remoteDispatchDomainSetBlkioParameters (args=0x7f422000a3e0, rerr=0x7f4243ffe8c0, msg=0x556ec2884b20, client=<optimized out>, server=0x556ec27fbe00) at ./remote/remote_daemon_dispatch_stubs.h:10567
#5  remoteDispatchDomainSetBlkioParametersHelper (server=0x556ec27fbe00, client=<optimized out>, msg=0x556ec2884b20, rerr=0x7f4243ffe8c0, args=0x7f422000a3e0, ret=0x0) at ./remote/remote_daemon_dispatch_stubs.h:10537
#6  0x00007f42543c1b39 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#7  0x00007f42543c9630 in virNetServerProcessMsg () from /lib64/libvirt.so.0
#8  0x00007f42543c9a9d in virNetServerHandleJob () from /lib64/libvirt.so.0
#9  0x00007f425425eef4 in virThreadPoolWorker () from /lib64/libvirt.so.0
#10 0x00007f425425db52 in virThreadHelper () from /lib64/libvirt.so.0
#11 0x00007f42505122de in start_thread (arg=<optimized out>) at pthread_create.c:486
#12 0x00007f4250243e83 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Comment 6 yisun 2020-03-02 15:02:42 UTC 
just make things clear that this happened with systemd-239-27.el8 but not systemd-239-25.el8 ( libvirt version not changed )
And we have a related issue reported to systemd which blocking our gating job - Bug 1808940 - cpuset controller group not created for qemu vm
Comment 7 yisun 2020-03-09 05:44:00 UTC 
Hi Pavel,
When I did a test with systemd scratch build, this libvirtd crash is not reproduced either.
Detailed test info is here: 
https://bugzilla.redhat.com/show_bug.cgi?id=1808940#c12
Will you still resolve the libvirtd crash issue with a problematic systemd in future? Or pls just set this bz to TESTONLY, thx.
Comment 8 Pavel Hrdina 2020-03-09 08:34:00 UTC 
We should fix the crash as well because it should not happen.
Comment 11 Pavel Hrdina 2020-08-12 06:59:17 UTC 
Upstream patch posted:

https://www.redhat.com/archives/libvir-list/2020-August/msg00434.html
Comment 12 Pavel Hrdina 2020-08-13 14:03:39 UTC 
Upstream commit:

commit a6886aafacaf25d1ce667ae961264ae1eb69900f
Author: Pavel Hrdina <phrdina>
Date:   Tue Aug 11 15:56:54 2020 +0200

    qemu: fix crash in qemuDomainSetBlkioParameters without cgroups
Comment 18 yisun 2020-09-02 09:12:07 UTC 
Tested on:
libvirt-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64
systemd-239-39.el8.x86_64

Result:
PASS

[root@dell-per740xd-11 snd]# mount | grep cgroup | grep blkio
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)

[root@dell-per740xd-11 snd]# virsh list
 Id   Name   State
----------------------
 14   gls    running
 46   vm1    running

[root@dell-per740xd-11 snd]# virsh blkiotune vm1 --weight 300

[root@dell-per740xd-11 snd]# virsh blkiotune vm1
weight         : 300
device_weight  : 
device_read_iops_sec: 
device_write_iops_sec: 
device_read_bytes_sec: 
device_write_bytes_sec:
Comment 21 errata-xmlrpc 2020-11-17 17:46:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (virt:8.3 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5137