Description of problem: RHEL-8 intends to ship with the cgroups v2 subsystem enabled out of the box, instead of cgroups v1. While still based on a virtual filesystem, this has a quite significantly different design and is not compatible with what libvirt does for v1. We thus need to rewrite our code to allow it to support v2 as an alternative to v1. This is a blocker for RHEL-8, because without this v2 support libvirt's resource mgmt will be dead in the water. v2 is available in Fedora kernels, but is not enabled by default - it needs a special boot option to enable. https://www.kernel.org/doc/Documentation/cgroup-v2.txt See also previous thread at: https://www.redhat.com/archives/libvir-list/2016-October/msg00921.html NB, since that thread the kernel *has* merged a CPU controller for v2 that allows per-thread placement, so the major blocking item discussed there is resolved. Version-Release number of selected component (if applicable): 3.9.0
NB, it occurred to me that it might be worth someone emailing Tejun at facebook, to find out if they did go ahead and implement libvirt cgroups v2 support out of tree as his mail suggested they might. If so they might be willing to contribute it back, at least as a starting point.
The current requirement documented here: https://docs.engineering.redhat.com/display/RHELPLAN/Support+control+group+v2 states: * CGroup v1 will be the default behavior in RHEL8, for backward compatability reason. So far no containers, vms support cgroupv2 yet. * All containers produced for RHEL7 and earlier will work on RHEL8. therefore, cgroupv1 will be the default cgroup to be mounted by default on RHEL8. Still it is desirable feature for libvirt.
(In reply to Jaroslav Suchanek from comment #2) > The current requirement documented here: > https://docs.engineering.redhat.com/display/RHELPLAN/Support+control+group+v2 > > states: > * CGroup v1 will be the default behavior in RHEL8, for backward > compatability reason. So far no containers, vms support cgroupv2 yet. > * All containers produced for RHEL7 and earlier will work on RHEL8. > therefore, cgroupv1 will be the default cgroup to be mounted by default on > RHEL8. > > Still it is desirable feature for libvirt. CGroup v2 is still going to be a fully supported part of RHEL-8 - just still defaulting to v1 out of the box. To be able to consider it fully supported, we need libvirt not to break, when an admin turns on v2, so this still looks like a blocking requirement for RHEL-8 GA wrt libvirt.
Pavel, What is the target upstream version for this feature? Is the work in progress upstream? Thanks.
Hi Karen, I would like to finish it before next release (4.6.0) but I'll see how it will go. The existing code needs some cleanup and I'm already working on it.
Libvirt and cgroupv2 is a blocker for RHEL 8.0. Otherwise, customers who use KVM cannot use cgroupv2, including Red Hat layered products. Thanks.
Pavel, Any update on cgroup v2 support? I see you posted v2 on Oct 5. Thanks. [libvirt] [PATCH v2 00/53] implement cgroup v2 support https://www.redhat.com/archives/libvir-list/2016-October/msg00921.html
Hi Karen, so that patch series is pushed into upstream so libvirt is now able to work with cgroups v2. However, there is still missing implementation for devices controller. I already have some patches prepared but there are still some issues that needs to be solved before I post it to mailing list.
The cgroups v2 controller is blocked by systemd BZ 1724617 where they need to add support for it.
Hi Pavel, I've tried some basic scenarios but got blocked, pls help to check if anything wrong with my usage, thx Test with following pkgs； 1. [root@dell-per740-07 /]# rpm -qa | egrep "^kernel-4|^libvirt-5|^qemu-kvm-4|^libcgroup" libcgroup-0.41-19.el8.x86_64 kernel-4.18.0-107.el8.x86_64 libvirt-5.5.0-1.module+el8.1.0+3580+d7f6488d.x86_64 libcgroup-tools-0.41-19.el8.x86_64 qemu-kvm-4.0.0-5.module+el8.1.0+3622+5812d9bf.x86_64 2. Cgroupv2 enabled after rebooting with following kernel options: 2.1 Reboot... 2.2 [root@dell-per740-07 /]# grub2-editenv - list | grep kernelopts kernelopts=root=/dev/mapper/rhel_dell--per740--07-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per740--07-swap rd.lvm.lv=rhel_dell-per740-07/root rd.lvm.lv=rhel_dell-per740-07/swap console=ttyS0,115200n81 cgroup_no_v1=all 3. Mount cgroup2 fs to /cgroup2/ [root@dell-per740-07 /]# mkdir cgroup2 [root@dell-per740-07 /]# mount -t cgroup2 none /cgroup2/ [root@dell-per740-07 /]# mount | grep cgroup2 none on /cgroup2 type cgroup2 (rw,relatime,seclabel) [root@dell-per740-07 /]# cd /cgroup2/ [root@dell-per740-07 cgroup2]# cat cgroup.controllers cpuset cpu io memory pids rdma 4. Start a vm and try some cgroup related virsh cmds [root@dell-per740-07 ~]# virsh list Id Name State ---------------------- 1 vm2 running [root@dell-per740-07 ~]# ps -ef | grep vm2 qemu 3449 1 44 04:04 ? 00:00:24 /usr/libexec/qemu-kvm -name guest=vm2 5. Block iodine cmd [root@dell-per740-07 ~]# virsh blkiotune vm2 --weight 999 error: Disconnected from qemu:///system due to end of file error: Unable to change blkio parameters error: End of file while reading data: Input/output error <==== qemu instance crashed here [root@dell-per740-07 ~]# virsh list Id Name State -------------------- [root@dell-per740-07 ~]# virsh dumpxml vm2 | grep 999 | wc -l 0 <===== the block weight not set to vm2 [root@dell-per740-07 ~]# virsh start vm2 error: Failed to start domain vm2 error: Unable to read from '/cgroup2/machine.slice/machine-qemu\x2d2\x2dvm2.scope/cgroup.controllers': No such file or directory <===== the vm cannot be started anymore [root@dell-per740-07 cgroup2]# ll /cgroup2/ total 0 -r--r--r--. 1 root root 0 Jul 9 04:06 cgroup.controllers -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.max.depth -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.max.descendants -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.procs -r--r--r--. 1 root root 0 Jul 9 04:09 cgroup.stat -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.subtree_control -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.threads -rw-r--r--. 1 root root 0 Jul 9 04:09 cpu.pressure -r--r--r--. 1 root root 0 Jul 9 04:09 cpuset.cpus.effective -r--r--r--. 1 root root 0 Jul 9 04:09 cpuset.mems.effective -rw-r--r--. 1 root root 0 Jul 9 04:09 io.pressure -rw-r--r--. 1 root root 0 Jul 9 04:09 memory.pressure <===== there is no machine slice, who's in charge to create the slice here? 6. Cpu cmd [root@dell-per740-07 ~]# cat /cgroup2/cgroup.controllers cpuset cpu io memory pids rdma [root@dell-per740-07 ~]# virsh schedinfo vm2 --set cpu_shares=2048 Scheduler : Unknown error: Requested operation is not valid: cgroup CPU controller is not mounted
(In reply to yisun from comment #20) > Hi Pavel, > I've tried some basic scenarios but got blocked, pls help to check if > anything wrong with my usage, thx > > Test with following pkgs； > 1. [root@dell-per740-07 /]# rpm -qa | egrep > "^kernel-4|^libvirt-5|^qemu-kvm-4|^libcgroup" > libcgroup-0.41-19.el8.x86_64 > kernel-4.18.0-107.el8.x86_64 > libvirt-5.5.0-1.module+el8.1.0+3580+d7f6488d.x86_64 > libcgroup-tools-0.41-19.el8.x86_64 > qemu-kvm-4.0.0-5.module+el8.1.0+3622+5812d9bf.x86_64 libcgroup is not used by libvirt for a long time so you might need to update your scripts/knowledge base to not include it. > 2. Cgroupv2 enabled after rebooting with following kernel options: > 2.1 Reboot... > 2.2 [root@dell-per740-07 /]# grub2-editenv - list | grep kernelopts > kernelopts=root=/dev/mapper/rhel_dell--per740--07-root ro crashkernel=auto > resume=/dev/mapper/rhel_dell--per740--07-swap > rd.lvm.lv=rhel_dell-per740-07/root rd.lvm.lv=rhel_dell-per740-07/swap > console=ttyS0,115200n81 cgroup_no_v1=all My understanding from systemd man page is that on systemd OSes you should put 'systemd.unified_cgroup_hierarchy=1' on the kernel command line. > 3. Mount cgroup2 fs to /cgroup2/ > [root@dell-per740-07 /]# mkdir cgroup2 > [root@dell-per740-07 /]# mount -t cgroup2 none /cgroup2/ > [root@dell-per740-07 /]# mount | grep cgroup2 > none on /cgroup2 type cgroup2 (rw,relatime,seclabel) This step feels wrong, on systemd OSes systemd is responsible for cgroups and you should not need to mount it manually. > [root@dell-per740-07 /]# cd /cgroup2/ > [root@dell-per740-07 cgroup2]# cat cgroup.controllers > cpuset cpu io memory pids rdma > > 4. Start a vm and try some cgroup related virsh cmds > > [root@dell-per740-07 ~]# virsh list > Id Name State > ---------------------- > 1 vm2 running > > [root@dell-per740-07 ~]# ps -ef | grep vm2 > qemu 3449 1 44 04:04 ? 00:00:24 /usr/libexec/qemu-kvm -name > guest=vm2 > > > 5. Block iodine cmd > [root@dell-per740-07 ~]# virsh blkiotune vm2 --weight 999 > error: Disconnected from qemu:///system due to end of file > error: Unable to change blkio parameters > error: End of file while reading data: Input/output error > <==== qemu instance crashed here This looks more like libvirtd crashed, can you please provide backtrace of the crash? > [root@dell-per740-07 ~]# virsh list > Id Name State > -------------------- > > [root@dell-per740-07 ~]# virsh dumpxml vm2 | grep 999 | wc -l > 0 > <===== the block weight not set to vm2 > > [root@dell-per740-07 ~]# virsh start vm2 > error: Failed to start domain vm2 > error: Unable to read from > '/cgroup2/machine.slice/machine-qemu\x2d2\x2dvm2.scope/cgroup.controllers': > No such file or directory > <===== the vm cannot be started anymore > > > [root@dell-per740-07 cgroup2]# ll /cgroup2/ > total 0 > -r--r--r--. 1 root root 0 Jul 9 04:06 cgroup.controllers > -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.max.depth > -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.max.descendants > -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.procs > -r--r--r--. 1 root root 0 Jul 9 04:09 cgroup.stat > -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.subtree_control > -rw-r--r--. 1 root root 0 Jul 9 04:09 cgroup.threads > -rw-r--r--. 1 root root 0 Jul 9 04:09 cpu.pressure > -r--r--r--. 1 root root 0 Jul 9 04:09 cpuset.cpus.effective > -r--r--r--. 1 root root 0 Jul 9 04:09 cpuset.mems.effective > -rw-r--r--. 1 root root 0 Jul 9 04:09 io.pressure > -rw-r--r--. 1 root root 0 Jul 9 04:09 memory.pressure > <===== there is no machine slice, who's in charge to create the slice here? As I stated above, systemd is the one managing cgroups and systemd is the one creating machine.slice for us. Might happen because of the manual mount. > 6. Cpu cmd > [root@dell-per740-07 ~]# cat /cgroup2/cgroup.controllers > cpuset cpu io memory pids rdma > > [root@dell-per740-07 ~]# virsh schedinfo vm2 --set cpu_shares=2048 > Scheduler : Unknown > error: Requested operation is not valid: cgroup CPU controller is not mounted Yes, this is a known limitation, if there is a real-time task running on the host the CPU controller cannot be enabled. We need to document it somehow.
For the crash issue: Steps: 1. adding "cgroup_no_v1=all" in kernel cmd line, and reboot host 2. having a running vm [root@dell-per740-07 /]# virsh list Id Name State -------------------------------- 2 avocado-vt-vm1 running 3. mount cgroupv2 to some path [root@dell-per740-07 /]# mount -t cgroup2 cgroup2 /cgroup2/ 4. do blkiotune to that vm [root@dell-per740-07 /]# virsh blkiotune avocado-vt-vm1 --weight 999 error: Disconnected from qemu:///system due to keepalive timeout error: Unable to change blkio parameters error: internal error: connection closed due to keepalive timeout ========================================================= The backtrace info for the crash in step 4 is as follow: ========================================================= Thread 2 "libvirtd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffedf06700 (LWP 9137)] 0x00007ffff7424caa in virCgroupBackendForController () from /lib64/libvirt.so.0 Missing separate debuginfos, use: yum debuginfo-install libiscsi-1.18.0-8.module+el8.1.0+3554+1a3a94a6.x86_64 libselinux-2.9-2.1.el8.x86_64 netcf-libs-0.2.8-11.module+el8.1.0+3554+1a3a94a6.x86_64 (gdb) bt #0 0x00007ffff7424caa in virCgroupBackendForController () from /lib64/libvirt.so.0 #1 0x00007ffff741f645 in virCgroupSetBlkioWeight () from /lib64/libvirt.so.0 #2 0x00007fffae59174b in qemuDomainSetBlkioParameters () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007ffff764a6e6 in virDomainSetBlkioParameters () from /lib64/libvirt.so.0 #4 0x000055555558ec03 in remoteDispatchDomainSetBlkioParameters (args=0x7fffe8002cc0, rerr=0x7fffedf05940, msg=0x55555583ff30, client=<optimized out>, server=0x5555558038a0) at remote/remote_daemon_dispatch_stubs.h:10295 #5 remoteDispatchDomainSetBlkioParametersHelper (server=0x5555558038a0, client=<optimized out>, msg=0x55555583ff30, rerr=0x7fffedf05940, args=0x7fffe8002cc0, ret=0x7fffe8002dd0) at remote/remote_daemon_dispatch_stubs.h:10262 #6 0x00007ffff758b256 in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #7 0x00007ffff758fee0 in virNetServerProcessMsg () from /lib64/libvirt.so.0 #8 0x00007ffff75901d4 in virNetServerHandleJob () from /lib64/libvirt.so.0 #9 0x00007ffff74a966d in virThreadPoolWorker () from /lib64/libvirt.so.0 #10 0x00007ffff74a89ae in virThreadHelper () from /lib64/libvirt.so.0 #11 0x00007ffff4d512de in start_thread (arg=<optimized out>) at pthread_create.c:486 #12 0x00007ffff4447463 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Hi Pavel, the crash backtrace info is in above comment And I prepare a new env to try cgroupv2 and cannot start vm due to related cgroup not created by systemd, as follow: kernel-4.18.0-107.el8.x86_64 systemd-239-15.el8.x86_64 libvirt-5.5.0-1.module+el8.1.0+3580+d7f6488d.x86_64 [root@dell-per730-61 grub2]# grub2-editenv - list | grep kernelopts kernelopts=root=/dev/mapper/rhel_dell--per730--61-root ro crashkernel=auto resume=/dev/mapper/rhel_dell--per730--61-swap rd.lvm.lv=rhel_dell-per730-61/root rd.lvm.lv=rhel_dell-per730-61/swap rhgb quiet systemd.unified_cgroup_hierarchy=1 [root@dell-per730-61 grub2]# mount | grep cgroup cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate) [root@dell-per730-61 grub2]# cat /sys/fs/cgroup/cgroup.controllers cpuset cpu io memory pids rdma [root@dell-per730-61 grub2]# ll /sys/fs/cgroup/| grep machine drwxr-xr-x. 2 root root 0 Jul 11 06:04 machine.slice [root@dell-per730-61 grub2]# virsh start avocado-vt-vm1 error: Failed to start domain avocado-vt-vm1 error: Unable to read from '/sys/fs/cgroup/machine.slice/machine-qemu\x2d2\x2davocado\x2dvt\x2dvm1.scope/emulator/cgroup.controllers': No such file or directory [root@dell-per730-61 grub2]# ll /sys/fs/cgroup/machine.slice/| grep qemu -i | wc -l 0 [root@dell-per730-61 grub2]# virsh dumpxml avocado-vt-vm1 <domain type='kvm'> <name>avocado-vt-vm1</name> <uuid>c4958ff2-6464-4276-a63b-2adb648de750</uuid> <metadata> <ns0:libosinfo xmlns:ns0="http://libosinfo.org/xmlns/libvirt/domain/1.0"> <ns0:os id="http://redhat.com/rhel/8.1"/> </ns0:libosinfo> </metadata> <memory unit='KiB'>1048576</memory> <currentMemory unit='KiB'>1048576</currentMemory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc-q35-rhel8.0.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model' check='partial'> <model fallback='allow'/> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/avocado/data/avocado-vt/images/jeos-27-x86_64.qcow2'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x12'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x13'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0x14'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0x15'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x16'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller> <interface type='network'> <mac address='52:54:00:89:5e:fe'/> <source network='default'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes'> <listen type='address'/> </graphics> <video> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video> <memballoon model='virtio'> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/urandom</backend> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </rng> </devices> </domain>
Thanks for the backtrace, I'll look into it. For the issue that you cannot start a VM, can you please provide libvirt debug logs? The emulator directory is created by libvirt and it worked in my testing scenarios.
Created attachment 1589745 [details] failed to start vm error log
(In reply to Pavel Hrdina from comment #24) > Thanks for the backtrace, I'll look into it. For the issue that you cannot > start a VM, can you please provide libvirt debug logs? The emulator > directory is created by libvirt and it worked in my testing scenarios. generate comment 25's log with following steps: [root@dell-per730-61 grub2]# service libvirtd restart Redirecting to /bin/systemctl restart libvirtd.service [root@dell-per730-61 grub2]# echo "" > /var/log/libvirtd-debug.log [root@dell-per730-61 grub2]# virsh create /tmp/avocado-vt-vm1.xml error: Failed to create domain from /tmp/avocado-vt-vm1.xml error: Unable to read from '/sys/fs/cgroup/machine.slice/machine-qemu\x2d1\x2davocado\x2dvt\x2dvm1.scope/emulator/cgroup.controllers': No such file or directory pls have a check, thx
So I managed to reproduce it. It happens only if there are no real-time tasks so the CPU controller can be enabled and in that case it fails. I'll have to fix it.
(In reply to Pavel Hrdina from comment #28) > So I managed to reproduce it. It happens only if there are no real-time > tasks so > the CPU controller can be enabled and in that case it fails. I'll have to > fix it. Hi Pavel, Do you plan to make that fix in current bug? I am not quite understanding about the "real-time tasks on host". I just tried to set a 'dd' process's real-time attribute but seems not working. In terminal 1: [root@dell-per730-61 ~]# sysctl -w kernel.sched_rt_runtime_us=-1 [root@dell-per730-61 ~]# dd if=/dev/zero of=/dev/null Open terminal 2: [root@dell-per730-61 ~]# ps -ef | grep dd root 3485 3464 99 04:40 pts/2 00:01:16 dd if=/dev/zero of=/dev/null [root@dell-per730-61 ~]# chrt -p -r 99 3485 [root@dell-per730-61 ~]# virsh start avocado-vt-vm1 error: Failed to start domain avocado-vt-vm1 error: Unable to read from '/sys/fs/cgroup/machine.slice/machine-qemu\x2d3\x2davocado\x2dvt\x2dvm1.scope/emulator/cgroup.controllers': No such file or directory [root@dell-per730-61 ~]# chrt -p -f 99 3485 [root@dell-per730-61 ~]# virsh start avocado-vt-vm1 error: Failed to start domain avocado-vt-vm1 error: Unable to read from '/sys/fs/cgroup/machine.slice/machine-qemu\x2d4\x2davocado\x2dvt\x2dvm1.scope/emulator/cgroup.controllers': No such file or directory
(In reply to yisun from comment #29) > (In reply to Pavel Hrdina from comment #28) > > So I managed to reproduce it. It happens only if there are no real-time > > tasks so > > the CPU controller can be enabled and in that case it fails. I'll have to > > fix it. > Hi Pavel, > Do you plan to make that fix in current bug? Yes, I'll fix it within this bug. > I am not quite understanding about the "real-time tasks on host". I just > tried to set a 'dd' process's real-time attribute but seems not working. If you have a server installation without GUI there will not be any real-time task by default, if you install GUI as well there will be pulseaudio process with real-time tasks. > In terminal 1: > [root@dell-per730-61 ~]# sysctl -w kernel.sched_rt_runtime_us=-1 > [root@dell-per730-61 ~]# dd if=/dev/zero of=/dev/null > Open terminal 2: > [root@dell-per730-61 ~]# ps -ef | grep dd > root 3485 3464 99 04:40 pts/2 00:01:16 dd if=/dev/zero of=/dev/null > [root@dell-per730-61 ~]# chrt -p -r 99 3485 > [root@dell-per730-61 ~]# virsh start avocado-vt-vm1 > error: Failed to start domain avocado-vt-vm1 > error: Unable to read from > '/sys/fs/cgroup/machine.slice/machine-qemu\x2d3\x2davocado\x2dvt\x2dvm1. > scope/emulator/cgroup.controllers': No such file or directory > [root@dell-per730-61 ~]# chrt -p -f 99 3485 > [root@dell-per730-61 ~]# virsh start avocado-vt-vm1 > error: Failed to start domain avocado-vt-vm1 > error: Unable to read from > '/sys/fs/cgroup/machine.slice/machine-qemu\x2d4\x2davocado\x2dvt\x2dvm1. > scope/emulator/cgroup.controllers': No such file or directory
After executing all existing test cases, the following potential issues found for now. bug 1~3 could block normal usage. So I'll set this RFE to FailedQA, please help to confirm if they are valid issues and should be fixed before 8.1 release. thx. 1. Bug 1734353 - [cgroup_v2] Error happened and xml not changed when use blkiotune to set blk cgroup values 2. Bug 1735740 - [cgroup_v2] Cannot get and set cpu cgroup params due to cpu.max error 3. Bug 1734737 - [cgroup_v2] qemu crashed when set memory.max to a too small value 4. Bug 1734625 - [cgroup_v2]When disable 'io' controller, the error message is code-level when try to set values to blk related params 5. Bug 1740049 - [cgroup_v2] Libvirt crashed when do blkiotune to running vm with cgroup2 manually mounted
Verfied on: libvirt-5.6.0-6.module+el8.1.0+4244+9aa4e6bb.x86_64 Auto scripts to test blkiotune/memtune/schedinfo: https://github.com/autotest/tp-libvirt/pull/2324 https://github.com/avocado-framework/avocado-vt/pull/2258 guest_resource_control (blkiotune/memtune/schedinfo) (cgroup_v1 enabled): PASSED https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-8.1 %20x86_64/job/libvirt-RHEL-8.1-runtest-x86_64-function-guest_resource_control/40/testReport/ guest_resource_control (blkiotune/memtune/schedinfo) (cgroup_v2 enabled): PASSED https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-8.1 %20x86_64/job/libvirt-RHEL-8.1-runtest-x86_64-function-guest_resource_control/41/testReport/ Since cpuset controller still having issue in systemd, the related function cannot be tested on cgroup2, just do a regression test on cgroup1 numatune(cgroup_v1 enabled): PASSED https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-8.1%20x86_64/job/libvirt-RHEL-8.1-runtest-x86_64-function-numa/26/testReport/rhel.virsh/numatune/ vcpu_affinity(cgroup_v1 enabled): PASSED https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-8.1%20x86_64/job/libvirt-RHEL-8.1-runtest-x86_64-function-cpu/31/testReport/rhel/vcpu_affinity/ One case failed due to script issue: rhel.vcpu_affinity.positive_test.cputune.offline_hostcpu ==> script issue, in cgroup v1 env, this function required the cpuset controller to be in v2 mode: "cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset,cpuset_v2_mode)" vcpupin(cgroup_v1 enabled): PASSED https://libvirt-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/libvirt/view/RHEL-8.1%20x86_64/job/libvirt-RHEL-8.1-runtest-x86_64-function-cpu/31/testReport/rhel.virsh/vcpupin/ device controller regression test on cgroupv1 (manually): RHEL7-18222 - [Device.list] Start VM with host serial port https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL7-18222 RHEL7-98668 - [Device.list] Start VM with backing file in host storage device https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL7-98668 RHEL7-98667 - [Device.list] Start VM with host storage device https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL7-98667 RHEL7-98984 - [Device.list] Start VM with sound device when setting different value to 'vnc_allow_host_audio' https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL7-98984 RHEL7-18232 - [Cgroup_device_acl] Default value of device.list of VM for default configuration of cgroup_device_acl in qemu.conf https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL7-18232 RHEL7-23878 - [Cgroup_controllers] Disable memory 'cgroup_controllers' in qemu.conf https://polarion.engineering.redhat.com/polarion/#/project/RedHatEnterpriseLinux7/workitem?id=RHEL7-23878 The test result is PASSED, and we'll continue tracking following issues by separated bzs cpuset controller not fully work due to systemd bz: Bug 1724617 - RFE: add support for cgroups v2 cpuset controller existing lower priority issues: Bug 1740049 - [cgroup_v2] Libvirt crashed when do blkiotune to running vm with cgroup2 manually mounted Bug 1734625 - [cgroup_v2]When disable 'io' controller, the error message is code-level when try to set values to blk related params Bug 1717394 - RFE: add cgroups v2 BPF devices support
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3723