Bug 1746517
Summary: | [RHEL 7.7] libvirtd: reset CPU affinity to all enabled CPUs, when runs in custom cpuset | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Valentina Krasnobaeva <valentina.krasnobaeva> | ||||
Component: | libvirt | Assignee: | Pavel Hrdina <phrdina> | ||||
Status: | CLOSED ERRATA | QA Contact: | jiyan <jiyan> | ||||
Severity: | high | Docs Contact: | |||||
Priority: | urgent | ||||||
Version: | 7.7 | CC: | bshephar, dyuan, dzheng, jdenemar, jean-mickael.guerin, jraju, jsuchane, lhuang, lmen, lmiksik, mkalinin, phrdina, sergey.kuya, xuzhang, yalzhang | ||||
Target Milestone: | rc | ||||||
Target Release: | --- | ||||||
Hardware: | x86_64 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | libvirt-4.5.0-29.el7 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | |||||||
: | 1781115 (view as bug list) | Environment: | |||||
Last Closed: | 2020-03-31 19:59:02 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 1781115 | ||||||
Attachments: |
|
Broken by upstream commit commit f136b83139c63f20de0df3285d9e82df2fb97bfc Author: Michal Privoznik <mprivozn> Date: Wed Jan 30 09:46:23 2019 +0100 qemu: Rework setting process affinity which was backported into RHEL-7 in order to fix different bug. It was meant to fix CPU affinity issue but it affected cgroups cpuset as well. Yes, I've also applied this commit to 4.5.0-10.el7_6.12 and it stops worked. I suppose, that issue may be in this part of the patch: } else { - VIR_DEBUG("Set CPU affinity with specified cpuset"); - if (vm->def->cpumask) { - cpumapToSet = vm->def->cpumask; - } else { - /* You may think this is redundant, but we can't assume libvirtd - * itself is running on all pCPUs, so we need to explicitly set - * the spawned QEMU instance to all pCPUs if no map is given in - * its config file */ - int hostcpus; - - if (virHostCPUHasBitmap()) { - hostcpumap = virHostCPUGetOnlineBitmap(); - cpumap = virProcessGetAffinity(vm->pid); - } - - if (hostcpumap && cpumap && virBitmapEqual(hostcpumap, cpumap)) { - /* we're using all available CPUs, no reason to set - * mask. If libvirtd is running without explicit - * affinity, we can use hotplugged CPUs for this VM */ - ret = 0; - goto cleanup; - } else { - /* setaffinity fails if you set bits for CPUs which - * aren't present, so we have to limit ourselves */ - if ((hostcpus = virHostCPUGetCount()) < 0) - goto cleanup; - - if (hostcpus > QEMUD_CPUMASK_LEN) - hostcpus = QEMUD_CPUMASK_LEN; - - virBitmapFree(cpumap); - if (!(cpumap = virBitmapNew(hostcpus))) - goto cleanup; - - virBitmapSetAll(cpumap); - - cpumapToSet = cpumap; - } - } + if (qemuProcessGetAllCpuAffinity(&hostcpumap) < 0) + goto cleanup; + cpumapToSet = hostcpumap; } cpumapToSet = hostcpumap ---> I suppose makes libvirtd rewrite 0-39 to machine.scope/cpuset.cpus instead of available cpus list No, that's not the broken part, it's in function qemuProcessSetupPid() where variable 'use_cpumask' is set to 0-39 and later passed to qemuSetupCgroupCpusetCpus() function. I'll fix it upstream and backport the patch into RHEL to fix this BZ. Upstream patch posted: https://www.redhat.com/archives/libvir-list/2019-November/msg00210.html Upstream commit: commit 4c0398b5284d14c55eca51095673b6fadbbd85fb Author: Pavel Hrdina <phrdina> Date: Thu Nov 7 22:23:04 2019 +0100 qemu_process: fix starting VMs if machine group has limited cpuset.cpus Reproduce this bug on libvirt-4.5.0-23.el7.x86_64 and verify this bug on libvirt-4.5.0-29.el7.x86_64. Version: libvirt-4.5.0-23.el7.x86_64 qemu-kvm-rhev-2.12.0-38.el7.x86_64 kernel-3.10.0-1115.el7.x86_64 Steps: 1. Check pid of libvirtd # ps -ef |grep "/usr/sbin/libvirtd" root 18905 1 0 01:22 ? 00:00:00 /usr/sbin/libvirtd # grep Cpus_allowed_list /proc/18905/task/*/status /proc/18905/task/18905/status:Cpus_allowed_list: 0-31 2. Modify Cpus_allowed_list for libvirtd # taskset -pc 2,4,6,8,10 18905 pid 18905's current affinity list: 0-31 pid 18905's new affinity list: 2,4,6,8,10 # grep Cpus_allowed_list /proc/18905/task/*/status /proc/18905/task/18905/status:Cpus_allowed_list: 2,4,6,8,10 3. Create domain from XML and check log # cat repro.xml ... <vcpu placement="static">4</vcpu> <cputune> <vcpupin vcpu="0" cpuset="2"/> <vcpupin vcpu="1" cpuset="4"/> <vcpupin vcpu="2" cpuset="6"/> <vcpupin vcpu="3" cpuset="8"/> </cputune> <cpu> <numa> <cell id="0" cpus="1" memory="1048576" memAccess="shared"/> </numa> </cpu> <numatune> <memory mode="strict" nodeset="0"/> </numatune> ... # virsh create repro.xml error: Failed to create domain from repro.xml error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus': Permission denied # tail -f /var/log/libvirt/libvirtd.log ... 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator for pid -1 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator for pid -1 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator for pid -1 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/ 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/ 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/ 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d2\x2drepro.scope -> /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/cpuset.cpus 2019-12-03 06:37:25.531+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0-2,4-31 2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus' to '0-2,4-31' 2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/cpuset.mems 2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.mems' to '0,4' 2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/cpuset.memory_migrate 2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.memory_migrate' to '1' 2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1126 : Done making controllers for group 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus' to '0-31' 2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26 2019-12-03 06:37:25.532+0000: 18910: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus': Permission denied ... 4. Update libvirt and repeat step-3 # yum update libvirt* -y # rpm -qa libvirt libvirt-4.5.0-29.el7.x86_64 # virsh create repro.xml Domain repro created from repro.xml # tail -f /var/log/libvirt/libvirtd.log ... 2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 for pid -1 2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 for pid -1 2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 for pid -1 2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/ 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/ 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/ 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2drepro.scope -> /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.cpus 2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0-2,4-31 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/cpuset.cpus' to '0-2,4-31' 2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.mems 2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/cpuset.mems' to '0,4' 2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.memory_migrate 2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/cpuset.memory_migrate' to '1' 2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29 ... # virsh vcpupin repro VCPU: CPU Affinity ---------------------------------- 0: 2 1: 4 2: 6 3: 8 Hi Pavel I am not sure whether "taskset" in my verifying steps (comment 14) could equals the operation of "Libvirt runs in a custom cpuset 'libvirt'" in bug description. So when I was verifying this bug, the log shows the following info: 2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0-2,4-31 (**I made CPU3 offline**) 2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus' to '0-31' Even though the taskset already takes effect: # taskset -pc 2,4,6,8,10 18905 pid 18905's current affinity list: 0-31 pid 18905's new affinity list: 2,4,6,8,10 # grep Cpus_allowed_list /proc/18905/task/*/status /proc/18905/task/18905/status:Cpus_allowed_list: 2,4,6,8,10 However, the bug description shows: 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0,2,4,6,8' So the questions are: 1> Will it okay to verify this bug with "taskset" cmds? 2> If not 1>; how could I set conf just like the bug description? Thank you. Hi Jing, taskset is not the correct tool, that one changes process affinity which is unrelated to cgroups. The reproducer is hidden in the first comment: On a host with more then 2 CPUs you can do this, the important part is to limit the whole machine.slice group to less CPUs then the host actually has: echo "0,1" > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus Now you can start a VM, with libvirt-4.5.0-23.el7 it should fail, with libvirt-4.5.0-29.el7 it should work. Pavel Hi Pavel Thank you very much for you timely reply. Reproduce this bug on libvirt-4.5.0-23.el7.x86_64 based on the info in comment 16. Verify this bug on libvirt-4.5.0-29.el7.x86_64. Version: libvirt-4.5.0-23.el7.x86_64 qemu-kvm-rhev-2.12.0-38.el7.x86_64 kernel-3.10.0-1115.el7.x86_64 Steps: 1. Modify cpuset.cpus # lscpu | grep On On-line CPU(s) list: 0-31 # echo "0,2,4,6,8" > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus # cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 0,2,4,6,8 2. Create domain with the following info # cat repro.xml ... <vcpu placement="static">2</vcpu> <cputune> <vcpupin vcpu="0" cpuset="2"/> <vcpupin vcpu="1" cpuset="4"/> </cputune> <cpu> <numa> <cell id="0" cpus="1" memory="1048576" memAccess="shared"/> </numa> </cpu> <numatune> <memory mode="strict" nodeset="0"/> </numatune> ... # virsh create repro.xml error: Failed to create domain from repro.xml error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus': Permission denied 3. Check log # tail -f /var/log/libvirt/libvirtd.log ... 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator for pid -1 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator for pid -1 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator for pid -1 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/ 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/ 2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/ 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d8\x2drepro.scope -> /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/cpuset.cpus 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus' to '0,2,4,6,8' 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/cpuset.mems 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.mems' to '0,4' 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/cpuset.memory_migrate 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.memory_migrate' to '1' 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1126 : Done making controllers for group 2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus' to '0-31' 2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27 2019-12-03 08:38:00.755+0000: 3494: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus': Permission denied ... 4. Update libvirt and restart libvirtd # yum update libvirtd # systemctl restart libvirtd # rpm -qa libvirt libvirt-4.5.0-29.el7.x86_64 5. Repeat step-3 and step-4 # virsh create repro.xml Domain repro created from repro.xml # cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 0,2,4,6,8 # virsh vcpupin repro VCPU: CPU Affinity ---------------------------------- 0: 2 1: 4 # tail -f /var/log/libvirt/libvirtd.log 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 for pid -1 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 for pid -1 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 for pid -1 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/ 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/ 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/ 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2drepro.scope -> /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.cpus 2019-12-03 08:41:25.266+0000: 3818: debug : virFileClose:111 : Closed fd 29 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.cpus' to '0,2,4,6,8' 2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.mems 2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.mems' to '0,4' 2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.memory_migrate 2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.memory_migrate' to '1' 2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1126 : Done making controllers for group ... 2019-12-03 08:41:25.265+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu0/cpuset.cpus' to '2' 2019-12-03 08:41:25.265+0000: 3818: debug : virFileClose:111 : Closed fd 29 ... 2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.cpus' to '4' 2019-12-03 08:41:25.268+0000: 3818: debug : virFileClose:111 : Closed fd 29 ... The result is as expected, move this bug to be verified. # rpm -qa libvirt libvirt-4.5.0-29.el7.x86_64 # virsh dumpxml test --inactive |grep "<vcpu" -A3 <vcpu placement='auto' current='3'>8</vcpu> <numatune> <memory mode='strict' placement='auto'/> </numatune> # virsh start test error: Failed to start domain test error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dtest.scope/emulator/cpuset.cpus': Permission denied # virsh edit test Domain test XML configuration edited. # virsh dumpxml test --inactive |grep "<vcpu" -A3 <vcpu placement='static' current='3'>8</vcpu> <numatune> <memory mode='strict' nodeset='0'/> </numatune> # virsh start test Domain test started Hi Pavel I found another scenario, when the placement=auto, the err raised again in the current fixing version. Please help check this issue, thank you. :) Hi Jing, I'm not sure about the usecase with placement="auto". From libvirt documentation: "Using "auto" indicates the domain process will be pinned to the advisory nodeset from querying numad and the value of attribute cpuset will be ignored if it's specified." Which means that libvirt is getting the value from numad. I would say this is not worth fixing. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1094 |
Created attachment 1609069 [details] VM XML config for virsh Description of problem: The problem appears since libvirt 4.5.0-23.el7 version. I can't reproduce this behavior with 4.5.0-10.el7_6.12. Libvirt runs in a custom cpuset 'libvirt', where the number of available cpus is restricted to 0,2,4,6,8. And this 'libvirt' cpuset is created in a system with total cpus number: 40 (all cpus are enabled in BIOS) and we have '0-39' range in /sys/fs/cgroup/cpuset/cpuset.cpus. When a VM with pinned vcpus is launched: <vcpu placement='static'>2</vcpu> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='6'/> I have the following error: # virsh create /tmp/vm1_2vms-2cores_host1.xml error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied And in the debug log I can see, that /sys/fs/cgroup/cpuset/machine.slice was created with a proper cpuset list: 0,2,4,6,8. This list firstly was successfully inherited and set in /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus, but finally libvirtd tries to reset inherited cpus list value and set there: "0-39" 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/ 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/ 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/ 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus 2019-08-28 15:11:03.357+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0,2,4,6,8' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.mems' to '0-1' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.memory_migrate' to '1' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1126 : Done making controllers for group 2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0-39' 2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27 2019-08-28 15:11:03.358+0000: 25536: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied The same log snippet with 4.5.0-10.el7_6.12 version, everything works well: 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/ 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/ 2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/ 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.cpus' to '0,2,4,6,8' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0-1' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.memory_migrate' to '1' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1126 : Done making controllers for group 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0' 2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29 2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/tasks' to '27045' How reproducible: Always Steps to Reproduce: 1. create a cpuset with reduced cpus list and launch libvirtd in it 2. prepare VM XML with pinned vcpus and <numatune> setting 3. virsh create vm.xml Actual results: error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied Expected results: should work as with a previous version 4.5.0-10.el7_6.12. Additional info: # ps auxf | grep libvirt root 24571 2.3 0.0 1895564 28052 ? Ssl 16:52 0:00 /usr/sbin/libvirtd # grep Cpus_allowed /proc/24571/task/*/status /proc/24571/task/24571/status:Cpus_allowed: 00,00000554 /proc/24571/task/24571/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24572/status:Cpus_allowed: 00,00000554 /proc/24571/task/24572/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24573/status:Cpus_allowed: 00,00000554 /proc/24571/task/24573/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24574/status:Cpus_allowed: 00,00000554 /proc/24571/task/24574/status:Cpus_allowed_list: 2,4,6,8,10 /proc/24571/task/24575/status:Cpus_allowed: 00,00000554 .. for all threads Processor: "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz" Sockets: 2, Cores: 20, HyperThreads: 40 socket 0 socket 1 +---------------------+ +---------------------+ | c0 c1 | | c0 c1 | | +-------+ +-------+ | | +-------+ +-------+ | | | 0| 20| | 2| 22| | | | 1| 21| | 3| 23| | | +-------+ +-------+ | | +-------+ +-------+ | | c2 c3 | | c2 c3 | | +-------+ +-------+ | | +-------+ +-------+ | | | 4| 24| | 6| 26| | | | 5| 25| | 7| 27| | | +-------+ +-------+ | | +-------+ +-------+ | | c4 c8 | | c4 c8 | | +-------+ +-------+ | | +-------+ +-------+ | | | 8| 28| | 10| 30| | | | 9| 29| | 11| 31| | | +-------+ +-------+ | | +-------+ +-------+ | | c9 c10 | | c9 c10 | | +-------+ +-------+ | | +-------+ +-------+ | | | 12| 32| | 14| 34| | | | 13| 33| | 15| 35| | | +-------+ +-------+ | | +-------+ +-------+ | | c11 c12 | | c11 c12 | | +-------+ +-------+ | | +-------+ +-------+ | | | 16| 36| | 18| 38| | | | 17| 37| | 19| 39| | | +-------+ +-------+ | | +-------+ +-------+ | +---------------------+ +---------------------+ # cat /sys/fs/cgroup/cpuset/cpuset.cpus 0-39 # cat /sys/fs/cgroup/cpuset/libvirt/cpuset.cpus 2,4,6,8,10 # cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 0,2,4,6,8