Bug 1746517

Summary: [RHEL 7.7] libvirtd: reset CPU affinity to all enabled CPUs, when runs in custom cpuset
Product: Red Hat Enterprise Linux 7 Reporter: Valentina Krasnobaeva <valentina.krasnobaeva>
Component: libvirtAssignee: Pavel Hrdina <phrdina>
Status: CLOSED ERRATA QA Contact: jiyan <jiyan>
Severity: high Docs Contact:
Priority: urgent    
Version: 7.7CC: bshephar, dyuan, dzheng, jdenemar, jean-mickael.guerin, jraju, jsuchane, lhuang, lmen, lmiksik, mkalinin, phrdina, sergey.kuya, xuzhang, yalzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: libvirt-4.5.0-29.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1781115 (view as bug list) Environment:
Last Closed: 2020-03-31 19:59:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1781115    
Attachments:
Description Flags
VM XML config for virsh none

Description Valentina Krasnobaeva 2019-08-28 16:23:07 UTC
Created attachment 1609069 [details]
VM XML config for virsh

Description of problem:

The problem appears since libvirt 4.5.0-23.el7 version. I can't reproduce this behavior with 4.5.0-10.el7_6.12.

Libvirt runs in a custom cpuset 'libvirt', where the number of available cpus is restricted to 0,2,4,6,8. And this 'libvirt' cpuset is created in a system with total cpus number: 40 (all cpus are enabled in BIOS) and we have '0-39' range in /sys/fs/cgroup/cpuset/cpuset.cpus. 

When a VM with pinned vcpus is launched:

  <vcpu placement='static'>2</vcpu>
  <vcpupin vcpu='0' cpuset='4'/>
  <vcpupin vcpu='1' cpuset='6'/>

I have the following error:

# virsh create /tmp/vm1_2vms-2cores_host1.xml 
error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied

And in the debug log I can see, that /sys/fs/cgroup/cpuset/machine.slice was created with a proper cpuset list: 0,2,4,6,8. This list firstly was successfully inherited and set in /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus, but finally libvirtd tries to reset inherited cpus list value and set there: "0-39"

2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator for pid -1
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 15:11:03.357+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0,2,4,6,8'


2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.mems' to '0-1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.memory_migrate' to '1'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupMakeGroup:1126 : Done making controllers for group


2019-08-28 15:11:03.358+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0-39'
2019-08-28 15:11:03.358+0000: 25536: debug : virFileClose:111 : Closed fd 27
2019-08-28 15:11:03.358+0000: 25536: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied


The same log snippet with 4.5.0-10.el7_6.12 version, everything works well:

2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1 for pid -1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.837+0000: 26937: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope -> /machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.cpus
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.cpus' to '0,2,4,6,8'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.mems
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0-1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0-1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/cpuset.memory_migrate
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.memory_migrate' to '1'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupMakeGroup:1126 : Done making controllers for group
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/cpuset.mems' to '0'
2019-08-28 16:13:22.838+0000: 26937: debug : virFileClose:111 : Closed fd 29
2019-08-28 16:13:22.838+0000: 26937: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/vcpu1/tasks' to '27045'

How reproducible:

Always

Steps to Reproduce:
1. create a cpuset with reduced cpus list and launch libvirtd in it
2. prepare VM XML with pinned vcpus and <numatune> setting
3. virsh create vm.xml

Actual results:

error: Failed to create domain from /tmp/vm1_2vms-2cores_host1.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus': Permission denied


Expected results:

should work as with a previous version 4.5.0-10.el7_6.12.


Additional info:

# ps auxf | grep libvirt
root     24571  2.3  0.0 1895564 28052 ?       Ssl  16:52   0:00 /usr/sbin/libvirtd

# grep Cpus_allowed /proc/24571/task/*/status
/proc/24571/task/24571/status:Cpus_allowed:     00,00000554
/proc/24571/task/24571/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24572/status:Cpus_allowed:     00,00000554
/proc/24571/task/24572/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24573/status:Cpus_allowed:     00,00000554
/proc/24571/task/24573/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24574/status:Cpus_allowed:     00,00000554
/proc/24571/task/24574/status:Cpus_allowed_list:        2,4,6,8,10
/proc/24571/task/24575/status:Cpus_allowed:     00,00000554
..
for all threads

Processor: "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz"

        Sockets: 2, Cores: 20, HyperThreads: 40

         socket 0                 socket 1        
  +---------------------+  +---------------------+
  |        c0        c1 |  |        c0        c1 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  0| 20| |  2| 22| |  | |  1| 21| |  3| 23| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c2        c3 |  |        c2        c3 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  4| 24| |  6| 26| |  | |  5| 25| |  7| 27| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c4        c8 |  |        c4        c8 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | |  8| 28| | 10| 30| |  | |  9| 29| | 11| 31| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |        c9       c10 |  |        c9       c10 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | | 12| 32| | 14| 34| |  | | 13| 33| | 15| 35| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  |       c11       c12 |  |       c11       c12 |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  | | 16| 36| | 18| 38| |  | | 17| 37| | 19| 39| |
  | +-------+ +-------+ |  | +-------+ +-------+ |
  +---------------------+  +---------------------+

# cat  /sys/fs/cgroup/cpuset/cpuset.cpus 
0-39

# cat  /sys/fs/cgroup/cpuset/libvirt/cpuset.cpus 
2,4,6,8,10

# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 
0,2,4,6,8

Comment 2 Pavel Hrdina 2019-08-29 14:44:34 UTC
Broken by upstream commit

commit f136b83139c63f20de0df3285d9e82df2fb97bfc
Author: Michal Privoznik <mprivozn>
Date:   Wed Jan 30 09:46:23 2019 +0100

    qemu: Rework setting process affinity

which was backported into RHEL-7 in order to fix different bug.

It was meant to fix CPU affinity issue but it affected cgroups cpuset as well.

Comment 3 Valentina Krasnobaeva 2019-08-29 14:49:48 UTC
Yes, I've also applied this commit to 4.5.0-10.el7_6.12 and it stops worked.

I suppose, that issue may be in this part of the patch:

     } else {
-        VIR_DEBUG("Set CPU affinity with specified cpuset");
-        if (vm->def->cpumask) {
-            cpumapToSet = vm->def->cpumask;
-        } else {
-            /* You may think this is redundant, but we can't assume libvirtd
-             * itself is running on all pCPUs, so we need to explicitly set
-             * the spawned QEMU instance to all pCPUs if no map is given in
-             * its config file */
-            int hostcpus;
-
-            if (virHostCPUHasBitmap()) {
-                hostcpumap = virHostCPUGetOnlineBitmap();
-                cpumap = virProcessGetAffinity(vm->pid);
-            }
-
-            if (hostcpumap && cpumap && virBitmapEqual(hostcpumap, cpumap)) {
-                /* we're using all available CPUs, no reason to set
-                 * mask. If libvirtd is running without explicit
-                 * affinity, we can use hotplugged CPUs for this VM */
-                ret = 0;
-                goto cleanup;
-            } else {
-                /* setaffinity fails if you set bits for CPUs which
-                 * aren't present, so we have to limit ourselves */
-                if ((hostcpus = virHostCPUGetCount()) < 0)
-                    goto cleanup;
-
-                if (hostcpus > QEMUD_CPUMASK_LEN)
-                    hostcpus = QEMUD_CPUMASK_LEN;
-
-                virBitmapFree(cpumap);
-                if (!(cpumap = virBitmapNew(hostcpus)))
-                    goto cleanup;
-
-                virBitmapSetAll(cpumap);
-
-                cpumapToSet = cpumap;
-            }
-        }
+        if (qemuProcessGetAllCpuAffinity(&hostcpumap) < 0)
+            goto cleanup;
+        cpumapToSet = hostcpumap;
     }

 cpumapToSet = hostcpumap ---> I suppose makes libvirtd rewrite 0-39 to machine.scope/cpuset.cpus instead of available cpus list

Comment 4 Pavel Hrdina 2019-08-29 15:04:30 UTC
No, that's not the broken part, it's in function qemuProcessSetupPid()
where variable 'use_cpumask' is set to 0-39 and later passed to
qemuSetupCgroupCpusetCpus() function.

I'll fix it upstream and backport the patch into RHEL to fix this BZ.

Comment 5 Pavel Hrdina 2019-11-12 11:13:24 UTC
Upstream patch posted:

https://www.redhat.com/archives/libvir-list/2019-November/msg00210.html

Comment 6 Pavel Hrdina 2019-11-18 09:43:39 UTC
Upstream commit:

commit 4c0398b5284d14c55eca51095673b6fadbbd85fb
Author: Pavel Hrdina <phrdina>
Date:   Thu Nov 7 22:23:04 2019 +0100

    qemu_process: fix starting VMs if machine group has limited cpuset.cpus

Comment 14 jiyan 2019-12-03 06:48:08 UTC
Reproduce this bug on libvirt-4.5.0-23.el7.x86_64 and verify this bug on libvirt-4.5.0-29.el7.x86_64.

Version:
libvirt-4.5.0-23.el7.x86_64
qemu-kvm-rhev-2.12.0-38.el7.x86_64
kernel-3.10.0-1115.el7.x86_64

Steps:
1. Check pid of libvirtd
# ps -ef |grep "/usr/sbin/libvirtd"
root     18905     1  0 01:22 ?        00:00:00 /usr/sbin/libvirtd

# grep Cpus_allowed_list /proc/18905/task/*/status
/proc/18905/task/18905/status:Cpus_allowed_list:	0-31

2. Modify Cpus_allowed_list for libvirtd
# taskset -pc 2,4,6,8,10 18905 
pid 18905's current affinity list: 0-31
pid 18905's new affinity list: 2,4,6,8,10

# grep Cpus_allowed_list /proc/18905/task/*/status
/proc/18905/task/18905/status:Cpus_allowed_list:	2,4,6,8,10

3. Create domain from XML and check log
# cat repro.xml 
...
	<vcpu placement="static">4</vcpu>
	<cputune>
	    <vcpupin vcpu="0" cpuset="2"/>
	    <vcpupin vcpu="1" cpuset="4"/>
	    <vcpupin vcpu="2" cpuset="6"/>
	    <vcpupin vcpu="3" cpuset="8"/>
	</cputune>
	<cpu>
		<numa>
			<cell id="0" cpus="1" memory="1048576" memAccess="shared"/>
		</numa>
	</cpu>
	<numatune>
		<memory mode="strict" nodeset="0"/>
	</numatune>
...

# virsh create repro.xml 
error: Failed to create domain from repro.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus': Permission denied

# tail -f /var/log/libvirt/libvirtd.log
...
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator for pid -1
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator for pid -1
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator for pid -1
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d2\x2drepro.scope -> /machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/cpuset.cpus
2019-12-03 06:37:25.531+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0-2,4-31
2019-12-03 06:37:25.531+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus' to '0-2,4-31'
2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/cpuset.mems
2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.mems' to '0,4'
2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/cpuset.memory_migrate
2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.memory_migrate' to '1'
2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupMakeGroup:1126 : Done making controllers for group
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus' to '0-31'
2019-12-03 06:37:25.532+0000: 18910: debug : virFileClose:111 : Closed fd 26
2019-12-03 06:37:25.532+0000: 18910: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus': Permission denied

...

4. Update libvirt and repeat step-3
# yum update libvirt* -y

# rpm -qa libvirt
libvirt-4.5.0-29.el7.x86_64

# virsh create repro.xml
Domain repro created from repro.xml

# tail -f /var/log/libvirt/libvirtd.log
...
2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 for pid -1
2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 for pid -1
2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2 for pid -1
2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2
2019-12-03 06:43:29.105+0000: 20194: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2drepro.scope -> /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.cpus
2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0-2,4-31
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/cpuset.cpus' to '0-2,4-31'
2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.mems
2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/cpuset.mems' to '0,4'
2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.memory_migrate
2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu2/cpuset.memory_migrate' to '1'
2019-12-03 06:43:29.106+0000: 20194: debug : virFileClose:111 : Closed fd 29
...

# virsh vcpupin repro
VCPU: CPU Affinity
----------------------------------
   0: 2
   1: 4
   2: 6
   3: 8

Comment 15 jiyan 2019-12-03 06:53:55 UTC
Hi Pavel
I am not sure whether "taskset" in my verifying steps (comment 14) could equals the operation of "Libvirt runs in a custom cpuset 'libvirt'" in bug description.

So when I was verifying this bug, the log shows the following info:
2019-12-03 06:43:29.106+0000: 20194: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0-2,4-31 (**I made CPU3 offline**)
2019-12-03 06:37:25.532+0000: 18910: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d2\x2drepro.scope/emulator/cpuset.cpus' to '0-31'

Even though the taskset already takes effect:
# taskset -pc 2,4,6,8,10 18905 
pid 18905's current affinity list: 0-31
pid 18905's new affinity list: 2,4,6,8,10

# grep Cpus_allowed_list /proc/18905/task/*/status
/proc/18905/task/18905/status:Cpus_allowed_list:	2,4,6,8,10

However, the bug description shows:
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-08-28 15:11:03.357+0000: 25536: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dvm12vms\x2d2coreshost1.scope/emulator/cpuset.cpus' to '0,2,4,6,8'

So the questions are:
1> Will it okay to verify this bug with "taskset" cmds?
2> If not 1>; how could I set conf just like the bug description?

Thank you.

Comment 16 Pavel Hrdina 2019-12-03 08:13:08 UTC
Hi Jing,

taskset is not the correct tool, that one changes process affinity which is unrelated to cgroups.

The reproducer is hidden in the first comment:

On a host with more then 2 CPUs you can do this, the important part is to limit the whole machine.slice group to less CPUs then the host actually has:

    echo "0,1" > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus

Now you can start a VM, with libvirt-4.5.0-23.el7 it should fail, with libvirt-4.5.0-29.el7 it should work.

Pavel

Comment 17 jiyan 2019-12-03 08:48:28 UTC
Hi Pavel
Thank you very much for you timely reply.

Reproduce this bug on libvirt-4.5.0-23.el7.x86_64 based on the info in comment 16. 
Verify this bug on libvirt-4.5.0-29.el7.x86_64.

Version:
libvirt-4.5.0-23.el7.x86_64
qemu-kvm-rhev-2.12.0-38.el7.x86_64
kernel-3.10.0-1115.el7.x86_64

Steps:
1. Modify cpuset.cpus
# lscpu | grep On
On-line CPU(s) list:   0-31

# echo "0,2,4,6,8" > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus

# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 
0,2,4,6,8

2. Create domain with the following info
# cat repro.xml 
...
	<vcpu placement="static">2</vcpu>
	<cputune>
	    <vcpupin vcpu="0" cpuset="2"/>
	    <vcpupin vcpu="1" cpuset="4"/>
	</cputune>
	<cpu>
		<numa>
			<cell id="0" cpus="1" memory="1048576" memAccess="shared"/>
		</numa>
	</cpu>
	<numatune>
		<memory mode="strict" nodeset="0"/>
	</numatune>
...

# virsh create repro.xml 
error: Failed to create domain from repro.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus': Permission denied

3. Check log 
# tail -f /var/log/libvirt/libvirtd.log
...
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator for pid -1
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator for pid -1
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator for pid -1
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/
2019-12-03 08:38:00.754+0000: 3494: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d8\x2drepro.scope -> /machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/cpuset.cpus
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus' to '0,2,4,6,8'
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/cpuset.mems
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.mems' to '0,4'
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/cpuset.memory_migrate
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.memory_migrate' to '1'
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupMakeGroup:1126 : Done making controllers for group
2019-12-03 08:38:00.755+0000: 3494: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus' to '0-31'
2019-12-03 08:38:00.755+0000: 3494: debug : virFileClose:111 : Closed fd 27
2019-12-03 08:38:00.755+0000: 3494: error : virCgroupSetValueStr:806 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d8\x2drepro.scope/emulator/cpuset.cpus': Permission denied
...

4. Update libvirt and restart libvirtd
# yum update libvirtd

# systemctl restart libvirtd

# rpm -qa libvirt
libvirt-4.5.0-29.el7.x86_64

5. Repeat step-3 and step-4
# virsh create repro.xml 
Domain repro created from repro.xml

# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 
0,2,4,6,8

# virsh vcpupin repro
VCPU: CPU Affinity
----------------------------------
   0: 2
   1: 4

# tail -f /var/log/libvirt/libvirtd.log
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupDetect:747 : Detected mount/mapping 0:cpu at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 for pid -1
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupDetect:747 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpu,cpuacct in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 for pid -1
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupDetect:747 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1 for pid -1
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1049 : Make group /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupMakeGroup:1073 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupCpuSetInherit:989 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2drepro.scope -> /machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.cpus
2019-12-03 08:41:25.266+0000: 3818: debug : virFileClose:111 : Closed fd 29
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.cpus = 0,2,4,6,8
2019-12-03 08:41:25.266+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.cpus' to '0,2,4,6,8'
2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.mems
2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.mems = 0,4
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.mems' to '0,4'
2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupGetValueStr:832 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/cpuset.memory_migrate
2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupCpuSetInherit:999 : Inherit cpuset.memory_migrate = 1
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.memory_migrate' to '1'
2019-12-03 08:41:25.267+0000: 3818: debug : virFileClose:111 : Closed fd 29
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller memory
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller devices
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller freezer
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller blkio
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller net_cls
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1062 : Skipping unmounted controller perf_event
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1055 : Not creating systemd controller group
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupMakeGroup:1126 : Done making controllers for group
...
2019-12-03 08:41:25.265+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu0/cpuset.cpus' to '2'
2019-12-03 08:41:25.265+0000: 3818: debug : virFileClose:111 : Closed fd 29
...
2019-12-03 08:41:25.267+0000: 3818: debug : virCgroupSetValueStr:796 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2drepro.scope/vcpu1/cpuset.cpus' to '4'
2019-12-03 08:41:25.268+0000: 3818: debug : virFileClose:111 : Closed fd 29
...

The result is as expected, move this bug to be verified.

Comment 18 jiyan 2019-12-09 09:16:55 UTC
# rpm -qa libvirt
libvirt-4.5.0-29.el7.x86_64

# virsh dumpxml test --inactive |grep "<vcpu" -A3
  <vcpu placement='auto' current='3'>8</vcpu>
  <numatune>
    <memory mode='strict' placement='auto'/>
  </numatune>

# virsh start test
error: Failed to start domain test
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d4\x2dtest.scope/emulator/cpuset.cpus': Permission denied

# virsh edit test
Domain test XML configuration edited.

# virsh dumpxml test --inactive |grep "<vcpu" -A3
  <vcpu placement='static' current='3'>8</vcpu>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>

# virsh start test
Domain test started

Hi Pavel
I found another scenario, when the placement=auto, the err raised again in the current fixing version.
Please help check this issue, thank you. :)

Comment 19 Pavel Hrdina 2019-12-09 11:00:01 UTC
Hi Jing,

I'm not sure about the usecase with placement="auto". From libvirt documentation:

"Using "auto" indicates the domain process will be pinned to the advisory nodeset from querying numad and the value of attribute cpuset will be ignored if it's specified."

Which means that libvirt is getting the value from numad. I would say this is not worth fixing.

Comment 34 errata-xmlrpc 2020-03-31 19:59:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1094