Bug 1781115

Summary:	libvirtd: reset CPU affinity to all enabled CPUs, when runs in custom cpuset
Product:	Red Hat Enterprise Linux Advanced Virtualization	Reporter:	jiyan <jiyan>
Component:	libvirt	Assignee:	Pavel Hrdina <phrdina>
Status:	CLOSED ERRATA	QA Contact:	jiyan <jiyan>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.2	CC:	dyuan, dzheng, jdenemar, jean-mickael.guerin, lhuang, lmen, lmiksik, phrdina, sergey.kuya, valentina.krasnobaeva, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	8.0
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-5.10.0-1.el8	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	1746517	Environment:
Last Closed:	2020-05-05 09:52:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1746517
Bug Blocks:

Description jiyan 2019-12-09 10:58:17 UTC

+++ This bug was initially created as a clone of Bug #1746517 +++

Description:
libvirtd: reset CPU affinity to all enabled CPUs, when runs in custom cpuset

How Reproducible:
100%

Version:
libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7.x86_64
qemu-kvm-4.2.0-1.module+el8.2.0+4793+b09dd2fb.x86_64
kernel-4.18.0-160.el8.x86_64

Steps:
1. Configure host cgroup cpus
# lscpu |grep On
On-line CPU(s) list: 0-31

# echo "0-3" > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 

# cat /sys/fs/cgroup/cpus
0-3

2. Prepare a shutdown VM with the following conf, and start VM
# virsh domstate test820 
shut off

# virsh dumpxml test820 --inactive |grep "<vcpu" -A3
  <vcpu placement='auto' current='3'>8</vcpu>
  <numatune>
    <memory mode='strict' placement='auto'/>
  </numatune>

# virsh start test820 
error: Failed to start domain test820
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/cpuset.cpus': Permission denied

# tail -f /var/log/libvirt/libvirtd.log
2019-12-09 10:51:30.080+0000: 4266: debug : virCgroupV1MakeGroup:600 : Make group /machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator
2019-12-09 10:51:30.080+0000: 4266: debug : virCgroupV1MakeGroup:620 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:620 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:620 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1CpuSetInherit:544 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2dtest820.scope -> /machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupGetValueRaw:488 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/cpuset.cpus
2019-12-09 10:51:30.081+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1CpuSetInherit:554 : Inherit cpuset.cpus = 0-3
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupSetValueRaw:462 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/cpuset.cpus' to '0-3'
2019-12-09 10:51:30.081+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupGetValueRaw:488 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/cpuset.mems
2019-12-09 10:51:30.081+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1CpuSetInherit:554 : Inherit cpuset.mems = 0,4
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupSetValueRaw:462 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/cpuset.mems' to '0,4'
2019-12-09 10:51:30.081+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupGetValueRaw:488 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/cpuset.memory_migrate
2019-12-09 10:51:30.081+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1CpuSetInherit:554 : Inherit cpuset.memory_migrate = 1
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupSetValueRaw:462 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/cpuset.memory_migrate' to '1'
2019-12-09 10:51:30.081+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:613 : Skipping unmounted controller memory
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:613 : Skipping unmounted controller devices
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:613 : Skipping unmounted controller freezer
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:613 : Skipping unmounted controller blkio
2019-12-09 10:51:30.081+0000: 4266: debug : virCgroupV1MakeGroup:613 : Skipping unmounted controller net_cls
2019-12-09 10:51:30.082+0000: 4266: debug : virCgroupV1MakeGroup:613 : Skipping unmounted controller perf_event
2019-12-09 10:51:30.082+0000: 4266: debug : virCgroupV1MakeGroup:606 : Not creating systemd controller group
2019-12-09 10:51:30.082+0000: 4266: debug : virCgroupV1MakeGroup:659 : Done making controllers for group
2019-12-09 10:51:30.082+0000: 4266: debug : virCgroupSetValueRaw:462 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/cpuset.cpus' to '0-1,8-17,24-31'
2019-12-09 10:51:30.082+0000: 4266: debug : virFileClose:114 : Closed fd 36
2019-12-09 10:51:30.082+0000: 4266: error : virCgroupSetValueRaw:472 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dtest820.scope/emulator/cpuset.cpus': Permission denied

3. Or the conf below, which can be also hit the issue as step-2
# virsh dumpxml test820 --inactive |grep "<vcpu" -A3
  <vcpu placement='static' current='3'>8</vcpu>
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>

Expected result: 
Vm should start successfully

Actual result:
VM starts failed

Additional info:
This issue can be reproduced in the following two libvirt, too.
libvirt-5.6.0-6.2.module+el8.1.0+4953+432c8346.x86_64
libvirt-5.6.0-9.module+el8.1.1+4955+f0b25565.x86_64

Comment 1 Pavel Hrdina 2020-03-10 12:25:11 UTC

Fixed in upstream by commit v5.9.0-261-g4c0398b528 and part of downstream build libvirt-5.10.0-1.el8.  Moving to MODIFIED.

Comment 4 jiyan 2020-03-27 11:14:08 UTC

Reproduce this bug with libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7.x86_64 and verified this bug with libvirt-6.0.0-15.module+el8.2.0+6106+b6345808.x86_64

Version:
libvirt-5.9.0-4.module+el8.2.0+4836+a8e32ad7.x86_64
qemu-kvm-4.2.0-16.module+el8.2.0+6092+4f2391c1.x86_64
kernel-4.18.0-192.el8.x86_64

Steps:
1: Configure cpuset.cpus of cgroup
# lscpu | grep On
On-line CPU(s) list: 0-39

# echo "0,2,4,6,8" > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 

# cat /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus 
0,2,4,6,8

2: Create a VM with the following XML
# cat vm.xml
...
  	<vcpu placement="static">2</vcpu>
	<cputune>
	    <vcpupin vcpu="0" cpuset="2"/>
	    <vcpupin vcpu="1" cpuset="4"/>
	</cputune>
	<cpu>
		<numa>
			<cell id="0" cpus="1" memory="1048576" memAccess="shared"/>
		</numa>
	</cpu>
	<numatune>
		<memory mode="strict" nodeset="0"/>
	</numatune>


# virsh create vm.xml 
error: Failed to create domain from vm.xml
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/cpuset.cpus': Permission denied

3. Check libvirtd log after step-2
# tail -f /var/log/libvirt/libvirtd.log
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupV1MakeGroup:613 : Make group /machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupV1MakeGroup:633 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupV1MakeGroup:633 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupV1MakeGroup:633 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupV1CpuSetInherit:557 : Setting up inheritance /machine.slice/machine-qemu\x2d1\x2dvm.scope -> /machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupGetValueRaw:489 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/cpuset.cpus
2020-03-27 11:03:25.845+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupV1CpuSetInherit:567 : Inherit cpuset.cpus = 0,2,4,6,8
2020-03-27 11:03:25.845+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/cpuset.cpus' to '0,2,4,6,8'
2020-03-27 11:03:25.846+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupGetValueRaw:489 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/cpuset.mems
2020-03-27 11:03:25.846+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1CpuSetInherit:567 : Inherit cpuset.mems = 0-1
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/cpuset.mems' to '0-1'
2020-03-27 11:03:25.846+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupGetValueRaw:489 : Get value /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/cpuset.memory_migrate
2020-03-27 11:03:25.846+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1CpuSetInherit:567 : Inherit cpuset.memory_migrate = 1
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/cpuset.memory_migrate' to '1'
2020-03-27 11:03:25.846+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:626 : Skipping unmounted controller memory
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:626 : Skipping unmounted controller devices
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:626 : Skipping unmounted controller freezer
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:626 : Skipping unmounted controller blkio
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:626 : Skipping unmounted controller net_cls
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:626 : Skipping unmounted controller perf_event
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:619 : Not creating systemd controller group
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupV1MakeGroup:672 : Done making controllers for group
2020-03-27 11:03:25.846+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/cpuset.cpus' to '0-39'
2020-03-27 11:03:25.846+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.846+0000: 13417: error : virCgroupSetValueRaw:473 : Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/emulator/cpuset.cpus': Permission denied

4. Update libvirt and restart libvirtd
# yum update libvirt* -y

# rpm -qa libvirt
libvirt-6.0.0-15.module+el8.2.0+6106+b6345808.x86_64

# systemctl restart libvirtd

5. Restart step-2 and step-3
# virsh create vm.xml 
Domain vm created from vm.xml

# tail -f /var/log/libvirt/libvirtd.log
2020-03-27 11:03:25.841+0000: 13417: debug : virCgroupV1MakeGroup:613 : Make group /machine.slice/machine-qemu\x2d1\x2dvm.scope
2020-03-27 11:03:25.841+0000: 13417: debug : virCgroupV1MakeGroup:633 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm.scope/
2020-03-27 11:03:25.841+0000: 13417: debug : virCgroupV1MakeGroup:633 : Make controller /sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d1\x2dvm.scope/
2020-03-27 11:03:25.841+0000: 13417: debug : virCgroupV1MakeGroup:633 : Make controller /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupV1CpuSetInherit:557 : Setting up inheritance /machine.slice -> /machine.slice/machine-qemu\x2d1\x2dvm.scope
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupGetValueRaw:489 : Get value /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus
2020-03-27 11:03:25.842+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupV1CpuSetInherit:567 : Inherit cpuset.cpus = 0,2,4,6,8
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/cpuset.cpus' to '0,2,4,6,8'
2020-03-27 11:03:25.842+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupGetValueRaw:489 : Get value /sys/fs/cgroup/cpuset/machine.slice/cpuset.mems
2020-03-27 11:03:25.842+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupV1CpuSetInherit:567 : Inherit cpuset.mems = 0-1
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/cpuset.mems' to '0-1'
2020-03-27 11:03:25.842+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupGetValueRaw:489 : Get value /sys/fs/cgroup/cpuset/machine.slice/cpuset.memory_migrate
2020-03-27 11:03:25.842+0000: 13417: debug : virFileClose:114 : Closed fd 37
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupV1CpuSetInherit:567 : Inherit cpuset.memory_migrate = 0
2020-03-27 11:03:25.842+0000: 13417: debug : virCgroupSetValueRaw:463 : Set value '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dvm.scope/cpuset.memory_migrate' to '0'

All the test results are as expected, move this bug to be verified.

Comment 6 errata-xmlrpc 2020-05-05 09:52:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2017