Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Cause: Libvirt supported "emulatorpin" to set the CPU affinity for qemu domain process. However, it overrides
the CPU affinity set by vcpu "auto" placement when creating
cgroup for the domain process. And the CPU affinity set by
vcpu "auto" placement uses the advisory nodeset from numad.
Consequence: The "auto" placement doesn't work for qemu domain process anymore. I.E. It breaks the numad support.
Fix: Inherit the nodeset from numad if the vcpu placement
is "auto", to set the affinity when creating cgroup for
the domain process. And to avoid the corruption, don't allow to change the domain process's CPU affinity via API virDomainPinEmulator if the vcpu placement is "auto".
Result: The vcpu "auto" placement with numad support works
again.
pkgs:
libvirt-0.10.2-6.el6.x86_64
kernel-2.6.32-330.el6.x86_64
qemu-kvm-0.12.1.2-2.316.el6.x86_64
steps:
1. check host numa node
# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 65514 MB
node 0 free: 62387 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 65536 MB
node 1 free: 61425 MB
node distances:
node 0 1
0: 10 11
1: 11 10
2. start a domain with placement as auto.
# virsh start libvirt_test_api
Domain libvirt_test_api started
# virsh dumpxml libvirt_test_api
...
<vcpu placement='auto'>2</vcpu>
<numatune>
<memory mode='strict' placement='auto'/>
</numatune>
...
3. check log
# grep Nodeset /var/log/libvirt/libvirtd.log
2012-10-30 07:53:56.438+0000: 28878: debug : qemuProcessStart:3605 : Nodeset returned from numad: 0
# grep numad /var/log/libvirt/qemu/libvirt_test_api.log
2012-10-30 07:53:56.564+0000: 6159: debug : qemuProcessInitCpuAffinity:1960 : Set CPU affinity with advisory nodeset from numad
2012-10-30 07:53:56.564+0000: 6159: debug : qemuProcessInitNumaMemoryPolicy:1779 : Set NUMA memory policy with advisory nodeset from numad
4. check vcpuinfo
# virsh vcpuinfo libvirt_test_api
VCPU: 0
CPU: 21
State: running
CPU time: 10.9s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 1
CPU: 0
State: running
CPU time: 5.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
5. check process
# ps aux|grep qemu
qemu 8441 36.1 0.0 1396368 26200 ? Sl 16:26 0:02 /usr/libexec/qemu-kvm -name libvirt_test_api -S -M rhel6.4.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -uuid 05867c1a-afeb-300e-e55e-2673391ae080 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/libvirt_test_api.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/libvirt-test-api,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=23,id=hostnet0,vhost=on,vhostfd=24 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:45:c3:8a,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:1 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
# pstree -apnh 8441
qemu-kvm,8441 -name libvirt_test_api -S -M rhel6.4.0 -enable-kvm -m 1024 -smp 2,sockets=2,cores=1,threads=1 -uuid 05867c1a-afeb-300e-e55e-2673391ae080 -nodefconfig -nodefaults -chardevsocket,id=charmonitor,path
├─{qemu-kvm},8461
└─{qemu-kvm},8462
# grep Cpus_allowed_list /proc/8441/status
Cpus_allowed_list: 0-7,16-23
The domain main process cpus allowed list value is as expected.
# sh test.sh
yyyyyyyy--------yyyyyyyy--------
# grep Cpus_allowed_list /proc/8461/status
Cpus_allowed_list: 0-31
# grep Cpus_allowed_list /proc/8462/status
Cpus_allowed_list: 0-31
The guest vcpu processes is untouched and pin to all available host cpus, so the vcpuinfo shows the right info.
6. do vcpupin
# virsh vcpupin libvirt_test_api 1 2
# virsh vcpupin libvirt_test_api 0 22
# virsh vcpuinfo libvirt_test_api
VCPU: 0
CPU: 22
State: running
CPU time: 11.0s
CPU Affinity: ----------------------y---------
VCPU: 1
CPU: 2
State: running
CPU time: 5.5s
CPU Affinity: --y-----------------------------
It's as expected, although vcpupin will not show up in xml since it can not show up with numatune at same time, but as emulator main process is split up with guest vcpu process, this also should be split up then. This problem is another bug which osier will help to file.
As steps show here, this is working now.
also tested on libvirt-0.10.2-4.el6.x86_64, the problem exist on it.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
http://rhn.redhat.com/errata/RHSA-2013-0276.html
The command vcpuinfo don't show CPU Affinity properly. And i'm not sure if there is some selinux configuration need change because of the guest log below virSecuritySELinuxSetSecurityProcessLabel:1569 : label=unconfined_u:system_r:svirt_t:s0:c223,c792 virGetUserIDByName:2561 : User record for user '107' does not exist virGetGroupIDByName:2643 : Group record for group '107' does not exist The function works well in libvirt 0.9.10-21 Version-Release number of selected component (if applicable): libvirt-0.10.2-4.el6.x86_64 qemu-kvm-rhev-0.12.1.2-2.317.el6.x86_64 kernel-2.6.32-323.el6.x86_64 numad-0.5-3.20120316git.el6.x86_64 How reproducible: 100% Steps to Reproduce: 1. # numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 2 4 6 8 10 node 0 size: 10205 MB node 0 free: 9499 MB node 1 cpus: 12 14 16 18 20 22 node 1 size: 8192 MB node 1 free: 7826 MB node 2 cpus: 1 3 5 7 9 11 node 2 size: 6144 MB node 2 free: 5912 MB node 3 cpus: 13 15 17 19 21 23 node 3 size: 8175 MB node 3 free: 7538 MB node distances: node 0 1 2 3 0: 10 20 20 20 1: 20 10 20 20 2: 20 20 10 20 3: 20 20 20 10 2. # virsh start test <vcpu placement='auto'>10</vcpu> <numatune> <memory mode='strict' placement='auto'/> </numatune> 3. # cat /var/log/libvirt/libvirtd.log| grep Nodeset 2012-10-22 10:28:04.603+0000: 7870: debug : qemuProcessStart:3569 : Nodeset returned from numad: 0,3 #cat /var/log/libvirt/qemu/test.log 2012-10-22 10:28:04.718+0000: 7997: debug : qemuProcessInitCpuAffinity:1895 : Setting CPU affinity 2012-10-22 10:28:04.725+0000: 7997: debug : qemuProcessInitCpuAffinity:1915 : Set CPU affinity with advisory nodeset from numad 2012-10-22 10:28:04.726+0000: 7997: debug : qemuProcessInitNumaMemoryPolicy:1765 : Set NUMA memory policy with advisory nodeset from numad 2012-10-22 10:28:04.726+0000: 7997: debug : qemuProcessHook:2675 : Setting up security labelling 2012-10-22 10:28:04.726+0000: 7997: debug : virSecuritySELinuxSetSecurityProcessLabel:1569 : label=unconfined_u:system_r:svirt_t:s0:c223,c792 2012-10-22 10:28:04.727+0000: 7997: debug : virGetUserIDByName:2561 : User record for user '107' does not exist 2012-10-22 10:28:04.727+0000: 7997: debug : virGetGroupIDByName:2643 : Group record for group '107' does not exist 2012-10-22 10:28:04.727+0000: 7997: debug : virSecurityDACSetProcessLabel:869 : Dropping privileges of DEF to 107:107 2012-10-22 10:28:04.728+0000: 7997: debug : qemuProcessHook:2682 : Hook complete ret=0 #cat /proc/`pidof qemu-kvm`/status Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000009 Mems_allowed_list: 0,3 #virsh vcpuinfo test VCPU: 9 CPU: 2 State: running CPU time: 3.3s CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyy------------------------ 4.run compare script #! /bin/sh for i in {0,3}; do numactl --hardware | grep "node $i cpus:" >> cpus; done cat cpus | awk -F':' '{print $2}' > cpus2 for i in {0..23}; do if grep "\b$i\b" cpus2 > /dev/null; then echo -n "y" else echo -n "-" fi done echo > cpus sh compare.sh y-y-y-y-y-y--y-y-y-y-y-y Actual results: CPU Affinity not right Expected results: CPU Affinity should show right Additional info: In 0.9.10-21 , all the result are expected. Guest log: 2012-10-22 10:42:11.054+0000: 9158: debug : qemuProcessInitCpuAffinity:1731 : Setting CPU affinity 2012-10-22 10:42:11.059+0000: 9158: debug : qemuProcessInitCpuAffinity:1749 : Set CPU affinity with advisory nodeset from numad 2012-10-22 10:42:11.059+0000: 9158: debug : qemuProcessInitNumaMemoryPolicy:1599 : Set NUMA memory policy with advisory nodeset from numad 2012-10-22 10:42:11.059+0000: 9158: debug : qemuProcessHook:2512 : Setting up security labelling 2012-10-22 10:42:11.059+0000: 9158: debug : virSecurityDACSetProcessLabel:637 : Dropping privileges of DEF to 107:107 2012-10-22 10:42:11.060+0000: 9158: debug : qemuProcessHook:2519 : Hook complete ret=0 #virsh vcpuinfo test ... VCPU: 9 CPU: 2 State: running CPU time: 3.9s CPU Affinity: y-y-y-y-y-y--y-y-y-y-y-y