Bug 1174125

Created attachment 968824 [details]
libvirtd debug log

Because the vcpu placement is 'auto', so cpuset is set according to numad.
But vcpupin is specified, if vcpupin is not included in the result numad returned. "cpuset.cpus: Permission denied" will happen.

So should we add some clear error message or documents to explain to confused user. 

I will post two log: success and fail of this case.

Created attachment 970002 [details]
success for vcpu auto placement and vcpupin

Created attachment 970003 [details]
failure for vcpu auto placement and vcpupin

also met this problem on ppc64le

pkgs:
libvirt-1.2.16-1.el7.ppc64le
qemu-kvm-rhev-2.3.0-5.el7.ppc64le
kernel-3.10.0-282.el7.ppc64le

steps: 
1.
# virsh dumpxml virt-tests-debug-clone
...
  <vcpu placement='auto'>5</vcpu>
  <cputune>
    <shares>262144</shares>
  </cputune>
  <numatune>
    <memory mode='strict' placement='auto'/>
  </numatune>
...

2. start domain
# virsh start virt-tests-debug-clone
error: Failed to start domain virt-tests-debug-clone
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dvirt\x2dtests\x2ddebug\x2dclone.scope/emulator/cpuset.cpus': Permission denied

it's not 100% persent reproducible though

(In reply to Wayne Sun from comment #8)
Could you post the debug logs for that, too?  It does look like a slightly different issue.  It would be nice to gdb to libvirt, break it right after this error and do 'head /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dvirt\x2dtests\x2ddebug\x2dclone.scope/cpuset.cpus /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dvirt\x2dtests\x2ddebug\x2dclone.scope/*/cpuset.cpus' so we know what's happening.  And also know what value libvirt was trying to write there.  But debug logs will be fine if you don't want to debug the process.

Created attachment 1044853 [details]
libvirtd log on ppc64le

Attached the libvirtd log

The problem is that you are using automatic placement and the nodes numad offers (0) are incompatible with the values of upper cpuset.cpus.  You have weird setting there anyway, could you post the output of the following command, please?

head /sys/fs/cgroup/cpuset/{,machine.slice/}cpuset.{mems,cpus}

fail to break gdb exactly at the error, so it take a little time, and the value fail for set with emulator cpuset.cpus:

Breakpoint 1, virCgroupSetValueStr (group=<optimized out>, controller=<optimized out>, key=<optimized out>, value=0x3fff5c005730 "0,8,16,24,32") at util/vircgroup.c:739
739	    ret = 0;
(gdb) 
Continuing.

"0,8,16,24,32" is from node 0, it failed at here.


The value under scope and inherit to under dir:
# cat /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dvirt\\x2dtests\\x2ddebug\\x2dclone.scope/cpuset.cpus /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dvirt\\x2dtests\\x2ddebug\\x2dclone.scope/*/cpuset.cpus
24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152
24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152


don't know why "0,8,16" are missing.

Check on host:
# lscpu
Architecture:          ppc64le
Byte Order:            Little Endian
CPU(s):                160
On-line CPU(s) list:   0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152
Off-line CPU(s) list:  1-7,9-15,17-23,25-31,33-39,41-47,49-55,57-63,65-71,73-79,81-87,89-95,97-103,105-111,113-119,121-127,129-135,137-143,145-151,153-159
Thread(s) per core:    1
Core(s) per socket:    5
Socket(s):             4
NUMA node(s):          4
Model:                 8247-22L
L1d cache:             64K
L1i cache:             32K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0,8,16,24,32
NUMA node1 CPU(s):     40,48,56,64,72
NUMA node16 CPU(s):    80,88,96,104,112
NUMA node17 CPU(s):    120,128,136,144,152

so after figured out the problem might be on node 0, change domain xml to:
# virsh dumpxml virt-tests-debug-clone
...
  <vcpu placement='static' cpuset='0,8,16,24,32'>2</vcpu>
...

# virsh start virt-tests-debug-clone
error: Failed to start domain virt-tests-debug-clone
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dvirt\x2dtests\x2ddebug\x2dclone.scope/emulator/cpuset.cpus': Permission denied

Breakpoint 1, virCgroupSetValueStr (group=<optimized out>, controller=<optimized out>, key=<optimized out>, value=0x3fff4c00aa80 "0,8,16,24,32") at util/vircgroup.c:739
739	    ret = 0;
(gdb) c
Continuing.

# cat /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dvirt\\x2dtests\\x2ddebug\\x2dclone.scope/cpuset.cpus /sys/fs/cgroup/cpuset/machine.slice/machine-qemu\\x2dvirt\\x2dtests\\x2ddebug\\x2dclone.scope/*/cpuset.cpus
24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152
24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152

so confirmed it did cause by node 0

# head /sys/fs/cgroup/cpuset/{,machine.slice/}cpuset.{mems,cpus}
==> /sys/fs/cgroup/cpuset/cpuset.mems <==
0-1,16-17

==> /sys/fs/cgroup/cpuset/cpuset.cpus <==
0,8,16,24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152

==> /sys/fs/cgroup/cpuset/machine.slice/cpuset.mems <==
0-1,16-17

==> /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus <==
24,32,40,48,56,64,72,80,88,96,104,112,120,128,136,144,152

I was searching through the code and found out that the first problem here is most probably fixed by v1.2.14-2-gc9f9fa25d3a2 and v1.2.14-4-gf0fa9080d47b:

commit c9f9fa25d3a26629e0b3c7ce3139ee2f7c47f127
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 27 10:11:00 2015 +0100

    qemu: cgroup: Store auto cpuset instead of re-creating it on demand

commit f0fa9080d47b7aedad6f4884b8879d88688752a6
Author: Peter Krempa <pkrempa>
Date:   Fri Mar 27 10:23:19 2015 +0100

    qemu: cgroup: Properly set up vcpu pinning

Still have to look at the issue Wayne is having in comment #12.  Could you make sure this is reproducible with current libvirt (1.2.17-3, I guess)?  Also please check the issue you had in comment #12, not the test in comment #13.  If you need further assistance, let me know.  I cannot reproduce the issue.

(In reply to Martin Kletzander from comment #15)

> 
> Still have to look at the issue Wayne is having in comment #12.  Could you
> make sure this is reproducible with current libvirt (1.2.17-3, I guess)? 
> Also please check the issue you had in comment #12, not the test in comment
> #13.  If you need further assistance, let me know.  I cannot reproduce the
> issue.

I think the issue in comment #12 is as comment #11 decribed, that incompatible value with upper cpuset.cpus, not numad related as test in comment #13 confirmed.

The incompatible could be simply set by:

1. check on a host without vm started
# rpm -q libvirt kernel
libvirt-1.2.17-3.el7.x86_64
kernel-3.10.0-302.el7.x86_64

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 65514 MB
node 0 free: 60859 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 65536 MB
node 1 free: 61649 MB
node distances:
node   0   1 
  0:  10  11 
  1:  11  10 


# head /sys/fs/cgroup/cpuset/{,machine.slice/}cpuset.{mems,cpus}
==> /sys/fs/cgroup/cpuset/cpuset.mems <==
0-1

==> /sys/fs/cgroup/cpuset/cpuset.cpus <==
0-15
head: cannot open ‘/sys/fs/cgroup/cpuset/machine.slice/cpuset.mems’ for reading: No such file or directory
head: cannot open ‘/sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus’ for reading: No such file or directory

2. start a vm
# virsh start virt-tests-vm1
Domain virt-tests-vm1 started

# head /sys/fs/cgroup/cpuset/{,machine.slice/}cpuset.{mems,cpus}
==> /sys/fs/cgroup/cpuset/cpuset.mems <==
0-1

==> /sys/fs/cgroup/cpuset/cpuset.cpus <==
0-15

==> /sys/fs/cgroup/cpuset/machine.slice/cpuset.mems <==
0-1

==> /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus <==
0-15

3. destroy the vm
# virsh destroy virt-tests-vm1
Domain virt-tests-vm1 destroyed

4. set small set of cpu in cpuset.cpus
# echo 3-15 > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus

# head /sys/fs/cgroup/cpuset/{,machine.slice/}cpuset.{mems,cpus}
==> /sys/fs/cgroup/cpuset/cpuset.mems <==
0-1

==> /sys/fs/cgroup/cpuset/cpuset.cpus <==
0-15

==> /sys/fs/cgroup/cpuset/machine.slice/cpuset.mems <==
0-1

==> /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus <==
3-15

5. start vm with cpuset config

# virsh dumpxml virt-tests-vm1
...
  <vcpu placement='static' cpuset='0-7'>2</vcpu>
...

# virsh start virt-tests-vm1
error: Failed to start domain virt-tests-vm1
error: Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2dvirt\x2dtests\x2dvm1.scope/emulator/cpuset.cpus': Permission denied

so incompatible value caused the fail.

After reset cpuset.cpus value back with:
# echo 0-15 > /sys/fs/cgroup/cpuset/machine.slice/cpuset.cpus

then start the vm:
# virsh start virt-tests-vm1
Domain virt-tests-vm1 started

So, I don't think there is a problem. 
Well, anyway if libvirt could tweak the value to solve the incompatible problem, that'll be better.

Martin, what do you think?

The difference is that if you have only CPUs 3-15 available and you explicitly request cpus 0-7 to be used, then we error out.  That makes sense.  The problem here is that when libvirt is recalculating NUMA nodes to CPUs, then it has to omit offlined cpus (if you offline the cpus, that's how it will get out of the cpuset.cpus).  So I tested it like this:

I moved /usr/bin/numad to /usr/bin/numad_, created wrapper script in /usr/bin/numad that looks like this for example:

#!/usr/bin/env python

import os
import sys

if len(sys.argv) == 3 and sys.argv[1] == '-w':
    print(1)
    sys.exit(0)

os.execv('/usr/bin/numad_', sys.argv)
sys.exit(1)


And then offlined one cpu in numa node 1, and tried to start a domain with vcpu placement='auto'.

(In reply to Martin Kletzander from comment #17)

Ok use the script for numad and offline cpu8 in node1

# cat /sys/devices/system/cpu/cpu8/online 
1

# echo 0 > /sys/devices/system/cpu/cpu8/online 

# virsh dumpxml virt-tests-vm1
...
  <vcpu placement='auto'>2</vcpu>
...

# virsh start virt-tests-vm1
error: Failed to start domain virt-tests-vm1
error: Invalid value '8-15' for 'cpuset.cpus': Invalid argument

so libvirt did not exclude the offline cpu, and it supposed to, right?

Well, yes, but now the problem is 'Invalid argument' and not 'Permission denied'.

Filed a new bug for tracking the problem in comment #18
Bug 1251445 - Fail to start vm with placement auto after offline certain cpu without restart libvirtd 
https://bugzilla.redhat.com/show_bug.cgi?id=1251445

So I went through the problem once again from the start.  The original problem reported is not a big deal, but mainly it's not a bug.  The thing is that we properly error out as the whole specified cpuset (0-3) is not usable because some of those cpus are not usable (not in the system, offline or just removed from the cgrpus cpuset.cpus).  Thus I'm closing this as a NOTABUG.

All the other issues reported in this BZ are various with various root causes, but not related to this BZ. The issues are described in Bug 1251445.

If you encounter similar problem, do not report it into this BZ, but rather search for one already created or create a new one. Thanks.