Bug 857013

Summary: Failed to run cpu-stats after vcpu hotplug
Product: Red Hat Enterprise Linux 6 Reporter: hongming <honzhang>
Component: libvirtAssignee: Osier Yang <jyang>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.4CC: acathrow, dallan, dyasny, dyuan, mzhan, pkrempa, rwu, yupzhang
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-3.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-21 07:23:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description hongming 2012-09-13 11:42:42 UTC
Description of problem:
Failed to run cpu-stats after vcpu hotplug. The following error occur.
error: Failed to virDomainGetCPUStats()
error: internal error error accessing cgroup cpuacct for vcpu 

Version-Release number of selected component (if applicable):
libvirt-0.10.1-1.el6.x86_64
qemu-kvm-0.12.1.2-2.307.el6.x86_64


How reproducible:
100% 

Steps to Reproduce:
1.# virsh start rhel6.2
Domain rhel6.2 started

2.# virsh vcpucount rhel6.2
maximum      config       160
maximum      live         160
current      config         1
current      live           1

3.# virsh cpu-stats rhel6.2
CPU0:
    cpu_time             7.947330261 seconds
    vcpu_time            7.660429534 seconds
CPU1:
    cpu_time             2.387255727 seconds
    vcpu_time            2.203674969 seconds
CPU2:
    cpu_time             0.523415331 seconds
    vcpu_time            0.322719706 seconds
CPU3:
    cpu_time             0.183748952 seconds
    vcpu_time            0.133049415 seconds
Total:
    cpu_time            11.041750271 seconds
    user_time            0.280000000 seconds
    system_time          2.970000000 seconds

4.# virsh setvcpus rhel6.2 2

5.# virsh vcpucount rhel6.2
maximum      config       160
maximum      live         160
current      config         1
current      live           2

6.# virsh cpu-stats rhel6.2
error: Failed to virDomainGetCPUStats()
error: internal error error accessing cgroup cpuacct for vcpu 
  
Actual results:
Failed to run cpu-stats after vcpu hotplug. 

Expected results:
Run cpu-stats successfully 

Additional info:

Comment 2 Peter Krempa 2012-09-13 12:03:02 UTC
The root issue that causes this bug will probably be very similar/identical to https://bugzilla.redhat.com/show_bug.cgi?id=855218 .

Comment 3 hongming 2012-09-21 06:54:03 UTC
I found another bug as follows . Do I need to file a new bug about it? They have different errors


Description
Failed to set vcpu scheduler parameters after hotplug vcpu.  The following errors occur.
error: Unable to find vcpu cgroup for rhel6.2(vcpu: 1): No such file or directory


Version
libvirt-0.10.2-0rc1.el6.x86_64
qemu-kvm-0.12.1.2-2.307.el6.x86_64

How reproducible:
100%

Steps

1.# virsh start rhel6.2
Domain rhel6.2 started

2.# virsh vcpucount rhel6.2
maximum      config        10
maximum      live          10
current      config         1
current      live           1

3.# virsh setvcpus rhel6.2 2

4.# virsh schedinfo rhel6.2
Scheduler      : posix
cpu_shares     : 2048
vcpu_period    : 1000000
vcpu_quota     : 18446744073709551
emulator_period: 1000000
emulator_quota : 18446744073709551

5.# virsh schedinfo rhel6.2 --set vcpu_quota=1000
Scheduler      : posix
error: Unable to find vcpu cgroup for rhel6.2(vcpu: 1): No such file or
directory

6.# ll /cgroup/cpu/libvirt/qemu/rhel6.2/vcpu1
ls: cannot access /cgroup/cpu/libvirt/qemu/rhel6.2/vcpu1: No such file
or directory


7.# vim /var/log/libvirtd.log
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupNew:617 : New group
/libvirt/qemu/rhel6.2/vcpu1
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 0:cpu at /cgroup/cpu in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 1:cpuacct at /cgroup/cpuacct in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 2:cpuset at /cgroup/cpuset in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 3:memory at /cgroup/memory in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 4:devices at /cgroup/devices in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 5:freezer at /cgroup/freezer in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupDetect:273 :
Detected mount/mapping 6:blkio at /cgroup/blkio in
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupMakeGroup:535 :
Make group /libvirt/qemu/rhel6.2/vcpu1
2012-09-20 03:22:18.687+0000: 1511: debug : virCgroupMakeGroup:560 :
Make controller /cgroup/cpu/libvirt/qemu/rhel6.2/vcpu1/
2012-09-20 03:22:18.687+0000: 1511: error : qemuSetVcpusBWLive:7745 :
Unable to find vcpu cgroup for rhel6.2(vcpu: 1): No such file or directory
2012-09-20 03:22:18.687+0000: 1511: debug : virDomainFree:2281 :
dom=0x7f852800dd70, (VM: name=rhel6.2,
uuid=17210b7f-6bcf-c46d-404e-9652796a93b9)
2012-09-20 03:22:18.687+0000: 1511: debug : virObjectUnref:135 :
OBJECT_UNREF: obj=0x7f852800dd70
2012-09-20 03:22:18.687+0000: 1511: debug : virObjectUnref:137 :
OBJECT_DISPOSE: obj=0x7f852800dd70
2012-09-20 03:22:18.687+0000: 1511: debug : virDomainDispose:225 :
release domain 0x7f852800dd70 rhel6.2 17210b7f-6bcf-c46d-404e-9652796a93b9
2012-09-20 03:22:18.687+0000: 1511: debug : virObjectUnref:135 :
OBJECT_UNREF: obj=0x7f851805b2f0


Actual result
Failed to set vcpu scheduler parameters after hotplug vcpu.

Expect result
Successfully set vcpu scheduler parameters after hotplug vcpu.

Additional info

Comment 4 Dave Allan 2012-10-08 17:14:12 UTC
Osier, FYI Peter thinks that this BZ and BZ 855218 are duplicates.

Comment 5 Osier Yang 2012-10-09 09:39:42 UTC
(In reply to comment #3)
> I found another bug as follows . Do I need to file a new bug about it? They
> have different errors
> 
> 
> Description
> Failed to set vcpu scheduler parameters after hotplug vcpu.  The following
> errors occur.
> error: Unable to find vcpu cgroup for rhel6.2(vcpu: 1): No such file or
> directory
> 

Yes, it's separate bug.

Comment 6 Osier Yang 2012-10-09 14:03:38 UTC
*** Bug 864381 has been marked as a duplicate of this bug. ***

Comment 11 yuping zhang 2012-10-16 07:20:52 UTC
Verified this issue with:
libvirt-0.10.2-3.el6.x86_64
qemu-kvm-0.12.1.2-2.316.el6.x86_64
kernel-2.6.32-287.el6.x86_64
libssh2-1.4.2-1.el6.x86_64

1.Verify issue in Description:
# virsh start RHEL6.4
Domain RHEL6.4 started

# virsh vcpucount RHEL6.4
maximum      config         4
maximum      live           4
current      config         1
current      live           1

# virsh cpu-stats RHEL6.4
CPU0:
	cpu_time             0.453463440 seconds
	vcpu_time            0.409547097 seconds
CPU1:
	cpu_time             0.515977155 seconds
	vcpu_time            0.478160264 seconds
CPU2:
	cpu_time            11.989434488 seconds
	vcpu_time           11.464623158 seconds
CPU3:
	cpu_time             6.079361532 seconds
	vcpu_time            5.480842340 seconds
Total:
	cpu_time            19.041352016 seconds
	user_time            0.600000000 seconds
	system_time          6.360000000 seconds

# virsh setvcpus RHEL6.4 2

# virsh vcpucount RHEL6.4
maximum      config         4
maximum      live           4
current      config         1
current      live           2

# virsh cpu-stats RHEL6.4
CPU0:
	cpu_time             1.007111478 seconds
	vcpu_time            0.491676600 seconds
CPU1:
	cpu_time             0.775450917 seconds
	vcpu_time            0.596590081 seconds
CPU2:
	cpu_time            14.678617165 seconds
	vcpu_time           13.331851397 seconds
CPU3:
	cpu_time             7.744602633 seconds
	vcpu_time            6.376867462 seconds
Total:
	cpu_time            24.205879763 seconds
	user_time            0.630000000 seconds
	system_time          7.130000000 seconds

2.Verify issue in Comment 3
# virsh vcpucount RHEL6.4
maximum      config         4
maximum      live           4
current      config         1
current      live           1

# virsh setvcpus RHEL6.4 2

# virsh schedinfo RHEL6.4
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : -1
emulator_period: 100000
emulator_quota : -1

# virsh schedinfo RHEL6.4 --set vcpu_quota=1000
Scheduler      : posix
cpu_shares     : 1024
vcpu_period    : 100000
vcpu_quota     : 1000
emulator_period: 100000
emulator_quota : -1

# ll /cgroup/cpu/libvirt/qemu/RHEL6.4/vcpu1
total 0
--w--w--w-. 1 root root 0 Oct 15 21:52 cgroup.event_control
-r--r--r--. 1 root root 0 Oct 15 21:52 cgroup.procs
-rw-r--r--. 1 root root 0 Oct 15 21:52 cpu.cfs_period_us
-rw-r--r--. 1 root root 0 Oct 15 21:53 cpu.cfs_quota_us
-rw-r--r--. 1 root root 0 Oct 15 21:52 cpu.rt_period_us
-rw-r--r--. 1 root root 0 Oct 15 21:52 cpu.rt_runtime_us
-rw-r--r--. 1 root root 0 Oct 15 21:52 cpu.shares
-r--r--r--. 1 root root 0 Oct 15 21:52 cpu.stat
-rw-r--r--. 1 root root 0 Oct 15 21:52 notify_on_release
-rw-r--r--. 1 root root 0 Oct 15 21:52 tasks

So change the bug status to VERIFIED.

Comment 12 errata-xmlrpc 2013-02-21 07:23:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2013-0276.html