Bug 689030

This proposed upstream patch may be the solution; I'm checking now:

https://www.redhat.com/archives/libvir-list/2011-March/msg00213.html

This patch enables cgroup controllers as much as possible by skipping
the creation of blkio controller when running with old kenels that
doesn't support multi-level directory for blkio controller.

In POST:
http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-March/msg00484.html

On my system, that patch is helpful, but not a complete fix.  With it I am able to start the container, but I cannot get a console:

[root@dhcp74-119 libvirt]# virsh -c lxc:/// console LXCtest
error: Unable to get domain status
error: internal error Unable to get cgroup for LXCtest

Disabling the blkio controller allows console to succeed.

back to assigned until rest of problem is solved

I'm not sure what to make of this: with the current git head, I am able to get a console with the blkio controller mounted.  AFAIK, there haven't been any code changes in this area, and the system hasn't been updated, so I can't explain why it's now working.

Just for the record, I when I disabled the blkio cgroups controller I did it by commenting it out in /etc/cgconfig.conf:

mount {
	cpuset	= /cgroup/cpuset;
	cpu	= /cgroup/cpu;
	cpuacct	= /cgroup/cpuacct;
	memory	= /cgroup/memory;
	devices	= /cgroup/devices;
	freezer	= /cgroup/freezer;
	net_cls	= /cgroup/net_cls;
	ns	= /cgroup/ns;
#	blkio	= /cgroup/blkio;
}

It works fine for me, I mean could connect to the lxc guest via console successfully with blkio controller mounted, and it looks to me "lxcDomainOpenConsole" has no relationship with cgroup, so it's strange that how could get that error when connecting with console.

Yeah, and it's working fine for me now, too.  I wonder if that error was improperly reported.

Hi Dave,
I saw Vivek Goyal had applied a patch 'blk-cgroup: Allow creation of hierarchical cgroups' to kernel since Mon Nov 15 2010, so I downloaded packages of the kernel before the patch such as 2.6.32-81.el6.x86_64(28 Oct 2010), however, I can't still reproduce the bug on rhel6.0(2.6.32-81.el6.x86_64) with libvirt-0.8.7-14.el6.x86_64, I'm not sure if my kernel version is newer than you or I'm missing something.


Thanks,
Alex

BTW, I haven't installed a real guest, just define by xml and then starting it, I think it should be enough for the bug, if not, please correct me, thanks.

# virsh -c lxc:/// define lxc.xml
Domain vm1 defined from lxc.xml

# virsh -c lxc:/// start vm1
error: Failed to start domain vm1
error: Unknown failure

# virsh -c lxc:/// dumpxml vm1
<domain type='lxc'>
  <name>vm1</name>
  <uuid>62fd647f-bc09-cfbf-80b5-73a243afdee6</uuid>
  <memory>500000</memory>
  <currentMemory>500000</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/bin/sh</init>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/libvirt_lxc</emulator>
    <interface type='network'>
      <mac address='52:54:00:cb:7f:b9'/>
      <source network='default'/>
    </interface>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
  </devices>
</domain>

Have you tried testing with qemu guests instead of lxc?  Also, are you sure that the blkio cgroup is mounted (that is, is the cgconfig service running)?

(In reply to comment #16)
> Have you tried testing with qemu guests instead of lxc?  Also, are you sure
> that the blkio cgroup is mounted (that is, is the cgconfig service running)?

Hi Eric,
I haven't tried testing with qemu guest, and I can make sure cgroup system has been mounted before previous testing.

The following is qemu guest testing result:
# lscgroup | grep blkio
blkio:/
blkio:/libvirt

Note: libvirt hasn't created a 'qemu' and 'lxc' hypervisor directory under the /cgroup/blkio/libvirt, please see the following additional information. 

# virsh uri
qemu:///system

# virsh define /tmp/vr-rhel6-x86_64-kvm.xml
Domain vr-rhel6-x86_64-kvm defined from /tmp/vr-rhel6-x86_64-kvm.xml

# virsh start vr-rhel6-x86_64-kvm
Domain vr-rhel6-x86_64-kvm started

Note: have no any error information in here.



Additional information:

# service cgconfig status
Running

# lscgroup
cpuset:/
cpuset:/libvirt
cpuset:/libvirt/lxc
cpuset:/libvirt/qemu
cpu:/
cpu:/libvirt
cpu:/libvirt/lxc
cpu:/libvirt/qemu
cpuacct:/
cpuacct:/libvirt
cpuacct:/libvirt/lxc
cpuacct:/libvirt/qemu
memory:/
memory:/libvirt
memory:/libvirt/lxc
memory:/libvirt/qemu
devices:/
devices:/libvirt
devices:/libvirt/lxc
devices:/libvirt/qemu
freezer:/
freezer:/libvirt
freezer:/libvirt/lxc
freezer:/libvirt/qemu
net_cls:/
blkio:/
blkio:/libvirt

# grep 'cgroup' /proc/mounts 
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0

# uname -r
2.6.32-81.el6.x86_64
# rpm -q libcgroup
libcgroup-0.37-1.el6.x86_64
# rpm -q libvirt
libvirt-0.8.7-14.el6.x86_64

Hi Dave and Eric,

I tried to use lxc and qemu guest to reproduce the bug on rhel6.0.z(2.6.32-71.7.1.el6.x86_64) with rhel6.1 libvirt(libvirt-0.8.7-16.el6.x86_64), however, I can't encounter any cgroup issues:

# service cgconfig status
Running

# lscgroup | grep blkio
blkio:/
blkio:/libvirt

For lxc guest, it seems recent libvirt version exists bug for lxc hypervisor, guest will can't be successfully started:
# virsh -c lxc:/// start vm1
error: Failed to start domain vm1
error: Unknown failure

As Dave said, this error perhaps is prior to the cgroups check.

For qemu guest, I can successfully start guest and without any cgroup error.


I have tried different kernel and libvirt version, and also tried different hyperviosr guest, but I still can't get expected cgroup wrong. so I don't know how to continue to test the bug, could you give me some advise?

Thanks,
Alex

Alex, I don't know how to reproduce this BZ on 6.1.  It was determined to be present in 6.1, AFAIK, by code inspection.  I was only able to reproduce it on Fedora 14.  I wish I could give you more guidance, but I've made all the suggestions that I can.  Eric, do you have any further ideas?

I have tried to use lxc and qemu guest to reproduce the bug on rhel6.0.z (2.6.32-71.7.1.el6.x86_64) with rhel6.1 libvirt (libvirt-0.8.7-16.el6.x86_64),
however, I can't encounter any cgroup issues, in other words, I can't reproduce it.

In addition, I have also verified the bug on rhel6.1 snapshot5 (2.6.32-131.0.5.el6.x86_64) with libvirt-0.8.7-18.el6.x86_64, the test result is the same to previous test for lxc and qemu guest, please see Comment 18.

So change the bug to SanityOnly VERIFIED.


Alex

FWIW. A similar issue exists in Fedora 14. The cgroup hierarchy for individual QEMU VMs is not created. Early F14 worked fine. Somewhere along the way, virt-preview updates caused it. I just commented out the blkio controller as suggested above and now get the individual VMs in the other cgroup hierarchies.

[root@kvmhost01 ~]# rpm -q libvirt
libvirt-0.8.8-4.fc14.x86_64
[root@kvmhost01 ~]# rpm -q libcgroup
libcgroup-0.36.2-6.fc14.x86_64
[root@kvmhost01 ~]# uname -r
2.6.35.12-90.fc14.x86_64

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0596.html