Bug 688616

Setting need info for Yaniv to verify correct mechanism to disable after discussing with platform QE

tested several scenario, all documented in the original bug.
in short, completely disabling cgroups is a workaround for this problem.

Haim, would you mind updating this bug with the correct steps.
thanks
Aic

Test cases and steps: 

- unpatched kernel with cgroups enabled - failed 
  * run vm failed on machine running with ~10 guests 
- unpatched kernel with cgroups 'cpu' commented out on conf file - failed  
  * run vm failed on machine running with ~10 guests 
- unpatched kernel with cgroups 'cpu' disabled via kernel cmd line - pass with 
  exception
  * run new vm succeeds ***but*** it takes 20 time longer to run new vm (need to 
    investigate further) 
- unpatched kernel without cgroups (service cgonfig stop) - pass  
  * managed to run 100 new vms  
- patched kernel with cgroups on - pass 
  * managed to run 100 new vms

We can easily add "cgroup_disable=cpu" argument to kernel on installation on RHEL. Alan, I believe a similar action should be done for RHEV-H?

I'm not saying that this is an ugly cowardly "solution" to the probelm, since that's clear to all of us.

(In reply to comment #6)
> We can easily add "cgroup_disable=cpu" argument to kernel on installation on
> RHEL. Alan, I believe a similar action should be done for RHEV-H?
> 
> I'm not saying that this is an ugly cowardly "solution" to the probelm, since
> that's clear to all of us.

Moreover, it'll remain there forever, long after the cgroups bug is fixed...

Haim - comment #4
Please could you verify exactly what we need to do in the vdsm install
Is it just chkconfig cgconfig off ?

Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Preintegration will check all the options for this issue and will come up with the best solution:
#236: [688616] - VDSM should disable cpu cgroup
--------------------------+------------------------
 Reporter:  hateya@…      |      Owner:  mgoldboi@…
     Type:  Feature       |     Status:  new
 Priority:  major         |  Milestone:  Rhev -2.3
Component:  VDSM-libvirt  |    Version:  Rhel6.1
 Keywords:                |
--------------------------+------------------------
 Due to bug 623712 (scalability problems with cgroups 'cpu' on large SMP
 systems), which is a version\scalability blocker, vdsm is required to
 disable part\all cgroups functionality till original bug will be fixed in
 kernel.

 pre-integration should map all possible ways for solution, and provide
 information on how each affect system functionality, so VDSM will take the
 best decision.

 BZ!#688616 !<https://bugzilla.redhat.com/show_bug.cgi?id=688616> has afew
 optional ways of resolution.

 After speaking with Haim, we agreed that all of them should be tested
 so that the best solution (with the least or no side effects) is chosen
 and implemented.

 = '''__Summary:__''' =
 || Bug || Assigned || Description ||
 || || || ||

Since bug 623712 isn't going to be fixed for 6.1, we recommend that vdsm disable all cgroup usage on the host.

Not going to fix since 623712 should be fixed for 6.2

(In reply to comment #14)
> Not going to fix since 623712 should be fixed for 6.2

No. We aren't going to live through this for beta
Only when we have POSITIVE confirmation that the kernel has the fix and it's verified in 6.2 then we can consider undoing this.
I'm also removing the depends on since this is the workaround for this issue, the bugzilla to re-enable cgroups would depend on the 623712 bug, not this one.

Turned off all cgroups (cpuset,ns,cpu,cpuacct,memory,devices,freezer,net_cls,blkio) in kernel parameters, and also turned off cgconfig.

I think that we should also add release notes to that version, since a persistent user can undo all those changes and get the scalability issue again.

Does it make any sense?

(In reply to comment #17)
> Turned off all cgroups
> (cpuset,ns,cpu,cpuacct,memory,devices,freezer,net_cls,blkio) in kernel
> parameters, and also turned off cgconfig.

Why should we turn them all off? as far as I understand https://bugzilla.redhat.com/show_bug.cgi?id=623712#c33 disabling cpu alone should be enough.

After testing that David Naori have made, it seems that we need to turn all cgroups off for the scalability issue to be gone, that's why I turned it all off.

I'm committing a patch that will disable cgroups on installation, hence libvirtd will be aware if the situation from the very beginning.

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
VDSM disables cgroups on installation due to scalability issues.

verified on ic114
vdsm-4.9-63.el6.x86_64
vdsm-debug-plugin-4.9-63.el6.x86_64
vdsm-cli-4.9-63.el6.x86_64
vdsm-debuginfo-4.9-63.el6.x86_64
qemu-kvm-0.12.1.2-2.160.el6.x86_64
qemu-kvm-debuginfo-0.12.1.2-2.160.el6.x86_64
qemu-kvm-tools-0.12.1.2-2.160.el6.x86_64
libvirt-0.8.7-18.el6.x86_64
libvirt-client-0.8.7-18.el6.x86_64
libvirt-python-0.8.7-18.el6.x86_64
libvirt-debuginfo-0.8.7-18.el6.x86_64
libvirt-devel-0.8.7-18.el6.x86_64



cgroup mount is disabled:

[root@blond-vdsg cgroup]# lscgroup 
cgroups can't be listed: Cgroup is not mounted
[root@blond-vdsg cgroup]# cat /proc/mounts |grep cgroup
[root@blond-vdsg cgroup]#

there is still a bug in bootsrap which require a second reboot after host installation to disable cgroup. it will be solved in the next ic. 
for the vdsm - I am verifying this bug since cgroup is disabled by vdsm during install. 

cgroup active: 

[root@blond-vdsg yum.repos.d]# cat /proc/mounts |grep cgroup
cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0
cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0
cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
cgroup /cgroup/memory cgroup rw,relatime,memory 0 0
cgroup /cgroup/devices cgroup rw,relatime,devices 0 0
cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0
cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0
cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0


after vdsm-4.9-63.el6.x86_64 install:

root@blond-vdsg vdsm]# cat /proc/mounts |grep cgroup
[root@blond-vdsg vdsm]#

*** Bug 703095 has been marked as a duplicate of this bug. ***

Now that kernel -171 is finally in the hybrid repo, the code disabling cgroups on installation has been reverted.

http://gerrit.usersys.redhat.com/437

Note that if you have an old host, which had cgroup disabled, you would have to enable it manually by editing grub.conf.

(In reply to comment #28)
> Now that kernel -171 is finally in the hybrid repo, the code disabling cgroups
> on installation has been reverted.
> 
> http://gerrit.usersys.redhat.com/437
> 
> Note that if you have an old host, which had cgroup disabled, you would have to
> enable it manually by editing grub.conf.

This was reverted in b48a4d9b2ce2162135b29695de66f1a250fecd3c
but we need it back for Beta3 with 6.2 kernel, right?

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2011-1782.html