Bug 1692131

Summary: CPU Accounting is not enabled by default
Product: OpenShift Container Platform Reporter: Derek Carr <decarr>
Component: NodeAssignee: Ryan Phillips <rphillips>
Status: CLOSED ERRATA QA Contact: Sunil Choudhary <schoudha>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs, jokerman, mmccomas, sjenning
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1777173 (view as bug list) Environment:
Last Closed: 2019-06-04 10:46:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1777173    

Description Derek Carr 2019-03-24 14:40:37 UTC 
Description of problem:
kubelet logs show the following message:

475] failed to find cgroups of kubelet - cpu and memory cgroup hierarchy not unified.  cpu: /, memory: /system.slice/kubelet.service

Version-Release number of selected component (if applicable):
4.0.0-0.alpha-2019-03-23-103439

How reproducible:
Always

Steps to Reproduce:
1. install cluster
2. ssh to node
3. journalctl -u kubelet

Actual results:
kubelet does not appear to be in a cpu cgroup

Expected results:
kubelet should be in cpu cgroup

Additional info:
If you cat /proc/<pid>/cgroup for kubelet, we see following:

cat /proc/951/cgroup 
12:freezer:/
11:memory:/system.slice/kubelet.service
10:cpuset:/
9:hugetlb:/
8:pids:/system.slice/kubelet.service
7:devices:/
6:net_cls,net_prio:/
5:rdma:/
4:blkio:/
3:cpu,cpuacct:/
2:perf_event:/
1:name=systemd:/system.slice/kubelet.service

It appears that RHEL CoreOS is not enabling default cpu accounting in systemd.

CPU accounting should be enabled by default.

cat /etc/systemd/system.conf | grep CPUAccounting
#DefaultCPUAccounting=no

In 3.x releases, we always enabled cpu accounting as a drop-in.
Comment 1 Seth Jennings 2019-03-25 22:28:23 UTC 
The drop-in is this
https://github.com/openshift/origin/blob/master/contrib/systemd/origin-accounting.conf

I guess we need the MCO to lay this down for us.

Ryan can you do this?
Comment 2 Ryan Phillips 2019-03-26 14:53:57 UTC 
I'll look into it.
Comment 3 Ryan Phillips 2019-03-26 15:11:24 UTC 
PR: https://github.com/openshift/machine-config-operator/pull/581
Comment 4 Ryan Phillips 2019-03-26 15:31:33 UTC 
Oops. Sounds like we want to modify the system.conf within the rpm. Looking into it.
Comment 5 Ryan Phillips 2019-04-01 22:08:45 UTC 
PR merged https://github.com/openshift/machine-config-operator/pull/581 and verified working.

systemctl show -all|grep -e 'DefaultCPUAccounting\|DefaultBlockIOAccounting\|DefaultMemoryAccounting'
Comment 9 errata-xmlrpc 2019-06-04 10:46:21 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758