Bug 1728242

Summary: Podman stats failed with Error: unable to obtain cgroup stats
Product: Red Hat Enterprise Linux 7 Reporter: Joy Pu <ypu>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Martin Jenner <mjenner>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: dornelas, dwalsh, gscrivan, jligon, jnovy, lsm5, mheon, umohnani
Target Milestone: rcKeywords: Extras, Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-1.4.4-2.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-29 16:17:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joy Pu 2019-07-09 12:08:47 UTC
Description of problem:
Podman stats can not get container stats and report cpuacct.usage_sys file is missing:
Error: unable to obtain cgroup stats: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: no such file or directory



Version-Release number of selected component (if applicable):
podman-1.4.4-1.el7.x86_64.rpm

How reproducible:
100%

Steps to Reproduce:
1. Start a container in the host
#podman run -d docker.io/library/alpine:latest top
2.Check the container statistics by podman stats
# podman stats --all --no-stream


Actual results:
Command failed with following error:
Error: unable to obtain cgroup stats: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: no such file or directory

Expected results:
The command succeed and can report the stats.

Additional info:
The same case works well with podman-1.4.3-1.el7.x86_64.rpm. And the cgroup files is the same for both version:
# ls /sys/fs/cgroup/cpuacct/machine.slice/libpod-c96dd29474cadc5c369f6453465d13f3225f39a252a572f4acc7aa252d248fb3.scope/
cgroup.clone_children  cgroup.procs  cpuacct.usage         cpu.cfs_period_us  cpu.rt_period_us   cpu.shares  notify_on_release
cgroup.event_control   cpuacct.stat  cpuacct.usage_percpu  cpu.cfs_quota_us   cpu.rt_runtime_us  cpu.stat    tasks

So this may caused by the cgroup version is not match the podman request.

Comment 3 Brent Baude 2019-07-09 15:21:01 UTC
can you add --log-level=debug and report it?

Comment 4 Daniel Walsh 2019-07-09 16:10:38 UTC
Is this a RHEL7 issue, does cpuacct.usage_sys exists?  Do we need to toggle something in the kernel to turn on accounting?

/sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys

Comment 5 Giuseppe Scrivano 2019-07-09 16:37:38 UTC
it seems the kernel is missing the patch for having user_usage and sys_usage.  I'll need to account for it in pkg/cgroups

Comment 6 Giuseppe Scrivano 2019-07-09 16:54:54 UTC
 I've added a patch to https://github.com/containers/libpod/pull/3506 to address this issue.

Comment 7 Joy Pu 2019-07-10 01:49:27 UTC
Hi Brent,

This is the output from podman with --log-level debug:
# podman --log-level debug stats --all --no-stream
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
ERRO[0000] unable to obtain cgroup stats: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-5dc2040ad82a521dbbbbb65b159dd2b8c4a480ceb6fb91a8096152468c9d84b7.scope/cpuacct.usage_sys: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-5dc2040ad82a521dbbbbb65b159dd2b8c4a480ceb6fb91a8096152468c9d84b7.scope/cpuacct.usage_sys: no such file or directory

Comment 12 Joy Pu 2019-07-24 05:56:40 UTC
Test with podman-1.4.4-2.el7.x86_64, seems it works well now. So set this to verified. Details:
# podman run -d docker.io/library/alpine:latest top
Trying to pull docker.io/library/alpine:latest...Getting image source signatures
Copying blob 050382585609 done
Copying config b7b28af77f done
Writing manifest to image destination
Storing signatures
eb4a043d2817e74472d611570b2b097a50212a194974b781ca0ac1aaa3c4667d
# podman stats --all --no-stream

ID             NAME              CPU %   MEM USAGE / LIMIT   MEM %   NET IO        BLOCK IO   PIDS
eb4a043d2817   trusting_carson   --      61.44kB / 8.201GB   0.00%   558B / 516B   -- / --    1

Comment 14 errata-xmlrpc 2019-07-29 16:17:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1907