RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1728242 - Podman stats failed with Error: unable to obtain cgroup stats
Summary: Podman stats failed with Error: unable to obtain cgroup stats
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: podman
Version: 7.6
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Martin Jenner
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-07-09 12:08 UTC by Joy Pu
Modified: 2019-07-29 16:17 UTC (History)
8 users (show)

Fixed In Version: podman-1.4.4-2.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-29 16:17:28 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2019:1907 0 None None None 2019-07-29 16:17:36 UTC

Description Joy Pu 2019-07-09 12:08:47 UTC 
Description of problem:
Podman stats can not get container stats and report cpuacct.usage_sys file is missing:
Error: unable to obtain cgroup stats: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: no such file or directory



Version-Release number of selected component (if applicable):
podman-1.4.4-1.el7.x86_64.rpm

How reproducible:
100%

Steps to Reproduce:
1. Start a container in the host
#podman run -d docker.io/library/alpine:latest top
2.Check the container statistics by podman stats
# podman stats --all --no-stream


Actual results:
Command failed with following error:
Error: unable to obtain cgroup stats: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys: no such file or directory

Expected results:
The command succeed and can report the stats.

Additional info:
The same case works well with podman-1.4.3-1.el7.x86_64.rpm. And the cgroup files is the same for both version:
# ls /sys/fs/cgroup/cpuacct/machine.slice/libpod-c96dd29474cadc5c369f6453465d13f3225f39a252a572f4acc7aa252d248fb3.scope/
cgroup.clone_children  cgroup.procs  cpuacct.usage         cpu.cfs_period_us  cpu.rt_period_us   cpu.shares  notify_on_release
cgroup.event_control   cpuacct.stat  cpuacct.usage_percpu  cpu.cfs_quota_us   cpu.rt_runtime_us  cpu.stat    tasks

So this may caused by the cgroup version is not match the podman request.
Comment 3 Brent Baude 2019-07-09 15:21:01 UTC 
can you add --log-level=debug and report it?
Comment 4 Daniel Walsh 2019-07-09 16:10:38 UTC 
Is this a RHEL7 issue, does cpuacct.usage_sys exists?  Do we need to toggle something in the kernel to turn on accounting?

/sys/fs/cgroup/cpuacct/machine.slice/libpod-c191203a03972ccd8423f392674722451417b7dff0df694e2f02b8254cd28aa2.scope/cpuacct.usage_sys
Comment 5 Giuseppe Scrivano 2019-07-09 16:37:38 UTC 
it seems the kernel is missing the patch for having user_usage and sys_usage.  I'll need to account for it in pkg/cgroups
Comment 6 Giuseppe Scrivano 2019-07-09 16:54:54 UTC 
 I've added a patch to https://github.com/containers/libpod/pull/3506 to address this issue.
Comment 7 Joy Pu 2019-07-10 01:49:27 UTC 
Hi Brent,

This is the output from podman with --log-level debug:
# podman --log-level debug stats --all --no-stream
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
ERRO[0000] unable to obtain cgroup stats: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-5dc2040ad82a521dbbbbb65b159dd2b8c4a480ceb6fb91a8096152468c9d84b7.scope/cpuacct.usage_sys: open /sys/fs/cgroup/cpuacct/machine.slice/libpod-5dc2040ad82a521dbbbbb65b159dd2b8c4a480ceb6fb91a8096152468c9d84b7.scope/cpuacct.usage_sys: no such file or directory
Comment 12 Joy Pu 2019-07-24 05:56:40 UTC 
Test with podman-1.4.4-2.el7.x86_64, seems it works well now. So set this to verified. Details:
# podman run -d docker.io/library/alpine:latest top
Trying to pull docker.io/library/alpine:latest...Getting image source signatures
Copying blob 050382585609 done
Copying config b7b28af77f done
Writing manifest to image destination
Storing signatures
eb4a043d2817e74472d611570b2b097a50212a194974b781ca0ac1aaa3c4667d
# podman stats --all --no-stream

ID             NAME              CPU %   MEM USAGE / LIMIT   MEM %   NET IO        BLOCK IO   PIDS
eb4a043d2817   trusting_carson   --      61.44kB / 8.201GB   0.00%   558B / 516B   -- / --    1
Comment 14 errata-xmlrpc 2019-07-29 16:17:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1907
Note You need to log in before you can comment on or make changes to this bug.