Bug 1849557
Summary: | Rootless Podman does not properly close and remove temporary files | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 8 | Reporter: | Mark Kluber <mark.kluber> | ||||||||
Component: | fuse-overlayfs | Assignee: | Giuseppe Scrivano <gscrivan> | ||||||||
Status: | CLOSED ERRATA | QA Contact: | atomic-bugs <atomic-bugs> | ||||||||
Severity: | medium | Docs Contact: | |||||||||
Priority: | unspecified | ||||||||||
Version: | 8.2 | CC: | bbaude, christoph.karl, dwalsh, gscrivan, jligon, jnovy, lsm5, mheon, mjtrangoni, toneata, tsweeney, ypu, yujiang | ||||||||
Target Milestone: | rc | ||||||||||
Target Release: | 8.0 | ||||||||||
Hardware: | x86_64 | ||||||||||
OS: | Linux | ||||||||||
Whiteboard: | |||||||||||
Fixed In Version: | fuse-overlayfs-1.0.0-2.el8 | Doc Type: | If docs needed, set a value | ||||||||
Doc Text: | Story Points: | --- | |||||||||
Clone Of: | |||||||||||
: | 1850661 (view as bug list) | Environment: | |||||||||
Last Closed: | 2020-11-04 03:05:17 UTC | Type: | Bug | ||||||||
Regression: | --- | Mount Type: | --- | ||||||||
Documentation: | --- | CRM: | |||||||||
Verified Versions: | Category: | --- | |||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||
Embargoed: | |||||||||||
Bug Depends On: | |||||||||||
Bug Blocks: | 1850661 | ||||||||||
Attachments: |
|
Description
Mark Kluber
2020-06-22 08:29:56 UTC
Are your containers generating large amounts of logs? That's the first thing that comes to mind when I think about disk space explosion. I'll try and reproduce locally, but given we've never heard about this before, I suspect it must be something about the environment or the containers in question. Can you provide an `lsof` when the system starts to fill up, to determine what the open files count actually is? Also, further details about the containers would be greatly appreciated - can you provide the commands used to start the containers, do they have healthchecks, are you performing a large number of `podman exec` calls into the containers? Finally, the output of `podman info` from an affected user on an affected machine would be helpful. Do you have fuse-overlayfs installed? Are you using the vfs driver? podman info Assigning to Giuseppe please also show what files you have in the storage, could you show the output for `find $graphroot` ? Created attachment 1698445 [details]
lsof command output
Created attachment 1698446 [details]
find command output
Created attachment 1698447 [details]
podman diff command output
Some additional info: There are two container images used, 1st one is exactly the same as selenium/standalone-firefox-debug (from docker hub offcial repository). 2nd one extends 1st + contains ibm java + jmeter + citrix receiver inside. I am attaching podman diff output which does not show many changes. The dockerfile of the base image: ############## FROM selenium/standalone-firefox-debug:latest LABEL authors=Falcon USER seluser RUN mkdir -p /home/seluser/.cache/ && \ sudo apt -y update && \ sudo apt -y upgrade ############## @Matthew Heon - The containers are not generating any significant amount of log files. - I am attaching lsof output of a time period when system was starting to get filled up. Affected user is 'falconapp021' - Command used to start containers: podman run -d --name epricer -e SE_OPTS="-sessionTimeout 450" -p 4444:4444 -p 5910:5900 -v /dev/shm:/dev/shm falcon-selenium-base - Healthchecks are not used. - No podman exec calls are being made. - 'Podman info' output as affected user 'falconapp021': host: BuildahVersion: 1.12.0-dev CgroupVersion: v1 Conmon: package: conmon-2.0.6-1.module+el8.2.0+6368+cf16aa14.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.6, commit: 9adfe850ef954416ea5dd0438d428a60f2139473' Distribution: distribution: '"rhel"' version: "8.2" IDMappings: gidmap: - container_id: 0 host_id: 1003 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 MemFree: 3685527552 MemTotal: 16643944448 OCIRuntime: name: runc package: runc-1.0.0-65.rc10.module+el8.2.0+6368+cf16aa14.x86_64 path: /usr/bin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 7526281216 SwapTotal: 8589930496 arch: amd64 cpus: 8 eventlogger: journald hostname: b03lciapp021 kernel: 4.18.0-193.el8.x86_64 os: linux rootless: true slirp4netns: Executable: /usr/bin/slirp4netns Package: slirp4netns-0.4.2-3.git21fdece.module+el8.2.0+6368+cf16aa14.x86_64 Version: |- slirp4netns version 0.4.2+dev commit: 21fdece2737dc24ffa3f01a341b8a6854f8b13b4 uptime: 650h 51m 15.76s (Approximately 27.08 days) registries: blocked: null insecure: null search: - registry.access.redhat.com - registry.redhat.io - docker.io store: ConfigFile: /home/falconapp021/.config/containers/storage.conf ContainerStore: number: 3 GraphDriverName: overlay GraphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-0.7.2-5.module+el8.2.0+6368+cf16aa14.x86_64 Version: |- fuse-overlayfs: version 0.7.2 FUSE library version 3.2.1 using FUSE kernel interface version 7.26 GraphRoot: /opt/IBM/falcon/podman/containers/storage GraphStatus: Backing Filesystem: xfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" ImageStore: number: 2 RunRoot: /tmp/run-1000 VolumePath: /opt/IBM/falcon/podman/containers/storage/volumes @Daniel Walsh fuse-overlayfs is used. @Giuseppe Scrivano I am attaching the output of the 'find' command. thanks, the issue is already fixed upstream with https://github.com/containers/fuse-overlayfs/pull/164/ I could reproduce it locally and bisecting fuse-overlayfs, I confirm the patches in https://github.com/containers/fuse-overlayfs/pull/164/ fix it for me. If you'd like to try with the latest fuse-overlayfs version, you can do something like: $ wget https://github.com/containers/fuse-overlayfs/releases/download/v1.1.1/fuse-overlayfs-x86_64 $ chmod +x fuse-overlayfs-x86_64 $ podman --storage-opt overlay.mount_program=/tmp/fuse-overlayfs-x86_64 run .... The fix for this issue is already present in both 8.2.1 and 8.3.0. It will be fixed once they are released. for QA, a good reproducer for the issue can be found here: https://github.com/containers/fuse-overlayfs/issues/210 We think we have the same problem on RHEL 7.8 also.
>rpm -qa| grep overlayfs
fuse-overlayfs-0.7.2-6.el7_8.x86_64
Yes, we have this problem on RH 7.8 also. We discovered it running a headless LibreOffice inside an ubi7 container, but we can also reproduce it with the reproducer from comment 12. On the very same machine both things (LibreOffice and the reproducer) works, if we change the storage driver to vfs. Please provide a fix for RH7 also. Thank you (In reply to Christoph Karl from comment #14) > Yes, we have this problem on RH 7.8 also. > We discovered it running a headless LibreOffice inside an ubi7 container, > but we can also reproduce it with the reproducer from comment 12. > > On the very same machine both things (LibreOffice and the reproducer) works, > if we change the storage driver to vfs. > > Please provide a fix for RH7 also. > > Thank you Christoph, seems it is too late for 7.8 so I targeted the fix at the 7.9 release in bug 1850661. Thanks. RHEL 7 will not be getting any more updates other then CVE Fixes. It is in the next phase of it's lifetime. Please move to RHEL8. JFTR, is fuse-overlayfs-1.0.0-2.el8 going to arrive to CentOS 8.x at any time? it is part of the last build, not sure how long it takes to get into CentOS Lokesh, do you happen to know the timing per Giuseppe's comment: https://bugzilla.redhat.com/show_bug.cgi?id=1849557#c24 (In reply to Tom Sweeney from comment #25) > Lokesh, do you happen to know the timing per Giuseppe's comment: > https://bugzilla.redhat.com/show_bug.cgi?id=1849557#c24 Not sure honestly, I've seen it vary from days to weeks. Maybe #centos on freenode could provide more info, or you could file this bug on https://bugs.centos.org with a link to this bug. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:4694 |