Are your containers generating large amounts of logs? That's the first thing that comes to mind when I think about disk space explosion.

I'll try and reproduce locally, but given we've never heard about this before, I suspect it must be something about the environment or the containers in question. Can you provide an `lsof` when the system starts to fill up, to determine what the open files count actually is? Also, further details about the containers would be greatly appreciated - can you provide the commands used to start the containers, do they have healthchecks, are you performing a large number of `podman exec` calls into the containers? Finally, the output of `podman info` from an affected user on an affected machine would be helpful.

Do you have fuse-overlayfs installed?  Are you using the vfs driver?

podman info

Assigning to Giuseppe

please also show what files you have in the storage, could you show the output for `find $graphroot` ?

Created attachment 1698445 [details]
lsof command output

Created attachment 1698446 [details]
find command output

Created attachment 1698447 [details]
podman diff command output

Some additional info:
There are two container images used, 1st one is exactly the same as selenium/standalone-firefox-debug (from docker hub offcial repository). 2nd one extends 1st + contains ibm java + jmeter + citrix receiver inside. 
I am attaching podman diff output which does not show many changes. 
The dockerfile of the base image:

##############
FROM selenium/standalone-firefox-debug:latest
LABEL authors=Falcon

USER seluser

RUN mkdir -p /home/seluser/.cache/  && \
    sudo apt -y update  && \
    sudo apt -y upgrade
##############

@Matthew Heon
- The containers are not generating any significant amount of log files. 
- I am attaching lsof output of a time period when system was starting to get filled up. Affected user is 'falconapp021'
- Command used to start containers: podman run -d --name epricer -e SE_OPTS="-sessionTimeout 450" -p 4444:4444 -p 5910:5900 -v /dev/shm:/dev/shm   falcon-selenium-base
- Healthchecks are not used. 
- No podman exec calls are being made.  
- 'Podman info' output as affected user 'falconapp021':
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module+el8.2.0+6368+cf16aa14.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 9adfe850ef954416ea5dd0438d428a60f2139473'
  Distribution:
    distribution: '"rhel"'
    version: "8.2"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1003
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 3685527552
  MemTotal: 16643944448
  OCIRuntime:
    name: runc
    package: runc-1.0.0-65.rc10.module+el8.2.0+6368+cf16aa14.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 7526281216
  SwapTotal: 8589930496
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: b03lciapp021
  kernel: 4.18.0-193.el8.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.2-3.git21fdece.module+el8.2.0+6368+cf16aa14.x86_64
    Version: |-
      slirp4netns version 0.4.2+dev
      commit: 21fdece2737dc24ffa3f01a341b8a6854f8b13b4
  uptime: 650h 51m 15.76s (Approximately 27.08 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  ConfigFile: /home/falconapp021/.config/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.2-5.module+el8.2.0+6368+cf16aa14.x86_64
      Version: |-
        fuse-overlayfs: version 0.7.2
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  GraphRoot: /opt/IBM/falcon/podman/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 2
  RunRoot: /tmp/run-1000
  VolumePath: /opt/IBM/falcon/podman/containers/storage/volumes

@Daniel Walsh
fuse-overlayfs is used.  

@Giuseppe Scrivano
I am attaching the output of the 'find' command.

thanks, the issue is already fixed upstream with https://github.com/containers/fuse-overlayfs/pull/164/

I could reproduce it locally and bisecting fuse-overlayfs, I confirm the patches in https://github.com/containers/fuse-overlayfs/pull/164/ fix it for me.

If you'd like to try with the latest fuse-overlayfs version, you can do something like:

$ wget https://github.com/containers/fuse-overlayfs/releases/download/v1.1.1/fuse-overlayfs-x86_64
$ chmod +x fuse-overlayfs-x86_64
$ podman --storage-opt overlay.mount_program=/tmp/fuse-overlayfs-x86_64 run ....

The fix for this issue is already present in both 8.2.1 and 8.3.0. It will be fixed once they are released.

for QA, a good reproducer for the issue can be found here: https://github.com/containers/fuse-overlayfs/issues/210

We think we have the same problem on RHEL 7.8 also.
>rpm -qa| grep overlayfs
fuse-overlayfs-0.7.2-6.el7_8.x86_64

Yes, we have this problem on RH 7.8 also.
We discovered it running a headless LibreOffice inside an ubi7 container,
but we can also reproduce it with the reproducer from comment 12.

On the very same machine both things (LibreOffice and the reproducer) works,
if we change the storage driver to vfs.

Please provide a fix for RH7 also.

Thank you

(In reply to Christoph Karl from comment #14)
> Yes, we have this problem on RH 7.8 also.
> We discovered it running a headless LibreOffice inside an ubi7 container,
> but we can also reproduce it with the reproducer from comment 12.
> 
> On the very same machine both things (LibreOffice and the reproducer) works,
> if we change the storage driver to vfs.
> 
> Please provide a fix for RH7 also.
> 
> Thank you

Christoph, seems it is too late for 7.8 so I targeted the fix at the 7.9 release in bug 1850661.

Thanks.

RHEL 7 will not be getting any more updates other then CVE Fixes.  It is in the next phase of it's lifetime.  Please move to RHEL8.

JFTR, is fuse-overlayfs-1.0.0-2.el8 going to arrive to CentOS 8.x at any time?

it is part of the last build, not sure how long it takes to get into CentOS

Lokesh, do you happen to know the timing per Giuseppe's comment: https://bugzilla.redhat.com/show_bug.cgi?id=1849557#c24

(In reply to Tom Sweeney from comment #25)
> Lokesh, do you happen to know the timing per Giuseppe's comment:
> https://bugzilla.redhat.com/show_bug.cgi?id=1849557#c24

Not sure honestly, I've seen it vary from days to weeks. Maybe #centos on freenode could provide more info, or you could file this bug on https://bugs.centos.org with a link to this bug.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4694