Bug 2040379

Summary: Podman exe failed to cleanup dir with NFS
Product: Red Hat Enterprise Linux 9 Reporter: Joy Pu <ypu>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Joy Pu <ypu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 9.0CC: bbaude, dwalsh, gscrivan, jnovy, lsm5, mheon, pthomas, santiago, tsweeney, umohnani
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: podman-4.1.0-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-11-15 09:50:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joy Pu 2022-01-13 15:40:36 UTC
Description of problem:
If we try to run commands inside container with podman exec. The podman exec will failed to exit normally and report an error message:
Error: unlinkat /var/lib/containers/storage/overlay-containers/4dfaf90689fb1d9e9af3c2bb5267732f34685645a6e5c3495bb6defa20d19b62/userdata/9fcb65686c00f255332df33c97afe7700add450c7542cf79f3b9c697c19e40ff: directory not empty

Version-Release number of selected component (if applicable):
podman-3.4.5-0.4.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Start a container
# podman run -d quay.io/libpod/busybox top
Trying to pull quay.io/libpod/busybox:latest...
Getting image source signatures
Copying blob 9758c28807f2 done  
Copying config f0b02e9d09 done  
Writing manifest to image destination
Storing signatures
4dfaf90689fb1d9e9af3c2bb5267732f34685645a6e5c3495bb6defa20d19b62

2. Run a command with exec
# podman exec 4dfaf90689fb echo hello
hello
Error: unlinkat /var/lib/containers/storage/overlay-containers/4dfaf90689fb1d9e9af3c2bb5267732f34685645a6e5c3495bb6defa20d19b62/userdata/9fcb65686c00f255332df33c97afe7700add450c7542cf79f3b9c697c19e40ff: directory not empty


Actual results:
podman exec exit with error message

Expected results:
The command can exit normally

Additional info:

Comment 1 Jindrich Novy 2022-01-13 15:46:24 UTC
Related to https://github.com/containers/podman/issues/11594

Comment 2 Giuseppe Scrivano 2022-01-13 18:08:57 UTC
what is the version of conmon used?

It looks similar to an error fixed by: https://github.com/containers/conmon/pull/300

Comment 3 Joy Pu 2022-01-14 04:09:28 UTC
I am using conmon-2.0.31-1.el9.x86_64 which is build at 2021-12-07.

Comment 4 Giuseppe Scrivano 2022-01-14 12:36:44 UTC
PR for podman: https://github.com/containers/podman/pull/12857
PR for conmon: https://github.com/containers/conmon/pull/319

Comment 5 Jindrich Novy 2022-05-10 08:10:45 UTC
Both these PRs are merged in podman-4.1.0 and conmon-2.1.0.

Comment 6 Joy Pu 2022-05-13 16:59:21 UTC
Tested with podman-4.1.0-2.el9.x86_64 and works as expected now. So set the Tested flag.
[root@kvm-01-guest09 ~]# podman run -d quay.io/libpod/busybox top
WARN[0000] Ignoring global metacopy option, not supported with booted kernel 
96910fddaa9ed338e1491efaa8bd38c247286c59f10baf469eaf8fc22911fe75
[root@kvm-01-guest09 ~]# podman ps
CONTAINER ID  IMAGE                          COMMAND     CREATED         STATUS             PORTS       NAMES
96910fddaa9e  quay.io/libpod/busybox:latest  top         21 seconds ago  Up 21 seconds ago              nostalgic_diffie
[root@kvm-01-guest09 ~]# podman exec 96910fddaa9e echo hello
hello

Comment 9 Joy Pu 2022-06-06 14:12:05 UTC
Test with podman-4.1.0-4.el9.x86_64 and it works as expected. So move this to verified:
# podman info
...
  graphStatus:
    Backing Filesystem: nfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1653479445
  BuiltTime: Wed May 25 07:50:45 2022
  GitCommit: ""
  GoVersion: go1.17.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0
[root@sweetpig-9 ~]# podman run -d quay.io/libpod/busybox top
2b5000ffd13754cfaa966feba28abc0dbe7ba3ac26668a52c09e63edff22b498
[root@sweetpig-9 ~]# podman ps
CONTAINER ID  IMAGE                          COMMAND     CREATED         STATUS             PORTS       NAMES
2b5000ffd137  quay.io/libpod/busybox:latest  top         3 seconds ago   Up 3 seconds ago               xenodochial_montalcini
[root@sweetpig-9 ~]# podman exec 2b5000ffd137 echo hello
hello

Comment 11 errata-xmlrpc 2022-11-15 09:50:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: podman security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7954