Bug 2028439

Summary: podman run with --rm will report "device or resource busy" with NFS storage
Product: Red Hat Enterprise Linux 9 Reporter: Joy Pu <ypu>
Component: conmonAssignee: Jindrich Novy <jnovy>
Status: CLOSED CURRENTRELEASE QA Contact: Joy Pu <ypu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 9.0CC: bbaude, dwalsh, jnovy, lsm5, mheon, pehunt, pthomas, tsweeney, umohnani
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: conmon-2.0.31-1.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-16 22:03:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joy Pu 2021-12-02 12:03:19 UTC
Description of problem:
podman run with --rm will report "device or resource busy" with NFS storage. The same error will also be report when try to remove a running container with -f. They should be similar problem.

Version-Release number of selected component (if applicable):
podman-3.4.3-0.7.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. mount NFS to /var/lib/containers/storage to let podman use NFS storage
2. RUN a container with --rm or start a container then try to remove it with -f
# podman run --rm quay.io/libpod/busybox sleep 2

or 

# podman run -d quay.io/libpod/busybox sleep 99
# podman rm -f $cid


Actual results:
Error message is raise by the command line, and the command finished with status 125.

ERRO[0002] Error removing container 5d4d1f0166c5d08db7b1f95b7185214aa0c4324b2848cc346e57badea593d9de: error removing container 5d4d1f0166c5d08db7b1f95b7185214aa0c4324b2848cc346e57badea593d9de root filesystem: 2 errors occurred:
	* unlinkat /var/lib/containers/storage/overlay-containers/5d4d1f0166c5d08db7b1f95b7185214aa0c4324b2848cc346e57badea593d9de/userdata/.nfs0000000006138ca5000003b1: device or resource busy
	* unlinkat /var/lib/containers/storage/overlay-containers/5d4d1f0166c5d08db7b1f95b7185214aa0c4324b2848cc346e57badea593d9de/userdata/.nfs0000000006138ca5000003b1: device or resource busy

Expected results:
container can be delete without the error message

Additional info:

We can remove a stopped container successfully. And if we checked the container status with podman ps -a after the command finished. We can see that the container is actually removed.

Comment 1 Tom Sweeney 2021-12-02 18:52:10 UTC
Aditya, can you take a look at this please?

Comment 2 Daniel Walsh 2021-12-02 20:11:03 UTC
This would be better with Giuseppe.

Comment 3 Giuseppe Scrivano 2021-12-03 08:35:56 UTC
I think we would need something similar to what we do in containers/storage: https://github.com/containers/storage/commit/bfaebafece6b3951bf350f71cb2a695738298479

Comment 4 Giuseppe Scrivano 2021-12-03 08:45:28 UTC
and make sure we have this fix in conmon: https://github.com/containers/conmon/pull/300  (Thanks to Aditya for finding it!)

Comment 5 Giuseppe Scrivano 2021-12-03 10:36:40 UTC
I think we are hitting the conmon issue fixed with: https://github.com/containers/conmon/pull/300

Comment 6 Daniel Walsh 2021-12-03 13:39:55 UTC
Looks like we need a new conmon release.

Comment 7 Tom Sweeney 2021-12-03 21:33:40 UTC
I'm going to set this back to assigned until a new common is released and vendored into Podman.  @pehunt any plans for that?

Comment 8 Peter Hunt 2021-12-06 14:54:31 UTC
version is tagged now! https://github.com/containers/conmon/releases/tag/v2.0.31

Comment 10 Tom Sweeney 2022-05-16 22:03:15 UTC
As the PR to fix this landed in December of 2021, I'm closing this as current release