Bug 1756059

Summary:	Podman errors with: cannot rm state directory `.....` not empty
Product:	[Fedora] Fedora	Reporter:	Ankur Sinha (FranciscoD) <sanjay.ankur>
Component:	podman	Assignee:	Matthew Heon <mheon>
Status:	CLOSED DUPLICATE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	31	CC:	bbaude, dwalsh, frantisek.kluknavsky, gscrivan, jnovy, lsm5, mheon, pbrobinson, pnemade, santiago, sgraf
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-10-01 12:35:20 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ankur Sinha (FranciscoD) 2019-09-26 16:45:51 UTC

Description of problem:
I'm working on the Fedora documentation, which uses podman to run antora. An example build script is here:

https://pagure.io/fedora-docs/quick-docs/blob/master/f/build.sh

In essence, it calls:	
podman run --rm -it -v $(pwd):/antora:z "docker.io/antora/antora" --html-url-extension-style=indexify site.yml


A lot of the times, but not always, this errors:
$ podman run --rm -it -v $(pwd):/antora:z "docker.io/antora/antora" --html-url-extension-style=indexify site.yml

2019-09-26T16:41:42.000845027Z: cannot rm state directory '/run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052': Directory not empty
ERRO[0002] Error removing container cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052: error cleaning up container cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052: error removing container cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052 from runtime: `/usr/bin/crun delete --force cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052` failed: exit status 1 


On inspection, we see that a socket file still exists in the directory:

$ tree /run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052

/run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052
└── notify
    └── notify


$ file /run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052/notify/notify 
/run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052/notify/notify: socket


Version-Release number of selected component (if applicable):
$ rpm -q podman crun
podman-1.5.1-3.git0005792.fc31.x86_64
crun-0.9.1-1.fc31.x86_64


How reproducible:
Almost always, not not always (I've had a run or two where it didn't error)


Steps to Reproduce:
1. git clone https://pagure.io/Fedora-Council/council-docs.git
2. cd council-docs
3. ./build.sh (a number of times maybe)

Actual results:
Errors as given above.

Expected results:
Should not error

Additional info:
Not sure, other docs users on F30 say they've not seen the issue yet.

Comment 1 Daniel Walsh 2019-09-28 06:14:07 UTC

Matt this looks like some of the sd_notify stuff that was just fixed in podman?

Comment 2 Parag Nemade 2019-09-30 05:36:49 UTC

Ankur,

I think if you update podman package from koji https://koji.fedoraproject.org/koji/buildinfo?buildID=1390091 then podman will work for you on Fedora 31.
If its still not working then try to rm ~/.config/containers/libpod.conf and run your ./build.sh

Comment 3 Ankur Sinha (FranciscoD) 2019-09-30 22:18:49 UTC

Hello,

No luck there either from the looks of it:

$ rm ~/.config/containers/libpod.conf -f

$ rpm -q podman
podman-1.6.0-0.2.rc2.git9181c65.fc31.x86_64

$ ./build.sh 

This build script is using Podman to run the build in an isolated environment.

WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
2019-09-30T22:16:23.000681096Z: cannot rm state directory '/run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4': Directory not empty
ERRO[0002] Error removing container 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4: error cleaning up container 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4: error removing container 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4 from runtime: `/usr/bin/crun delete --force 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4` failed: exit status 1 



Is there some other configuration or package I should be modifying/updating too?

Comment 4 Ankur Sinha (FranciscoD) 2019-09-30 22:20:04 UTC

$ tree /run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4
/run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4
└── notify
    └── notify

1 directory, 1 file

$ file /run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4/notify/notify 
/run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4/notify/notify: socket

Comment 5 Matthew Heon 2019-09-30 22:22:21 UTC

@Dan - I feel like crun just added support for sdnotify?

Also, Podman fixing the bug that unconditionally turned it off probably has not helped.

Comment 6 Parag Nemade 2019-10-01 04:42:53 UTC

Strange I cannot reproduce your problem again on any of Fedora 31 systems provided I have podman installed from koji.

Comment 7 Ankur Sinha (FranciscoD) 2019-10-01 09:48:37 UTC

Do I need to restart the system or any services when podman is updated, maybe?

Comment 8 Giuseppe Scrivano 2019-10-01 11:30:58 UTC

can it be the gnome issue where NOTIFY_SOCKET is always defined?

I think it can be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1753328

Comment 9 Ankur Sinha (FranciscoD) 2019-10-01 12:31:22 UTC

(In reply to Giuseppe Scrivano from comment #8)
> can it be the gnome issue where NOTIFY_SOCKET is always defined?
> 
> I think it can be a duplicate of
> https://bugzilla.redhat.com/show_bug.cgi?id=1753328

That does seem to be the case. I rebooted, ensured that NOTIFY_SOCKET was not set and now podman seems to run just fine.

I still get this warning, but it doesn't break anything:
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 

Thanks for the help, everyone. Please close this as a duplicate if that's the correct resolution here.

Comment 10 Giuseppe Scrivano 2019-10-01 12:35:20 UTC


*** This bug has been marked as a duplicate of bug 1753328 ***