Bug 1756059 - Podman errors with: cannot rm state directory `.....` not empty
Summary: Podman errors with: cannot rm state directory `.....` not empty
Keywords:
Status: CLOSED DUPLICATE of bug 1753328
Alias: None
Product: Fedora
Classification: Fedora
Component: podman
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Matthew Heon
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-26 16:45 UTC by Ankur Sinha (FranciscoD)
Modified: 2019-10-01 12:35 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-01 12:35:20 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description Ankur Sinha (FranciscoD) 2019-09-26 16:45:51 UTC
Description of problem:
I'm working on the Fedora documentation, which uses podman to run antora. An example build script is here:

https://pagure.io/fedora-docs/quick-docs/blob/master/f/build.sh

In essence, it calls:	
podman run --rm -it -v $(pwd):/antora:z "docker.io/antora/antora" --html-url-extension-style=indexify site.yml


A lot of the times, but not always, this errors:
$ podman run --rm -it -v $(pwd):/antora:z "docker.io/antora/antora" --html-url-extension-style=indexify site.yml

2019-09-26T16:41:42.000845027Z: cannot rm state directory '/run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052': Directory not empty
ERRO[0002] Error removing container cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052: error cleaning up container cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052: error removing container cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052 from runtime: `/usr/bin/crun delete --force cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052` failed: exit status 1 


On inspection, we see that a socket file still exists in the directory:

$ tree /run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052

/run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052
└── notify
    └── notify


$ file /run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052/notify/notify 
/run/user/1000/crun/cd0681394b239fed85eb2fe3c30dd0bd0dc43c87058b19ad85e10e6eb0256052/notify/notify: socket


Version-Release number of selected component (if applicable):
$ rpm -q podman crun
podman-1.5.1-3.git0005792.fc31.x86_64
crun-0.9.1-1.fc31.x86_64


How reproducible:
Almost always, not not always (I've had a run or two where it didn't error)


Steps to Reproduce:
1. git clone https://pagure.io/Fedora-Council/council-docs.git
2. cd council-docs
3. ./build.sh (a number of times maybe)

Actual results:
Errors as given above.

Expected results:
Should not error

Additional info:
Not sure, other docs users on F30 say they've not seen the issue yet.

Comment 1 Daniel Walsh 2019-09-28 06:14:07 UTC
Matt this looks like some of the sd_notify stuff that was just fixed in podman?

Comment 2 Parag Nemade 2019-09-30 05:36:49 UTC
Ankur,

I think if you update podman package from koji https://koji.fedoraproject.org/koji/buildinfo?buildID=1390091 then podman will work for you on Fedora 31.
If its still not working then try to rm ~/.config/containers/libpod.conf and run your ./build.sh

Comment 3 Ankur Sinha (FranciscoD) 2019-09-30 22:18:49 UTC
Hello,

No luck there either from the looks of it:

$ rm ~/.config/containers/libpod.conf -f

$ rpm -q podman
podman-1.6.0-0.2.rc2.git9181c65.fc31.x86_64

$ ./build.sh 

This build script is using Podman to run the build in an isolated environment.

WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
2019-09-30T22:16:23.000681096Z: cannot rm state directory '/run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4': Directory not empty
ERRO[0002] Error removing container 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4: error cleaning up container 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4: error removing container 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4 from runtime: `/usr/bin/crun delete --force 68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4` failed: exit status 1 



Is there some other configuration or package I should be modifying/updating too?

Comment 4 Ankur Sinha (FranciscoD) 2019-09-30 22:20:04 UTC
$ tree /run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4
/run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4
└── notify
    └── notify

1 directory, 1 file

$ file /run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4/notify/notify 
/run/user/1000/crun/68ac7a64119049762e74a6ea25606f9233c0afa320179bc5dc83bb50490c7fa4/notify/notify: socket

Comment 5 Matthew Heon 2019-09-30 22:22:21 UTC
@Dan - I feel like crun just added support for sdnotify?

Also, Podman fixing the bug that unconditionally turned it off probably has not helped.

Comment 6 Parag Nemade 2019-10-01 04:42:53 UTC
Strange I cannot reproduce your problem again on any of Fedora 31 systems provided I have podman installed from koji.

Comment 7 Ankur Sinha (FranciscoD) 2019-10-01 09:48:37 UTC
Do I need to restart the system or any services when podman is updated, maybe?

Comment 8 Giuseppe Scrivano 2019-10-01 11:30:58 UTC
can it be the gnome issue where NOTIFY_SOCKET is always defined?

I think it can be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1753328

Comment 9 Ankur Sinha (FranciscoD) 2019-10-01 12:31:22 UTC
(In reply to Giuseppe Scrivano from comment #8)
> can it be the gnome issue where NOTIFY_SOCKET is always defined?
> 
> I think it can be a duplicate of
> https://bugzilla.redhat.com/show_bug.cgi?id=1753328

That does seem to be the case. I rebooted, ensured that NOTIFY_SOCKET was not set and now podman seems to run just fine.

I still get this warning, but it doesn't break anything:
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 

Thanks for the help, everyone. Please close this as a duplicate if that's the correct resolution here.

Comment 10 Giuseppe Scrivano 2019-10-01 12:35:20 UTC

*** This bug has been marked as a duplicate of bug 1753328 ***


Note You need to log in before you can comment on or make changes to this bug.