Bug 2083997

Summary:	catatonit not found when starting pod (podman 4.0 under RHEL 8.6)
Product:	Red Hat Enterprise Linux 8	Reporter:	Miroslav Hostinsky <mhostinsky>
Component:	podman	Assignee:	Valentin Rothberg <vrothber>
Status:	CLOSED ERRATA	QA Contact:	Yuhui Jiang <yujiang>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	8.6	CC:	bbaude, dornelas, dwalsh, gnecasov, jligon, jnovy, lsm5, mheon, pthomas, tsweeney, umohnani, vrothber, ypu, yujiang
Target Milestone:	rc	Keywords:	Reopened, Triaged, ZStream
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	container-tools-rhel8-8060020220615093047.3b538bd8 podman-4.1.1-1.module+el8.6.0+15659+f3ff693b	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2100143 (view as bug list)		Environment:
Last Closed:	2022-11-08 09:15:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2100143

Description Miroslav Hostinsky 2022-05-11 07:49:02 UTC

Description of problem:

Unable to start container when running within pod

Version-Release number of selected component (if applicable):

podman-4.0.2-6.module+el8.6.0+14877+f643d2d6.x86_64
redhat-release-8.6-0.1.el8.x86_64

Steps to Reproduce:

# podman pod create --name pokus && podman run -it --rm --pod pokus alpine sh
926ab05b5645a351002a08418821594718fd9e8a5ff5172f573881d9798e773d
ERRO[0003] Starting some container dependencies
ERRO[0003] "runc: container_linux.go:380: starting container process caused: exec: \"/catatonit\": stat /catatonit: no such file or directory: OCI runtime attempted to invoke a command that was not found"
Error: error starting some containers: internal libpod error

Additional info:

It works when executed with /catatonit in pod

# podman pod create --name pokus -v /usr/libexec/catatonit/catatonit:/catatonit && podman run -it --rm --pod pokus alpine id
4f6222d18c2a355c1f58b478c36a2378cd844eb373cfb56c70f7d1748cf3d865
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel),11(floppy),20(dialout),26(tape),27(video)

Comment 3 Jindrich Novy 2022-05-11 08:19:09 UTC

Miroslav, do you have podman-catatonit installed? Wasn't it forcibly removed by a chance?

Comment 4 Miroslav Hostinsky 2022-05-11 08:21:09 UTC

Yes, it's installed:

podman-catatonit-4.0.2-6.module+el8.6.0+14877+f643d2d6.x86_64

It works if I create pod with bind mount of /usr/libexec/catatonit/catatonit:/catatonit (so it must be installed):
# podman pod create --name pokus -v /usr/libexec/catatonit/catatonit:/catatonit

Comment 5 Miroslav Hostinsky 2022-05-11 08:30:40 UTC

I tried rootless podman instance from scratch and it works (I can see "/catatonit -P" running under pod):

$ podman system reset
$ podman pod create --name pokus
$ podman run -it --rm --pod pokus alpine sh

But, the issue above is when running podman (under root) upgraded from 3->4 (RHEL8.5->8.6). No "podman system reset"/clean install. Just upgraded from 8.5->8.6.

Comment 6 Miroslav Hostinsky 2022-05-11 08:41:13 UTC

Older pods created with podman 3.x before upgrade work (running with /pause in pod). It seems that only new pod images are created without /catatonit.

Comment 7 Valentin Rothberg 2022-05-11 09:16:48 UTC

@Miroslav, could you share the run with `podman --log-level=debug` and share the output?

Also, could you share the output of `podman images`?  All for the failing root user.

Comment 11 Miroslav Hostinsky 2022-05-11 19:31:49 UTC

It seems, that I found the problem. I created first pod with following command (there is no "localhost/podman-pause" image, it needs to be generated when first pod is created):

# podman pod create --name pod-httpd_php --uidmap=0:11900000:65536 --gidmap=0:11900000:65536 --ip 10.165.12.100 --ip6 2001:123:1234::1000
ERRO[0000] Can't add file /var/lib/containers/storage/overlay/1c4de2e5de894afaa1227fdb1eec59c2b763415cff277382d3bdf5e842aea1ee/merged/catatonit to tar: Host ID 0 cannot be mapped to a container ID

There is error (pod is not created), but still, "localhost/podman-pause" image is created with missing /catatonit (podman-pause image size around 3kB).

To fix the issue, I simply removed corrupted "localhost/podman-pause" image, and generated again first pod (in order to create "localhost/podman-pause") and then deleted it:
# podman pod create pokus && podman pod rm pokus

In this case, "localhost/podman-pause" is created with /catatonit binary and my complex pod (with uidmap) now works.

It seems, that there is difference in the podman behavior:
- podman 3.x - pod image is fetched from the hub: k8s.gcr.io/pause
- podman 4.x - pod image is generated locally on the first pod create and this process must be finished without error

Comment 12 Jindrich Novy 2022-05-12 04:45:13 UTC

Miroslav, yes, this is expected behaviour and an actual bugfix - see bug 2059066.

Comment 13 Miroslav Hostinsky 2022-05-12 06:05:37 UTC

OK. Problem solved from my point of view. Thanks!

Comment 14 Jindrich Novy 2022-05-12 06:27:23 UTC

Thanks for reporting this anyway Miroslav!

Comment 15 Valentin Rothberg 2022-05-12 08:01:47 UTC

Bug 2059066 looks different to me.  The one here does not have a relation to containers.conf but there seems to be a bug when creating the _local_ pause image.

I am reopening as I can reproduce.

Comment 17 Jindrich Novy 2022-05-12 08:08:21 UTC

It is a feature Valentin: https://github.com/containers/podman/issues/12771#issuecomment-1021326507

Comment 18 Valentin Rothberg 2022-05-12 08:23:06 UTC

(In reply to Jindrich Novy from comment #17)
> It is a feature Valentin:
> https://github.com/containers/podman/issues/12771#issuecomment-1021326507

I know, I built it :^)

But it is apparently broken when using `--uidmap=0:11900000:65536 --gidmap=0:11900000:65536` as described in this bug.

The issue is that `catatonit` cannot be copied into the locally built image, yet the image is built - only a log is emitted but it does not error out.  When starting the pod, the infra container cannot start (and hence the pod) because `catatonit` is not present in the image.

Comment 19 Valentin Rothberg 2022-05-12 09:36:29 UTC

Until a fix is shipped, I suggest using the infra image with Podman v4.0 again.

You can either use `--infra-image=k8s.gcr.io/pause` when creating the pod or configure /etc/containers/containers.conf as follows:
```
[engine]
  infra_image = "k8s.gcr.io/pause"
```

Comment 20 Valentin Rothberg 2022-05-13 08:11:56 UTC

Upstream PR to fix the issue: https://github.com/containers/podman/pull/14232

Comment 50 errata-xmlrpc 2022-11-08 09:15:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7457