Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2104116

Summary:	unable to start `toolbox` on RHCOS using `podman` 4.0 [4.11.z]
Product:	OpenShift Container Platform	Reporter:	Micah Abbott <miabbott>
Component:	RHCOS	Assignee:	Timothée Ravier <travier>
Status:	CLOSED ERRATA	QA Contact:	Michael Nguyen <mnguyen>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.11	CC:	aaradhak, dornelas, ealcaniz, hhei, jligon, miabbott, mnguyen, mrussell, ngu, nstielau, openshift-bugs-escalate, palonsor, travier
Target Milestone:	---
Target Release:	4.11.z
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:	2093040	Environment:
Last Closed:	2022-09-28 05:09:53 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2093040
Bug Blocks:	2049591, 2105456

Description Micah Abbott 2022-07-05 14:53:58 UTC

+++ This bug was initially created as a clone of Bug #2093040 +++

RHCOS Version at Install Time: 4.11 dev build
Platform: qemu
Architecture: x86_64


What are you trying to do? What is your use case?

Run `toolbox` on a single RHCOS node that has `podman` 4.0 installed.

What happened? What went wrong or what did you expect?

The `toolbox` command reports that the container is in an unknown state `created`:

```
[core@cosa-devsh ~]$ rpm -q podman toolbox
podman-4.0.2-6.module+el8.6.0+14877+f643d2d6.x86_64
toolbox-0.0.9-1.rhaos4.11.el8.noarch
[core@cosa-devsh ~]$ toolbox
Trying to pull registry.redhat.io/rhel8/support-tools:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob cae7320018ab done  
Copying blob f70d60810c69 done  
Copying blob 545277d80005 done  
Copying config a8419564b7 done  
Writing manifest to image destination
Storing signatures
a8419564b7cc1417d26300d4560ce5812dccaee0c5ec88e22956ad5e67b32037
Spawning a container 'toolbox-core' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...
220b8a1616e3993fcfe479ad283fec70847ac65be533e6d99d15f0afe84bf1fd
Container 'toolbox-core' in unknown state: 'created'
```

What are the steps to reproduce your issue? Please try to reduce these steps to something that can be reproduced with a single RHCOS node.


1.  Build RHCOS 4.11 with RHEL 8.6 content, including `podman` 4.0
2.  `cosa run`
3.  `toolbox`


It looks like in `podman` 4.0 when you do `podman create` the new state reported is `created` but the `toolbox` code is not checking for that state

https://github.com/coreos/toolbox/blob/main/rhcos-toolbox#L56


Workaround:

Manually run the toolbox container:

podman run -it --name toolbox-root --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=toolbox-root -e IMAGE=registry.redhat.io/rhel8/support-tools -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools

--- Additional comment from Micah Abbott on 2022-06-02 19:27:13 UTC ---

This is going to be a problem when we move to RHEL 8.6

--- Additional comment from Aashish Radhakrishnan on 2022-06-13 18:53:59 UTC ---

Addressed the issue by adding the state 'created' in the conditional statements of the toolbox state : https://github.com/coreos/toolbox/pull/78

--- Additional comment from Micah Abbott on 2022-07-05 14:52:43 UTC ---

While it would nice to get this as part of OCP 4.11, the code freeze deadline has passed and we'll have to target this as part of OCP 4.12

We can easily backport this to 4.11.z in the near future.

Comment 1 Aashish Radhakrishnan 2022-07-21 20:56:26 UTC

Addressed the issue by adding the state 'created' in the conditional statements of the toolbox state : https://github.com/coreos/toolbox/pull/78

Comment 4 HuijingHei 2022-09-21 00:33:40 UTC

Test with scratch build in https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=47840764, toolbox works well

[core@cosa-devsh ~]$ rpm -q toolbox
toolbox-0.1.0-1.rhaos4.11.el8.noarch
[core@cosa-devsh ~]$ toolbox
Trying to pull registry.redhat.io/rhel8/support-tools:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 1b3417e31a5e done  
Copying blob 809fe483e885 done  
Copying blob 4c54166b2fbb done  
Copying config 99f45b4332 done  
Writing manifest to image destination
Storing signatures
99f45b4332e9bd0c20f726c38b03e52a9d4e0ed28350b770c2b3f858174b47db
Spawning a container 'toolbox-core' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...
aefb6845361cb1729e356f92d05ffa2bee3b9e66f9e897d6f525ea9ab1940b56
toolbox-core
Container started successfully. To exit, type 'exit'.

Comment 7 Timothée Ravier 2022-09-22 09:37:24 UTC

Moving back to MODIFIED as it's now in a 4.11 build.

Comment 10 Michael Nguyen 2022-09-22 15:27:00 UTC

Verified on  4.11.0-0.nightly-2022-09-22-034852 which is running RHCOS 411.86.202209211811-0 with podman-4.0.2-6.rhaos4.11.el8.x86_64 and toolbox-0.1.0-1.rhaos4.11.el8.noarch

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-09-22-034852   True        False         8m11s   Cluster version is 4.11.0-0.nightly-2022-09-22-034852
$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-129-129.us-east-2.compute.internal   Ready    worker   18m   v1.24.0+3882f8f
ip-10-0-139-157.us-east-2.compute.internal   Ready    worker   16m   v1.24.0+3882f8f
ip-10-0-145-187.us-east-2.compute.internal   Ready    master   23m   v1.24.0+3882f8f
ip-10-0-160-68.us-east-2.compute.internal    Ready    master   24m   v1.24.0+3882f8f
ip-10-0-221-10.us-east-2.compute.internal    Ready    master   23m   v1.24.0+3882f8f
ip-10-0-234-2.us-east-2.compute.internal     Ready    worker   17m   v1.24.0+3882f8f
$ oc debug node/ip-10-0-139-157.us-east-2.compute.internal
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/ip-10-0-139-157us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.139.157
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot host
sh-4.4# rpm-ostree status
State: idle
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:42049fccd994c2bd4be11005a788c6e3e9590e32b2c95bd8174984da45585804
              CustomOrigin: Managed by machine-config-operator
                   Version: 411.86.202209211811-0 (2022-09-21T18:13:54Z)
sh-4.4# rpm -q podman toolbox
podman-4.0.2-6.rhaos4.11.el8.x86_64
toolbox-0.1.0-1.rhaos4.11.el8.noarch
sh-4.4# toolbox
Trying to pull registry.redhat.io/rhel8/support-tools:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 4c54166b2fbb done  
Copying blob 1b3417e31a5e done  
Copying blob 809fe483e885 done  
Copying config 99f45b4332 done  
Writing manifest to image destination
Storing signatures
99f45b4332e9bd0c20f726c38b03e52a9d4e0ed28350b770c2b3f858174b47db
Spawning a container 'toolbox-root' with image 'registry.redhat.io/rhel8/support-tools'
Detected RUN label in the container image. Using that as the default...
b9a5b2fa5c9e83cd909f4d5fd0aa01fb11a1d25a53d30b9d3bb90daa72097ad3
toolbox-root
Container started successfully. To exit, type 'exit'.
[root@ip-10-0-139-157 /]# 
[root@ip-10-0-139-157 /]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
[root@ip-10-0-139-157 /]# exit
exit

sh-4.4# exit
exit
sh-4.4# exit
exit

Removing debug pod ...

Comment 12 errata-xmlrpc 2022-09-28 05:09:53 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.11.6 packages update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6658