Bug 1850230
| Summary: | Using toolbox with fedora:latest image fails, exec fails with "OCI runtime command not found" | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Caden Marchese <cmarches> |
| Component: | toolbox | Assignee: | Jindrich Novy <jnovy> |
| Status: | CLOSED ERRATA | QA Contact: | Micah Abbott <miabbott> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 8.1 | CC: | ajia, aos-bugs, bbaude, bbreard, debarshir, dornelas, dwalsh, harrymichal, imcleod, jligon, jnovy, jokerman, lsm5, mheon, miabbott, mnguyen, nagrawal, nstielau, pducai, rr193m, tsweeney, ypu |
| Target Milestone: | rc | ||
| Target Release: | 8.3 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | toolbox-0.0.8-1.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-11-04 03:05:17 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1186913, 1804543 | ||
|
Description
Caden Marchese
2020-06-23 18:53:53 UTC
Testing this on 4.4.3
# ./oc debug node/worker-1.sharedocp4upi44.lab.upshift.rdu2.redhat.com
Starting pod/worker-1sharedocp4upi44labupshiftrdu2redhatcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.92.62
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# rpm -q podman conmon toolbox
podman-1.6.4-12.rhaos4.4.el8.x86_64
conmon-2.0.15-1.rhaos4.4.el8.x86_64
toolbox-0.0.7-1.rhaos4.4.el8.noarch
sh-4.4# bash -x -e toolbox
+ set -eo pipefail
+ trap cleanup EXIT
+ REGISTRY=registry.redhat.io
+ IMAGE=rhel8/support-tools
+ TOOLBOX_NAME=toolbox-
+ TOOLBOXRC=/root/.toolboxrc
+ '[' '!' -n '' ']'
+ set /bin/sh
+ main /bin/sh
+ setup
+ '[' -f /root/.toolboxrc ']'
+ echo '.toolboxrc file detected, overriding defaults...'
.toolboxrc file detected, overriding defaults...
+ source /root/.toolboxrc
++ REGISTRY=docker.io
++ IMAGE=fedora:latest
+ TOOLBOX_IMAGE=docker.io/fedora:latest
+ [[ /bin/sh =~ ^(--help|-h)$ ]]
+ run /bin/sh
+ image_exists
+ sudo podman inspect docker.io/fedora:latest
++ image_runlabel
++ sudo podman container runlabel --display RUN docker.io/fedora:latest
Error: docker.io/fedora:latest does not have a label of RUN
+ local runlabel=
+ container_exists
+ sudo podman inspect toolbox-
+ echo 'Spawning a container '\''toolbox-'\'' with image '\''docker.io/fedora:latest'\'''
Spawning a container 'toolbox-' with image 'docker.io/fedora:latest'
+ [[ -z '' ]]
+ container_create
+ sudo podman create --hostname toolbox --name toolbox- --network host --privileged --security-opt label=disable --tty --volume /:/media/root:rslave docker.io/fedora:latest
f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727
++ container_state
++ sudo podman inspect toolbox- --format '{{.State.Status}}'
+ local state=configured
+ [[ configured == configured ]]
+ container_start
+ sudo podman start toolbox-
toolbox-
+ echo 'Container started successfully. To exit, type '\''exit'\''.'
Container started successfully. To exit, type 'exit'.
+ container_exec /bin/sh
+ sudo podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/21/attr/exec: no such file or directory": OCI runtime command not found error
+ cleanup
+ sudo podman stop toolbox-
sh-4.4# podman start toolbox-
toolbox-
sh-4.4# sudo podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/20/attr/exec: no such file or directory": OCI runtime command not found error
sh-4.4# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f8421fbd1d76 docker.io/library/fedora:latest /bin/bash 5 minutes ago Up 3 minutes ago toolbox-
sh-4.4# podman inspect toolbox-
[
{
"Id": "f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727",
"Created": "2020-06-24T16:40:03.11924902Z",
"Path": "/bin/bash",
"Args": [
"/bin/bash"
],
"State": {
"OciVersion": "1.0.1-dev",
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1431040,
"ConmonPid": 1431027,
"ExitCode": 0,
"Error": "",
"StartedAt": "2020-06-24T16:41:16.768655413Z",
"FinishedAt": "2020-06-24T16:40:06.236030042Z",
"Healthcheck": {
"Status": "",
"FailingStreak": 0,
"Log": null
}
},
"Image": "adfbfa4a115a799771d3060d0aa213584c91e549187da4fb0036240294ca4a8f",
"ImageName": "docker.io/library/fedora:latest",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/resolv.conf",
"HostnamePath": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/hostname",
"HostsPath": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/hosts",
"StaticDir": "/var/lib/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata",
"OCIConfigPath": "/var/lib/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/config.json",
"OCIRuntime": "runc",
"LogPath": "/var/lib/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/ctr.log",
"ConmonPidFile": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/conmon.pid",
"Name": "toolbox-",
"RestartCount": 0,
"Driver": "overlay",
"MountLabel": "system_u:object_r:container_file_t:s0:c253,c776",
"ProcessLabel": "",
"AppArmorProfile": "",
"EffectiveCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"BoundingCaps": [
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_KILL",
"CAP_SETGID",
"CAP_SETUID",
"CAP_SETPCAP",
"CAP_LINUX_IMMUTABLE",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_ADMIN",
"CAP_NET_RAW",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_SYS_MODULE",
"CAP_SYS_RAWIO",
"CAP_SYS_CHROOT",
"CAP_SYS_PTRACE",
"CAP_SYS_PACCT",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_NICE",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_MKNOD",
"CAP_LEASE",
"CAP_AUDIT_WRITE",
"CAP_AUDIT_CONTROL",
"CAP_SETFCAP",
"CAP_MAC_OVERRIDE",
"CAP_MAC_ADMIN",
"CAP_SYSLOG",
"CAP_WAKE_ALARM",
"CAP_BLOCK_SUSPEND",
"CAP_AUDIT_READ"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "overlay",
"Data": {
"LowerDir": "/var/lib/containers/storage/overlay/3b53a1a1ef4a56c8148e562801ca7cf92c2e4d342c1bbb8ccd6f0810bebd5628/diff",
"MergedDir": "/var/lib/containers/storage/overlay/024be59aa281eb63b0d20b8b6f03a4c63bbc7e69e33bf9c03bfba5aaa1f72cc6/merged",
"UpperDir": "/var/lib/containers/storage/overlay/024be59aa281eb63b0d20b8b6f03a4c63bbc7e69e33bf9c03bfba5aaa1f72cc6/diff",
"WorkDir": "/var/lib/containers/storage/overlay/024be59aa281eb63b0d20b8b6f03a4c63bbc7e69e33bf9c03bfba5aaa1f72cc6/work"
}
},
"Mounts": [
{
"Type": "bind",
"Name": "",
"Source": "/",
"Destination": "/media/root",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": true,
"Propagation": "rslave"
}
],
"Dependencies": [],
"NetworkSettings": {
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": [],
"SandboxKey": "",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": ""
},
"ExitCommand": [
"/usr/bin/podman",
"--root",
"/var/lib/containers/storage",
"--runroot",
"/var/run/containers/storage",
"--log-level",
"error",
"--cgroup-manager",
"systemd",
"--tmpdir",
"/var/run/libpod",
"--runtime",
"runc",
"--storage-driver",
"overlay",
"--events-backend",
"journald",
"container",
"cleanup",
"f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727"
],
"Namespace": "",
"IsInfra": false,
"Config": {
"Hostname": "toolbox",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"TERM=xterm",
"HOSTNAME=toolbox",
"container=podman",
"FGC=f31",
"FBR=f31",
"DISTTAG=f31container",
"HOME=/root"
],
"Cmd": [
"/bin/bash"
],
"Image": "docker.io/library/fedora:latest",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "",
"OnBuild": null,
"Labels": {
"maintainer": "Clement Verna <cverna>"
},
"Annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.ContainerType": "sandbox",
"io.kubernetes.cri-o.Created": "2020-06-24T16:40:03.11924902Z",
"io.kubernetes.cri-o.TTY": "true",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.label": "disable",
"io.podman.annotations.privileged": "TRUE",
"io.podman.annotations.publish-all": "FALSE",
"org.opencontainers.image.stopSignal": "15"
},
"StopSignal": 15
},
"HostConfig": {
"Binds": [
"/:/media/root:rslave,rw,rbind"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "k8s-file",
"Config": null
},
"NetworkMode": "host",
"PortBindings": {},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"label=disable"
],
"Tmpfs": {},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 4096,
"Ulimits": [
{
"Name": "RLIMIT_NOFILE",
"Soft": 1048576,
"Hard": 1048576
},
{
"Name": "RLIMIT_NPROC",
"Soft": 1048576,
"Hard": 1048576
}
],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0
}
}
]
sh-4.4# podman stop toolbox- && podman rm toolbox-
f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727
f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727
* Manually creating the container the same way toolbox does
sh-4.4# podman create --hostname toolbox --name toolbox- --network host --privileged --security-opt label=disable --tty --volume /:/media/root:rslave docker.io/edora:latest
e59a5c5fb330a6514deecfbc45c99408e1149bdf82cdbfb9f10363a640637e71
sh-4.4# podman start toolbox-
toolbox-
sh-4.4# podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/21/attr/exec: no such file or directory": OCI runtime command not found error
* Testing with support-tools image instead of fedora
sh-4.4# podman create --hostname toolbox --name toolbox- --network host --privileged --security-opt label=disable --tty --volume /:/media/root:rslave registry.rdhat.io/rhel8/support-tools:latest
aff1dd7ab78564bdd46961a1d8ba05fda7cab9512d55a85a689af76a0b820ed4
sh-4.4# podman start toolbox-
toolbox-
sh-4.4# podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/15/attr/exec: no such file or directory": OCI runtime command not found error
Seems like an issue with podman and/or the way that the contaienr is being created.
`toolbox` is in this weird place, where it is part of `container-tools` module but the code is generally owned/maintained by the CoreOS team. I'm moving the BZ, but I'll stick this on our Jira board, aiming to have it addressed during the OCP 4.6 cycle. The default image (registry.redhat.io/rhel8/support-tools) uses the image's run label to run the image. docker.io/fedora:latest does not have a run label so it is started differently. This error only shows up when running inside the oc debug container and chrooted into /host. Adding the `-pid host` option (which is included in the default image run label) gets rid of the error message but only if `podman run` is used. Using `podman create` with `-pid=host` then `podman start` and `podman exec` doesn't work as expected. When running directly on the machine (via console or SSH), toolbox has no issues running a different image so this is probably an issue with running nested podman. These issues can be fixed by starting all toolbox containers with the same options as the default image's run label options. `toolbox` seems redundant since `oc debug node` was introduced. `oc debug node/<node_name> already puts users inside the rhel support-tools container. Users can use `oc debug node/<node_name> --image=docker.io/fedora:latest` if they want to use another image. Just a note that toolbox gives us a mechanism to capture data from the node when the cluster is down (ie, during installation, failing api-server, etc.) It's also the tool we test our data collection tool, sosreport, against, and I don't believe 'oc debug node' sets up the container properly to run it. I think we're seeing users try to use different, untested images because it's not possible to get access to entitled RHEL content on an OpenShift cluster (in a sane way), so additional debug tools aren't available via yum. Should we move this over to the podman folks and see if they can pinpoint the problem with podman create/start? Once that works toolbox could be updated to use '--pid=host' for images without a run label, correct? We may not necessarily want to specifically enable this use-case on RHCOS, but the underlying issue sounds like a podman bug. @derrick, lets move it to podman to see if they can pinpoint the issue and they can move it back if its the expected behavior. My initial impression is that this is probably a runc issue - it wants to grab a file from /proc of the init process in the toolbox container (I believe to set security attributes of the exec session to match the original container, but it's been a while since I was this deep in runc). It's failing to open the file, because of the `chroot` masking /proc from us (why it works with `--pid=host` I don't know; maybe the check in the `runc exec` is skipped in that case?). My advice here is going to be to add the `--pid=host` to Toolbox, given it makes runc happy and doesn't really change much in a security sense in the context of a toolbox container that already has access to the host. It may also be possible to mount over `/proc` in the chroot with a fresh procfs (I haven't actually run an `oc debug` container to confirm, but my suspicion is we're seeing the host's `/proc` coming in through the mount, which has greatly different PIDs from the ones we're expecting in the container. It's also possible that procfs isn't being propagated through the bind mount and `/proc/` in the chroot is just empty. Both should be able to be resolved with a procfs mount over it.) Here is a reproducer:
# podman run -it --privileged --net=host --restart=no --pid=host -v /:/host registry.redhat.io/rhel7/support-tools /bin/sh
# chroot /host
# podman create \
--hostname toolbox \
--name coreos-toolbox \
--network host \
--privileged \
--security-opt label=disable \
--tty \
--volume /:/media/root:rslave \
docker.io/fedora:latest
# podman start coreos-toolbox
# podman exec \
--env LANG="$LANG" \
--env TERM="$TERM" \
--tty \
--interactive \
coreos-toolbox \
/bin/sh
@mheon would that also explain this behavior? # podman run -it --privileged --net=host --restart=no --pid=host -v /:/host registry.redhat.io/rhel7/support-tools /bin/sh # chroot /host # podman container runlabel --name mytoolbox RUN registry.redhat.io/rhel8/support-tools I am now in a rhel8 support tools container, `cat /etc/os-release` # exit I am now back in the chroot # podman start mytoolbox # podman exec -it mytoolbox /bin/sh I should be back in the rhel8 support tools container but I it seems like I am in the original rhel7 support tools container if i cat `/etc/os-release` The `--pid=host` in the outer container somewhat debunks my initial theory, given that PIDs and outside the container should match. It seems that even `/proc/self/` is functioning correctly inside /proc in the chroot, which genuinely surprises me. The low numbers of the PIDs it's looking at do make me quite suspicious, though. As for the issue you've given - by "the original rhel7 support tools container" - do you mean *outside* the chroot? It seems like it is outside the chroot Directly on RHCOS (not part of a cluster) [root@ibm-p8-kvm-03-guest-02 ~]# cat /etc/os-release NAME="Red Hat Enterprise Linux CoreOS" VERSION="46.82.202007071437-0" VERSION_ID="4.6" OPENSHIFT_VERSION="4.6" RHEL_VERSION="8.2" PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202007071437-0 (Ootpa)" ID="rhcos" ID_LIKE="rhel fedora" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION="4.6" REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION="4.6" OSTREE_VERSION='46.82.202007071437-0' == Simulate `oc debug node` by starting the debug container - note it is running RHEL 7.8 == [root@ibm-p8-kvm-03-guest-02 ~]# podman run -it --privileged --net=host --restart=no --pid=host -v /:/host registry.redhat.io/rhel7/support-tools /bin/sh Trying to pull registry.redhat.io/rhel7/support-tools... Getting image source signatures Copying blob 8e3c93d02cd2 done Copying blob 16b6dc064ae8 [======================================] 54.8MiB / 54.8MiB Copying blob dc0665975713 done Copying config 03dc5763c1 done Writing manifest to image destination Storing signatures sh-4.2# cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.8 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.8" PRETTY_NAME="Red Hat Enterprise Linux Server 7.8 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.8" == chroot /host so we are back into RHEL CoreOS == sh-4.2# chroot /host sh-4.4# cat /etc/os-release NAME="Red Hat Enterprise Linux CoreOS" VERSION="46.82.202007071437-0" VERSION_ID="4.6" OPENSHIFT_VERSION="4.6" RHEL_VERSION="8.2" PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202007071437-0 (Ootpa)" ID="rhcos" ID_LIKE="rhel fedora" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION="4.6" REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION="4.6" OSTREE_VERSION='46.82.202007071437-0' == Run toolbox container - Note it is running RHEL 8.2 == sh-4.4# podman container runlabel --name mytoolbox RUN registry.redhat.io/rhel8/support-tools Trying to pull registry.redhat.io/rhel8/support-tools... Getting image source signatures Copying blob 121d5409a427 done Copying blob a905c078265c done Copying blob ffbd2fd7eca5 done Copying config 8392dc0bfd done Writing manifest to image destination Storing signatures command: podman run -it --name mytoolbox --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=mytoolbox -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest [root@ibm-p8-kvm-03-guest-02 /]# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.2 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.2" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.2 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.2" == Exit and we are back in RHEL CoreOS == [root@ibm-p8-kvm-03-guest-02 /]# exit exit sh-4.4# cat /etc/os-release NAME="Red Hat Enterprise Linux CoreOS" VERSION="46.82.202007071437-0" VERSION_ID="4.6" OPENSHIFT_VERSION="4.6" RHEL_VERSION="8.2" PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202007071437-0 (Ootpa)" ID="rhcos" ID_LIKE="rhel fedora" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform" REDHAT_BUGZILLA_PRODUCT_VERSION="4.6" REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform" REDHAT_SUPPORT_PRODUCT_VERSION="4.6" OSTREE_VERSION='46.82.202007071437-0' == Start the toolbox container again and exec into it -- We should be in RHEL 8.2 but we are actually in RHEL 7.8 == sh-4.4# podman start mytoolbox mytoolbox sh-4.4# podman exec -it mytoolbox /bin/sh sh-4.2# cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.8 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.8" PRETTY_NAME="Red Hat Enterprise Linux Server 7.8 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.8 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.8" sh-4.2# podman sh: podman: command not found New bugfix release of toolbox (0.0.8) is out to address this bug. Verification Steps: 1. oc debug node/<node_name> 2. chroot /host 3. cat << EOF > ~/.toolboxrc IMAGE=fedora:latest REGISTRY=docker.io EOF 4. toolbox 5. verify no errors 1. Reroduced this bug on toolbox-0.0.4-1.module+el8.1.1+4407+ac444e5d sh-4.4# rpm -q toolbox toolbox-0.0.4-1.module+el8.1.1+4407+ac444e5d.x86_64 sh-4.4# toolbox .toolboxrc file detected, overriding defaults... Trying to pull docker.io/fedora:latest... Getting image source signatures Copying blob c7def56d621e done Copying config a368cbcfa6 done Writing manifest to image destination Storing signatures a368cbcfa6789bc347345f6d19132afe138b62ff5373d2aa5f37120277c90b54 Error: docker.io/fedora:latest does not have a label of RUN Spawning a container 'toolbox-' with image 'docker.io/fedora:latest' b34debf2b9e5bec2225f5b8879af484635c77a5289a667b22fd96a22c69a9cbb toolbox- Container started successfully. To exit, type 'exit'. Error: exec failed: container_linux.go:346: starting container process caused "open /proc/self/task/19/attr/exec: no such file or directory": OCI runtime command not found error sh-4.4# Removing debug pod ... 2. Verified it on toolbox-0.0.8-1.module+el8.3.0+7627+c01ededd sh-4.4# rpm -q toolbox toolbox-0.0.8-1.module+el8.3.0+7627+c01ededd.noarch sh-4.4# toolbox .toolboxrc file detected, overriding defaults... Error: docker.io/fedora:latest does not have a label of RUN Spawning a container 'toolbox-' with image 'docker.io/fedora:latest' [root@toolbox /]# ls bin boot dev etc home host lib lib64 lost+found media mnt opt proc root run sbin srv sys tmp usr var *** Bug 1868343 has been marked as a duplicate of this bug. *** Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:4694 |