Bug 1850230 - Using toolbox with fedora:latest image fails, exec fails with "OCI runtime command not found"
Summary: Using toolbox with fedora:latest image fails, exec fails with "OCI runtime co...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: toolbox
Version: 8.1
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: rc
: 8.3
Assignee: Jindrich Novy
QA Contact: Micah Abbott
URL:
Whiteboard:
: 1868343 (view as bug list)
Depends On:
Blocks: 1186913 1804543
TreeView+ depends on / blocked
 
Reported: 2020-06-23 18:53 UTC by Caden Marchese
Modified: 2021-08-30 18:21 UTC (History)
22 users (show)

Fixed In Version: toolbox-0.0.8-1.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 03:05:17 UTC
Type: Bug
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4885641 0 None None None 2021-01-18 18:51:07 UTC

Internal Links: 1868343

Description Caden Marchese 2020-06-23 18:53:53 UTC
Description of problem:
Fedora toolbox debugging no longer works in 4.4.8 and 4.4.9 (have not tested this in earlier versions of 4.4). 

$ oc debug node/master-0.cluster.example.com
Starting pod/master-0-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.92.235
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# vi /root/.toolboxrc
sh-4.4# toolbox
.toolboxrc file detected, overriding defaults...
Trying to pull docker.io/fedora:latest...
Getting image source signatures
Copying blob 4c69497db035 done  
Copying config adfbfa4a11 done  
Writing manifest to image destination
Storing signatures
adfbfa4a115a799771d3060d0aa213584c91e549187da4fb0036240294ca4a8f
Error: docker.io/fedora:latest does not have a label of RUN
Spawning a container 'toolbox-' with image 'docker.io/fedora:latest'
e93ac182af4d685fb6779d601d2532659859e30eee83d3f369695de2c0d118fe
toolbox-
Container started successfully. To exit, type 'exit'.
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/21/attr/exec: no such file or directory": OCI runtime command not found error

The toolbox file in question:

REGISTRY=docker.io
IMAGE=fedora:latest

Version-Release number of selected component (if applicable):
4.4.8, 4.4.9

Steps to Reproduce:
1. Follow steps in https://access.redhat.com/solutions/4929021 to change toolboxrc
2. Run # toolbox on the node.

Actual results:
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/21/attr/exec: no such file or directory": OCI runtime command not found error

Expected results:
Remote shell starts successfully.

Additional info:
We use this in support to get dnf access to diagnostic tools not available directly on the RHEL CoreOS nodes. Would be great to continue to have this available in 4.4.

Comment 3 Derrick Ornelas 2020-06-24 17:00:43 UTC
Testing this on 4.4.3

# ./oc debug node/worker-1.sharedocp4upi44.lab.upshift.rdu2.redhat.com
Starting pod/worker-1sharedocp4upi44labupshiftrdu2redhatcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.92.62
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host

sh-4.4# rpm -q podman conmon toolbox
podman-1.6.4-12.rhaos4.4.el8.x86_64
conmon-2.0.15-1.rhaos4.4.el8.x86_64
toolbox-0.0.7-1.rhaos4.4.el8.noarch


sh-4.4# bash -x -e toolbox
+ set -eo pipefail
+ trap cleanup EXIT
+ REGISTRY=registry.redhat.io
+ IMAGE=rhel8/support-tools
+ TOOLBOX_NAME=toolbox-
+ TOOLBOXRC=/root/.toolboxrc
+ '[' '!' -n '' ']'
+ set /bin/sh
+ main /bin/sh
+ setup
+ '[' -f /root/.toolboxrc ']'
+ echo '.toolboxrc file detected, overriding defaults...'
.toolboxrc file detected, overriding defaults...
+ source /root/.toolboxrc
++ REGISTRY=docker.io
++ IMAGE=fedora:latest
+ TOOLBOX_IMAGE=docker.io/fedora:latest
+ [[ /bin/sh =~ ^(--help|-h)$ ]]
+ run /bin/sh
+ image_exists
+ sudo podman inspect docker.io/fedora:latest
++ image_runlabel
++ sudo podman container runlabel --display RUN docker.io/fedora:latest
Error: docker.io/fedora:latest does not have a label of RUN
+ local runlabel=
+ container_exists
+ sudo podman inspect toolbox-
+ echo 'Spawning a container '\''toolbox-'\'' with image '\''docker.io/fedora:latest'\'''
Spawning a container 'toolbox-' with image 'docker.io/fedora:latest'
+ [[ -z '' ]]
+ container_create
+ sudo podman create --hostname toolbox --name toolbox- --network host --privileged --security-opt label=disable --tty --volume /:/media/root:rslave docker.io/fedora:latest
f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727
++ container_state
++ sudo podman inspect toolbox- --format '{{.State.Status}}'
+ local state=configured
+ [[ configured == configured ]]
+ container_start
+ sudo podman start toolbox-
toolbox-
+ echo 'Container started successfully. To exit, type '\''exit'\''.'
Container started successfully. To exit, type 'exit'.
+ container_exec /bin/sh
+ sudo podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/21/attr/exec: no such file or directory": OCI runtime command not found error
+ cleanup
+ sudo podman stop toolbox-


sh-4.4# podman start toolbox-
toolbox-

sh-4.4# sudo podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/20/attr/exec: no such file or directory": OCI runtime command not found error

sh-4.4# podman ps         
CONTAINER ID  IMAGE                            COMMAND    CREATED        STATUS            PORTS  NAMES
f8421fbd1d76  docker.io/library/fedora:latest  /bin/bash  5 minutes ago  Up 3 minutes ago         toolbox-


sh-4.4# podman inspect toolbox-
[
    {
        "Id": "f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727",
        "Created": "2020-06-24T16:40:03.11924902Z",
        "Path": "/bin/bash",
        "Args": [
            "/bin/bash"
        ],
        "State": {
            "OciVersion": "1.0.1-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 1431040,
            "ConmonPid": 1431027,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-06-24T16:41:16.768655413Z",
            "FinishedAt": "2020-06-24T16:40:06.236030042Z",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }
        },
        "Image": "adfbfa4a115a799771d3060d0aa213584c91e549187da4fb0036240294ca4a8f",
        "ImageName": "docker.io/library/fedora:latest",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/resolv.conf",
        "HostnamePath": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/hostname",
        "HostsPath": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/hosts",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata",
        "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/config.json",
        "OCIRuntime": "runc",
        "LogPath": "/var/lib/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/ctr.log",
        "ConmonPidFile": "/var/run/containers/storage/overlay-containers/f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727/userdata/conmon.pid",
        "Name": "toolbox-",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_file_t:s0:c253,c776",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_DAC_READ_SEARCH",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_SETGID",
            "CAP_SETUID",
            "CAP_SETPCAP",
            "CAP_LINUX_IMMUTABLE",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_BROADCAST",
            "CAP_NET_ADMIN",
            "CAP_NET_RAW",
            "CAP_IPC_LOCK",
            "CAP_IPC_OWNER",
            "CAP_SYS_MODULE",
            "CAP_SYS_RAWIO",
            "CAP_SYS_CHROOT",
            "CAP_SYS_PTRACE",
            "CAP_SYS_PACCT",
            "CAP_SYS_ADMIN",
            "CAP_SYS_BOOT",
            "CAP_SYS_NICE",
            "CAP_SYS_RESOURCE",
            "CAP_SYS_TIME",
            "CAP_SYS_TTY_CONFIG",
            "CAP_MKNOD",
            "CAP_LEASE",
            "CAP_AUDIT_WRITE",
            "CAP_AUDIT_CONTROL",
            "CAP_SETFCAP",
            "CAP_MAC_OVERRIDE",
            "CAP_MAC_ADMIN",
            "CAP_SYSLOG",
            "CAP_WAKE_ALARM",
            "CAP_BLOCK_SUSPEND",
            "CAP_AUDIT_READ"
        ],
        "BoundingCaps": [
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_DAC_READ_SEARCH",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_KILL",
            "CAP_SETGID",
            "CAP_SETUID",
            "CAP_SETPCAP",
            "CAP_LINUX_IMMUTABLE",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_BROADCAST",
            "CAP_NET_ADMIN",
            "CAP_NET_RAW",
            "CAP_IPC_LOCK",
            "CAP_IPC_OWNER",
            "CAP_SYS_MODULE",
            "CAP_SYS_RAWIO",
            "CAP_SYS_CHROOT",
            "CAP_SYS_PTRACE",
            "CAP_SYS_PACCT",
            "CAP_SYS_ADMIN",
            "CAP_SYS_BOOT",
            "CAP_SYS_NICE",
            "CAP_SYS_RESOURCE",
            "CAP_SYS_TIME",
            "CAP_SYS_TTY_CONFIG",
            "CAP_MKNOD",
            "CAP_LEASE",
            "CAP_AUDIT_WRITE",
            "CAP_AUDIT_CONTROL",
            "CAP_SETFCAP",
            "CAP_MAC_OVERRIDE",
            "CAP_MAC_ADMIN",
            "CAP_SYSLOG",
            "CAP_WAKE_ALARM",
            "CAP_BLOCK_SUSPEND",
            "CAP_AUDIT_READ"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/3b53a1a1ef4a56c8148e562801ca7cf92c2e4d342c1bbb8ccd6f0810bebd5628/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/024be59aa281eb63b0d20b8b6f03a4c63bbc7e69e33bf9c03bfba5aaa1f72cc6/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/024be59aa281eb63b0d20b8b6f03a4c63bbc7e69e33bf9c03bfba5aaa1f72cc6/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/024be59aa281eb63b0d20b8b6f03a4c63bbc7e69e33bf9c03bfba5aaa1f72cc6/work"
            }
        },
        "Mounts": [
            {
                "Type": "bind",
                "Name": "",
                "Source": "/",
                "Destination": "/media/root",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rslave"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": [],
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": ""
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/var/run/containers/storage",
            "--log-level",
            "error",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/var/run/libpod",
            "--runtime",
            "runc",
            "--storage-driver",
            "overlay",
            "--events-backend",
            "journald",
            "container",
            "cleanup",
            "f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "toolbox",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": true,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "HOSTNAME=toolbox",
                "container=podman",
                "FGC=f31",
                "FBR=f31",
                "DISTTAG=f31container",
                "HOME=/root"
            ],
            "Cmd": [
                "/bin/bash"
            ],
            "Image": "docker.io/library/fedora:latest",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": "",
            "OnBuild": null,
            "Labels": {
                "maintainer": "Clement Verna <cverna@fedoraproject.org>"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.ContainerType": "sandbox",
                "io.kubernetes.cri-o.Created": "2020-06-24T16:40:03.11924902Z",
                "io.kubernetes.cri-o.TTY": "true",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.label": "disable",
                "io.podman.annotations.privileged": "TRUE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15
        },
        "HostConfig": {
            "Binds": [
                "/:/media/root:rslave,rw,rbind"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "Tmpfs": {},
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 4096,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 1048576,
                    "Hard": 1048576
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 1048576,
                    "Hard": 1048576
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        }
    }
]


sh-4.4# podman stop toolbox- && podman rm toolbox-
f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727
f8421fbd1d76c8535f8d8f3b4c13d8ead86ab7cdee0a9dfa02f0a859cee43727



* Manually creating the container the same way toolbox does

sh-4.4# podman create --hostname toolbox --name toolbox- --network host --privileged --security-opt label=disable --tty --volume /:/media/root:rslave docker.io/edora:latest
e59a5c5fb330a6514deecfbc45c99408e1149bdf82cdbfb9f10363a640637e71

sh-4.4# podman start toolbox-
toolbox-

sh-4.4# podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/21/attr/exec: no such file or directory": OCI runtime command not found error



* Testing with support-tools image instead of fedora

sh-4.4# podman create --hostname toolbox --name toolbox- --network host --privileged --security-opt label=disable --tty --volume /:/media/root:rslave registry.rdhat.io/rhel8/support-tools:latest
aff1dd7ab78564bdd46961a1d8ba05fda7cab9512d55a85a689af76a0b820ed4

sh-4.4# podman start toolbox-
toolbox-

sh-4.4# podman exec --env LANG= --env TERM=xterm --tty --interactive toolbox- /bin/sh
Error: exec failed: container_linux.go:349: starting container process caused "open /proc/self/task/15/attr/exec: no such file or directory": OCI runtime command not found error


Seems like an issue with podman and/or the way that the contaienr is being created.

Comment 4 Micah Abbott 2020-06-24 19:31:41 UTC
`toolbox` is in this weird place, where it is part of `container-tools` module but the code is generally owned/maintained by the CoreOS team.

I'm moving the BZ, but I'll stick this on our Jira board, aiming to have it addressed during the OCP 4.6 cycle.

Comment 6 Michael Nguyen 2020-07-10 13:47:57 UTC
The default image (registry.redhat.io/rhel8/support-tools) uses the image's run label to run the image.  docker.io/fedora:latest does not have a run label so it is started differently.  

This error only shows up when running inside the oc debug container and chrooted into /host.  Adding the `-pid host` option (which is included in the default image run label) gets rid of the error message but only if `podman run` is used.  Using `podman create` with `-pid=host` then `podman start` and `podman exec` doesn't work as expected.

When running directly on the machine (via console or SSH), toolbox has no issues running a different image so this is probably an issue with running nested podman.

These issues can be fixed by starting all toolbox containers with the same options as the default image's run label options.  `toolbox` seems redundant since `oc debug node` was introduced.  `oc debug node/<node_name> already puts users inside the rhel support-tools container.  Users can use `oc debug node/<node_name> --image=docker.io/fedora:latest` if they want to use another image.

Comment 7 Derrick Ornelas 2020-07-13 19:06:35 UTC
Just a note that toolbox gives us a mechanism to capture data from the node when the cluster is down (ie, during installation, failing api-server, etc.)  It's also the tool we test our data collection tool, sosreport, against, and I don't believe 'oc debug node' sets up the container properly to run it.  

I think we're seeing users try to use different, untested images because it's not possible to get access to entitled RHEL content on an OpenShift cluster (in a sane way), so additional debug tools aren't available via yum.  

Should we move this over to the podman folks and see if they can pinpoint the problem with podman create/start?  Once that works toolbox could be updated to use '--pid=host' for images without a run label, correct?  We may not necessarily want to specifically enable this use-case on RHCOS, but the underlying issue sounds like a podman bug.

Comment 8 Michael Nguyen 2020-07-21 19:47:10 UTC
@derrick, lets move it to podman to see if they can pinpoint the issue and they can move it back if its the expected behavior.

Comment 9 Matthew Heon 2020-07-21 20:31:05 UTC
My initial impression is that this is probably a runc issue - it wants to grab a file from /proc of the init process in the toolbox container (I believe to set security attributes of the exec session to match the original container, but it's been a while since I was this deep in runc). It's failing to open the file, because of the `chroot` masking /proc from us (why it works with `--pid=host` I don't know; maybe the check in the `runc exec` is skipped in that case?). My advice here is going to be to add the `--pid=host` to Toolbox, given it makes runc happy and doesn't really change much in a security sense in the context of a toolbox container that already has access to the host. It may also be possible to mount over `/proc` in the chroot with a fresh procfs (I haven't actually run an `oc debug` container to confirm, but my suspicion is we're seeing the host's `/proc` coming in through the mount, which has greatly different PIDs from the ones we're expecting in the container. It's also possible that procfs isn't being propagated through the bind mount and `/proc/` in the chroot is just empty. Both should be able to be resolved with a procfs mount over it.)

Comment 10 Michael Nguyen 2020-07-21 20:59:45 UTC
Here is a reproducer:

# podman run -it --privileged --net=host --restart=no --pid=host -v /:/host registry.redhat.io/rhel7/support-tools /bin/sh
# chroot /host
# podman create \
                 --hostname toolbox \
                 --name coreos-toolbox \
                 --network host \
                 --privileged \
                 --security-opt label=disable \
                 --tty \
                 --volume /:/media/root:rslave \
                 docker.io/fedora:latest
# podman start coreos-toolbox
# podman exec  \
            --env LANG="$LANG" \
            --env TERM="$TERM" \
            --tty \
            --interactive \
            coreos-toolbox \
            /bin/sh

Comment 11 Michael Nguyen 2020-07-21 21:10:16 UTC
@mheon would that also explain this behavior?

# podman run -it --privileged --net=host --restart=no --pid=host -v /:/host registry.redhat.io/rhel7/support-tools /bin/sh
# chroot /host
# podman container runlabel --name mytoolbox RUN registry.redhat.io/rhel8/support-tools
I am now in a rhel8 support tools container, `cat /etc/os-release`
# exit
I am now back in the chroot
# podman start mytoolbox
# podman exec -it mytoolbox /bin/sh
I should be back in the rhel8 support tools container but I it seems like I am in the original rhel7 support tools container if i cat `/etc/os-release`

Comment 12 Matthew Heon 2020-07-21 21:26:09 UTC
The `--pid=host` in the outer container somewhat debunks my initial theory, given that PIDs and outside the container should match. It seems that even `/proc/self/` is functioning correctly inside /proc in the chroot, which genuinely surprises me. The low numbers of the PIDs it's looking at do make me quite suspicious, though.

Comment 13 Matthew Heon 2020-07-21 21:27:33 UTC
As for the issue you've given - by "the original rhel7 support tools container" - do you mean *outside* the chroot?

Comment 14 Michael Nguyen 2020-07-22 12:38:21 UTC
It seems like it is outside the chroot

Directly on RHCOS (not part of a cluster)
[root@ibm-p8-kvm-03-guest-02 ~]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="46.82.202007071437-0"
VERSION_ID="4.6"
OPENSHIFT_VERSION="4.6"
RHEL_VERSION="8.2"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202007071437-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.6"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.6"
OSTREE_VERSION='46.82.202007071437-0'

== Simulate `oc debug node` by starting the debug container - note it is running RHEL 7.8 ==
[root@ibm-p8-kvm-03-guest-02 ~]# podman run -it --privileged --net=host --restart=no --pid=host -v /:/host registry.redhat.io/rhel7/support-tools /bin/sh
Trying to pull registry.redhat.io/rhel7/support-tools...
Getting image source signatures
Copying blob 8e3c93d02cd2 done  
Copying blob 16b6dc064ae8 [======================================] 54.8MiB / 54.8MiB
Copying blob dc0665975713 done  
Copying config 03dc5763c1 done  
Writing manifest to image destination
Storing signatures
sh-4.2# cat /etc/os-release 
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.8 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"

== chroot /host so we are back into RHEL CoreOS ==
sh-4.2# chroot /host
sh-4.4# cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="46.82.202007071437-0"
VERSION_ID="4.6"
OPENSHIFT_VERSION="4.6"
RHEL_VERSION="8.2"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202007071437-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.6"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.6"
OSTREE_VERSION='46.82.202007071437-0'

== Run toolbox container - Note it is running RHEL 8.2 ==
sh-4.4# podman container runlabel --name mytoolbox RUN registry.redhat.io/rhel8/support-tools
Trying to pull registry.redhat.io/rhel8/support-tools...
Getting image source signatures
Copying blob 121d5409a427 done  
Copying blob a905c078265c done  
Copying blob ffbd2fd7eca5 done  
Copying config 8392dc0bfd done  
Writing manifest to image destination
Storing signatures
command: podman run -it --name mytoolbox --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=mytoolbox -e IMAGE=registry.redhat.io/rhel8/support-tools:latest -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest
[root@ibm-p8-kvm-03-guest-02 /]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="8.2 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.2"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.2 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.2:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.2"

== Exit and we are back in RHEL CoreOS ==
[root@ibm-p8-kvm-03-guest-02 /]# exit
exit
sh-4.4# cat /etc/os-release 
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="46.82.202007071437-0"
VERSION_ID="4.6"
OPENSHIFT_VERSION="4.6"
RHEL_VERSION="8.2"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 46.82.202007071437-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.6"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.6"
OSTREE_VERSION='46.82.202007071437-0'

== Start the toolbox container again and exec into it -- We should be in RHEL 8.2 but we are actually in RHEL 7.8 ==
sh-4.4# podman start mytoolbox
mytoolbox
sh-4.4# podman exec -it mytoolbox /bin/sh
sh-4.2# cat /etc/os-release 
NAME="Red Hat Enterprise Linux Server"
VERSION="7.8 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.8"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.8 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.8:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.8"
sh-4.2# podman
sh: podman: command not found

Comment 15 Jindrich Novy 2020-07-29 13:25:31 UTC
New bugfix release of toolbox (0.0.8) is out to address this bug.

Comment 20 Michael Nguyen 2020-07-30 16:57:13 UTC
Verification Steps:

1. oc debug node/<node_name>
2. chroot /host
3. cat << EOF > ~/.toolboxrc
IMAGE=fedora:latest
REGISTRY=docker.io
EOF
4. toolbox
5. verify no errors

Comment 21 Alex Jia 2020-08-11 08:56:55 UTC
1. Reroduced this bug on toolbox-0.0.4-1.module+el8.1.1+4407+ac444e5d

sh-4.4# rpm -q toolbox
toolbox-0.0.4-1.module+el8.1.1+4407+ac444e5d.x86_64
sh-4.4# toolbox
.toolboxrc file detected, overriding defaults...
Trying to pull docker.io/fedora:latest...
Getting image source signatures
Copying blob c7def56d621e done
Copying config a368cbcfa6 done
Writing manifest to image destination
Storing signatures
a368cbcfa6789bc347345f6d19132afe138b62ff5373d2aa5f37120277c90b54
Error: docker.io/fedora:latest does not have a label of RUN
Spawning a container 'toolbox-' with image 'docker.io/fedora:latest'
b34debf2b9e5bec2225f5b8879af484635c77a5289a667b22fd96a22c69a9cbb
toolbox-
Container started successfully. To exit, type 'exit'.
Error: exec failed: container_linux.go:346: starting container process caused "open /proc/self/task/19/attr/exec: no such file or directory": OCI runtime command not found error
sh-4.4#
Removing debug pod ...

2. Verified it on toolbox-0.0.8-1.module+el8.3.0+7627+c01ededd

sh-4.4# rpm -q toolbox
toolbox-0.0.8-1.module+el8.3.0+7627+c01ededd.noarch
sh-4.4# toolbox
.toolboxrc file detected, overriding defaults...
Error: docker.io/fedora:latest does not have a label of RUN
Spawning a container 'toolbox-' with image 'docker.io/fedora:latest'
[root@toolbox /]# ls
bin  boot  dev  etc  home  host  lib  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Comment 26 Steve Milner 2020-08-12 14:33:30 UTC
*** Bug 1868343 has been marked as a duplicate of this bug. ***

Comment 29 errata-xmlrpc 2020-11-04 03:05:17 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:4694


Note You need to log in before you can comment on or make changes to this bug.