Bug 2033365
| Summary: | Toolbox created container incorrectly mounts /proc/sys/fs/binfmt_misc | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Jake Hunsaker <jhunsaker> |
| Component: | podman | Assignee: | Tom Sweeney <tsweeney> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | atomic-bugs <atomic-bugs> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 8.4 | CC: | aos-bugs, bbaude, dornelas, dwalsh, jligon, jnovy, lsm5, mdekan, mheon, mnguyen, mrussell, npinaeva, nstielau, pthomas, travier, tsweeney, umohnani, vrothber |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-05-08 19:42:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2042069 | ||
| Bug Blocks: | |||
In RHEL 8.5, the implementation of the toolbox RPM was changed to the one from github.com/containers/toolbox (see bug 1914687), and I can't reproduce this bug with the new implementation. I am reassigning the bug since RHCOS still uses the old implementation, and that's what you seem to be using here. Regardless, I'd still be curious to know if you hit this problem with the new implementation from github.com/containers/toolbox I tried running this without toolbox just using the options in run label for the rhel8/support-tools container and it does the same thing. If I remove the container and run the same command again it doesn't the problem.
[core@localhost ~]$ sudo podman pull registry.redhat.io/rhel8/support-tools
Trying to pull registry.redhat.io/rhel8/support-tools:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 29bfe07dbe66 done
Copying blob 26f1167feaf7 done
Copying blob adffa6963146 done
Copying config e0a3910607 done
Writing manifest to image destination
Storing signatures
e0a3910607766bb10a9d84a0223db78d9856407e34af50a4f8fc0587c6c041a9
[core@localhost ~]$ sudo podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.redhat.io/rhel8/support-tools latest e0a391060776 5 weeks ago 296 MB
[core@localhost ~]$ sudo podman image inspect registry.redhat.io/rhel8/support-tools --format "{{.Labels.run}}"
podman run -it --name NAME --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=NAME -e IMAGE=IMAGE -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host IMAGE
[core@localhost ~]$ sudo podman run -it --name TEST_TOOLBOX --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=TEST_TOOLBOX -e IMAGE=registry.redhat.io/rhel8/support-tools -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest
[root@localhost /]# ls -l /host/proc/sys/fs
ls: cannot access '/host/proc/sys/fs/binfmt_misc': Too many levels of symbolic links
total 0
-rw-r--r--. 1 root root 0 Jan 20 19:14 aio-max-nr
-r--r--r--. 1 root root 0 Jan 20 19:14 aio-nr
d?????????? ? ? ? ? ? binfmt_misc
-r--r--r--. 1 root root 0 Jan 20 19:14 dentry-state
-rw-r--r--. 1 root root 0 Jan 20 19:14 dir-notify-enable
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 epoll
-rw-r--r--. 1 root root 0 Jan 20 19:14 file-max
-r--r--r--. 1 root root 0 Jan 20 19:14 file-nr
-r--r--r--. 1 root root 0 Jan 20 19:14 inode-nr
-r--r--r--. 1 root root 0 Jan 20 19:14 inode-state
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 inotify
-rw-r--r--. 1 root root 0 Jan 20 19:14 lease-break-time
-rw-r--r--. 1 root root 0 Jan 20 19:14 leases-enable
-rw-r--r--. 1 root root 0 Jan 20 19:14 mount-max
dr-xr-xr-x. 1 root root 0 Jan 20 19:10 mqueue
-rw-r--r--. 1 root root 0 Jan 20 19:10 nr_open
-rw-r--r--. 1 root root 0 Jan 20 19:14 overflowgid
-rw-r--r--. 1 root root 0 Jan 20 19:14 overflowuid
-rw-r--r--. 1 root root 0 Jan 20 19:14 pipe-max-size
-rw-r--r--. 1 root root 0 Jan 20 19:14 pipe-user-pages-hard
-rw-r--r--. 1 root root 0 Jan 20 19:14 pipe-user-pages-soft
-rw-------. 1 root root 0 Jan 20 19:14 protected_fifos
-rw-------. 1 root root 0 Jan 20 19:10 protected_hardlinks
-rw-------. 1 root root 0 Jan 20 19:14 protected_regular
-rw-------. 1 root root 0 Jan 20 19:10 protected_symlinks
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 quota
-rw-r--r--. 1 root root 0 Jan 20 19:14 suid_dumpable
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 xfs
[root@localhost /]# exit
exit
[core@localhost ~]$ sudo podman ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
21c6cd9d8b90 registry.redhat.io/rhel8/support-tools:latest /usr/bin/bash About a minute ago Exited (1) 11 seconds ago TEST_TOOLBOX
[core@localhost ~]$ sudo podman rm 21c6cd9d8b90
21c6cd9d8b909d5d0550e429bfeabaaaf2abb214395010e4d4e4106bad9426d5
[core@localhost ~]$ sudo podman run -it --name TEST_TOOLBOX --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=TEST_TOOLBOX -e IMAGE=registry.redhat.io/rhel8/support-tools -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest
[root@localhost /]# ls -l /host/proc/sys/fs
total 0
-rw-r--r--. 1 root root 0 Jan 20 19:14 aio-max-nr
-r--r--r--. 1 root root 0 Jan 20 19:14 aio-nr
drwxr-xr-x. 2 root root 0 Jan 20 19:14 binfmt_misc
-r--r--r--. 1 root root 0 Jan 20 19:14 dentry-state
-rw-r--r--. 1 root root 0 Jan 20 19:14 dir-notify-enable
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 epoll
-rw-r--r--. 1 root root 0 Jan 20 19:14 file-max
-r--r--r--. 1 root root 0 Jan 20 19:14 file-nr
-r--r--r--. 1 root root 0 Jan 20 19:14 inode-nr
-r--r--r--. 1 root root 0 Jan 20 19:14 inode-state
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 inotify
-rw-r--r--. 1 root root 0 Jan 20 19:14 lease-break-time
-rw-r--r--. 1 root root 0 Jan 20 19:14 leases-enable
-rw-r--r--. 1 root root 0 Jan 20 19:14 mount-max
dr-xr-xr-x. 1 root root 0 Jan 20 19:10 mqueue
-rw-r--r--. 1 root root 0 Jan 20 19:10 nr_open
-rw-r--r--. 1 root root 0 Jan 20 19:14 overflowgid
-rw-r--r--. 1 root root 0 Jan 20 19:14 overflowuid
-rw-r--r--. 1 root root 0 Jan 20 19:14 pipe-max-size
-rw-r--r--. 1 root root 0 Jan 20 19:14 pipe-user-pages-hard
-rw-r--r--. 1 root root 0 Jan 20 19:14 pipe-user-pages-soft
-rw-------. 1 root root 0 Jan 20 19:14 protected_fifos
-rw-------. 1 root root 0 Jan 20 19:10 protected_hardlinks
-rw-------. 1 root root 0 Jan 20 19:14 protected_regular
-rw-------. 1 root root 0 Jan 20 19:10 protected_symlinks
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 quota
-rw-r--r--. 1 root root 0 Jan 20 19:14 suid_dumpable
dr-xr-xr-x. 1 root root 0 Jan 20 19:14 xfs
More info:
podman version 3.2.3
I was also able to reproduce on Fedora 35 Silverblue with podman version 3.4.4
Reproduction Steps:
1. # podman pull registry.redhat.io/rhel8/support-tools
2. # podman image inspect registry.redhat.io/rhel8/support-tools --format "{{.Labels.run}}"
3. Take the output of the step 2 and replace "IMAGE" with `registry.redhat.io/rhel8/support-tools`
# podman run -it --name NAME --privileged --ipc=host --net=host --pid=host -e HOST=/host -e NAME=NAME -e IMAGE=registry.redhat.io/rhel8/support-tools -v /run:/run -v /var/log:/var/log -v /etc/machine-id:/etc/machine-id -v /etc/localtime:/etc/localtime -v /:/host registry.redhat.io/rhel8/support-tools:latest
4. Once the container starts, run `ls -l /host/proc/sys/fs` inside the container and verify `ls: cannot access '/host/proc/sys/fs/binfmt_misc': Too many levels of symbolic links` error
Note this only happens the very first time. I did not see the error in subsequent runs.
Re-assigning to the containers team so they can take a look since I can reproduce this on Fedora also and it doesn't seem to be specific to toolbox. Valentin, could you take a quick peak at this please? @vrothber did you ever peak at this one? (In reply to Tom Sweeney from comment #6) > @vrothber did you ever peak at this one? No, please assign the bug to me or put me in "needinfo". I added it to my todo list and will take a look asap. I cannot reproduce the issue anymore. @Tom, I think we can close it given the age and that it doesn't occur anymore. Based on the age and not being able to reproduce this anymore, I'm going to close this BZ. If the problem still exists, please open a new BZ. |
Description of problem: Launching a container with toolbox will mount /proc/sys/fs/binfmt_misc in such a way that any operation on it results in an error of "Too many levels of symbolic links" ------8<----------- [root@worker-0 ~]# toolbox Trying to pull registry.redhat.io/rhel8/support-tools:latest... Getting image source signatures Checking if image destination supports signatures Copying blob 63f9f4c31162 done Copying blob ce3c6836540f done Copying blob b4a401c484bd done Copying config a1ce51e86e done Writing manifest to image destination Storing signatures a1ce51e86e78e6af4dd3d91f6ee563a69116faa88c6dce19ecfc39f8ab2d1e4f Spawning a container 'toolbox-root' with image 'registry.redhat.io/rhel8/support-tools' Detected RUN label in the container image. Using that as the default... [root@worker-0 /]# ls -l /host/proc/sys/fs ls: cannot access '/host/proc/sys/fs/binfmt_misc': Too many levels of symbolic links total 0 -rw-r--r--. 1 root root 0 Dec 16 15:25 aio-max-nr -r--r--r--. 1 root root 0 Dec 16 15:25 aio-nr d?????????? ? ? ? ? ? binfmt_misc ------8<----------- However, directly launching the same container image using podman shows the correct behavior: ------8<----------- [root@worker-0 ~]# podman container runlabel RUN registry.redhat.io/rhel8/support-tools [root@worker-0 /]# ls -l /host/proc/sys/fs total 0 -rw-r--r--. 1 root root 0 Dec 16 15:26 aio-max-nr -r--r--r--. 1 root root 0 Dec 16 15:26 aio-nr drwxr-xr-x. 2 root root 0 Dec 16 15:25 binfmt_misc ------8<----------- Version-Release number of selected component (if applicable): toolbox-0.0.8-3.rhaos4.9.el8.noarch # rpm-ostree status State: idle Deployments: * pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:480e39d63063bae8992542905d48442fd1d9d1325a5136a3be8256d123efe490 CustomOrigin: Managed by machine-config-operator Version: 49.84.202110220538-0 (2021-10-22T05:41:35Z) ostree://3093d4596a48e37c9926dc53240af084c077e3bf2063ef2a8d8a81421b6e9987 Version: 49.84.202110081407-0 (2021-10-08T14:10:58Z) How reproducible: This seems like it may be intermittent in some capacity. On most systems I can reproduce this easily by simply connecting to the RHCOS node, switching to root, and running `toolbox` - however, there are times where it gets mounted correctly though I cannot determine what the cause for this is. Steps to Reproduce: 1. Connect to an RHCOS node 2. Switch to root 3. Run `toolbox` and try `ls -l /host/proc/sys/fs` or anything else that would stat /host/proc/sys/fs/binfmt_misc Actual results: /proc/sys/fs/binfmt_misc from the host is not mounted correctly Expected results: binfmt_misc should be mounted correctly and stat'able etc... Additional info: The above nodes are deployed as a cluster in quicklab and I can provide access if needed. I wasn't sure if this should go to the podman team or not, as I thought toolbox just leveraged the RUN label but there is apparently something else going on as manually using that label shows the expected behavior.