Bug 2021203
| Summary: | Symlinking /dev/ptmx to /dev/pts/ptmx prevents regular user from forking terminal | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | David Vallee Delisle <dvd> |
| Component: | crun | Assignee: | Giuseppe Scrivano <gscrivan> |
| Status: | CLOSED WONTFIX | QA Contact: | Alex Jia <ajia> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | CentOS Stream | CC: | bdobreli, bstinson, cjeanner, dtardon, jwboyer, systemd-maint-list, ykarel, ypu |
| Target Milestone: | rc | Keywords: | Reopened |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-05-17 07:27:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Possibly related: https://github.com/systemd/systemd/issues/20463 Another workaround is to simply: chmod 666 /dev/pts/ptmx After doing further investivation, Openshift product has seen a similar issue in Bug 1950408 that resulted in this article [a] and this doc change [b] Since this issues starts to happen only after the undercloud installation is completed, we can presume that some openstack containers are having bad mounts [1] In Bug 1950408 comment 18, it's recommended that, if we need to bind mount /dev, we should also add a mount type=devpts,destination=/dev/pts. [a] https://access.redhat.com/solutions/6205892 [b] https://github.com/openshift/openshift-docs/pull/34464/commits/b137fa8a5c9f7c715c9c8f2e8264008b048f1b59 [1] ~~~ [root@undercloud-0 ~]# podman ps -q | while read l;do echo "$l";podman inspect $l | jq -r '.[].Mounts[] | [.Source, .Destination] | @tsv' | grep -P "/dev[^\/]";done 229b0319bb29 560fa91d2da3 82bccf849215 fe0cdd41acb8 /dev /dev/ c7c11279eacb 7f842c077598 c22873137bfe 2ad7c661e2c0 8a703f6f40f9 0546bb29d745 88df372ab2af 8100522a45d4 /dev /dev d4725a57676b 9940f74ff548 63a170016303 6d978a2fc69a 0b7824b15fe9 [root@undercloud-0 ~]# podman ps | grep -P "fe0cdd41acb8|8100522a45d4" fe0cdd41acb8 undercloud-0.ctlplane.home.arpa:8787/tripleo_centos9/openstack-iscsid:latest kolla_start 2 days ago Up 2 days ago (healthy) iscsid 8100522a45d4 undercloud-0.ctlplane.home.arpa:8787/tripleo_centos9/openstack-ironic-conductor:latest kolla_start 2 days ago Up 2 days ago (unhealthy) ironic_conductor ~~~ (In reply to David Vallee Delisle from comment #0) > Description of problem: > > When forking a pty as regular user, we get a failure to read/write on > /dev/ptmx since it's just a symlink to /dev/pts/ptmx. This is only done in nspawn containers. The reason for it is explained in https://github.com/systemd/systemd/issues/20463#issuecomment-900916712 . > > In fedora, /dev/ptmx is its own node. Not if you run Fedora in a container. There's no difference between Fedora/RHEL/CentOS in this. > Workaround: > rm /dev/ptmx > mknod -m 666 /dev/ptmx c 5 2 This will put any created pty device into the host's devpts namespace. I think this issue should be opened against "crun" instead based on https://bugs.launchpad.net/tripleo/+bug/1950176/comments/19 the issue was reassigned to crun without any context. How does the problem happen? How the container was created? Is there any reproducer? If you'd like to have a fresh devpts mount in the container that overrides what is coming from the host, please use: podman run ... --mount type=devpts,target=/dev/pts .... Sorry for that. The context can be found here: https://bugs.launchpad.net/tripleo/+bug/1950176/comments/19 and looks related to the runc sibling https://github.com/opencontainers/runc/issues/80 Also, cjeanner tried to use the --mount dev=devpts,destination=/dev/pts and it failed for selinux reasons. That's mostly because we bind mount /dev into the container, and selinux rules/types on the host do not match the those in the container. He has more context for that, but currently is on PTO. thanks for the info. Do you have any pointer to the SELinux failures? We might need to fix something there, but I still think --mount dev=devpts,destination=/dev/pts is cleaner than expecting the OCI runtime to silently handle it. in the container crw-rw-rw-. 1 root root system_u:object_r:container_file_t:s0:c810,c818 5, 2 Nov 9 10:49 /dev/pts/ptmx crw-rw-rw-. 1 root tty system_u:object_r:ptmx_t:s0 5, 2 Nov 9 11:52 /dev/ptmx vs on the host c---------. 1 root root system_u:object_r:devpts_t:s0 5, 2 Nov 9 07:29 /dev/pts/ptmx and the reproducer should be (have no cs9 at hand to verify though), on the centos9/stream host do: sudo podman run --runtime=crun -it -v /dev:/dev --rm --privileged --name foo quay.io/centos/centos:stream9 python3 -c 'import os; _,_ = os.openpty()' forgot to post a failure example:
type=AVC msg=audit(11/09/2021 11:53:48.654:744) : avc: denied { getattr } for pid=16894 comm=ls path=/dev/ptmx dev="devtmpfs" ino=99 scontext=system_u:system_r:container_t:s0:c810,c818 tcontext=system_u:object_r:ptmx_t:s0 tclass=chr_file permissive=0
Hello there, I have a cs9 env, and here's what I tested/saw: - Bogdan's command doesn't affect the /dev/ptmx on the cs9 with crun - if we edit it and use `-v /dev/:/dev/', we can see the /dev/ptmx being replaced by the symlink This really looks like the discussions on runc back in 2015: - https://github.com/opencontainers/runc/pull/96 - https://github.com/opencontainers/runc/pull/742 I tried to find the code in crun responsible for the issue, but unfortunately, I'm not too fluent with that project. I suspect it's nested in there: https://github.com/containers/crun/blob/main/src/libcrun/linux.c but finding the exact line is a bit hard for me. I see there's a "force" flag for the /dev/ptmx symlink creation here: https://github.com/containers/crun/blob/main/src/libcrun/linux.c#L1270 maybe that's (part of) the issue we're facing? I'm currently running another OSP deploy with a patch replacing all the "/dev/:/dev/" by /dev:/dev" in order to ensure I'm not crazy (I did run the mentioned commands, but since I had rough nights for the past 2 weeks....) Stay tuned, I'll come back as soon as I get something more precise from the OSP side. Cheers, C. I am not sure the issue is caused by crun. I've not managed to reproduce it.
When `/dev` is mounted from the host, crun skips any modification there.
If you look at the code you've linked, `create_missing_devs` is called only when `/dev` is not mounted from the host:
if (! get_private_data (container)->mount_dev_from_host)
{
ret = create_missing_devs (container, is_user_ns ? true : false, err);
if (UNLIKELY (ret < 0))
return ret;
}
Could you please share a reproducer? Could it be the container payload to create that symlink?
> When `/dev` is mounted from the host, crun skips any modification there
I think as Cedric commented above, the issue is that
When `/dev/` is mounted from the host, crun *does not* skip modifications there
ok thanks, I see the issue now. I'll open a PR upstream to fix it. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |
Description of problem: When forking a pty as regular user, we get a failure to read/write on /dev/ptmx since it's just a symlink to /dev/pts/ptmx. In fedora, /dev/ptmx is its own node. [1] centos 9 [2] fedora [3] strace [4] reproducer script Version-Release number of selected component (if applicable): 5.14.0-7.el9.x86_64 How reproducible: All the time Steps to Reproduce: 1. Run script [4] Workaround: rm /dev/ptmx mknod -m 666 /dev/ptmx c 5 2 Additional info: [1] ~~~ (undercloud) [stack@undercloud-0 ~]$ ls -tlra /dev/ptmx lrwxrwxrwx. 1 root root 8 Nov 7 12:34 /dev/ptmx -> pts/ptmx ~~~ [2] ~~~ [dvd@fedora ~]$ ls -tlra /dev/ptmx crw-rw-rw-. 1 root tty 5, 2 Nov 8 09:39 /dev/ptmx [dvd@fedora ~]$ ls -tlra /dev/pts/ptmx c---------. 1 root root 5, 2 Oct 25 16:50 /dev/pts/ptmx ~~~ [3] ~~~ 14:48:20.542970 openat(AT_FDCWD, "/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied) 14:48:20.543061 write(2, "Traceback (most recent call last):\n", 35Traceback (most recent call last): ) = 35 14:48:20.543116 write(2, " File \"/home/stack/./test.py\", line 10, in <module>\n", 53 File "/home/stack/./test.py", line 10, in <module> ) = 53 14:48:20.543168 openat(AT_FDCWD, "/home/stack/./test.py", O_RDONLY|O_CLOEXEC) = 3 14:48:20.543216 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=437, ...}, AT_EMPTY_PATH) = 0 14:48:20.543266 ioctl(3, TCGETS, 0x7ffe22bd1f30) = -1 ENOTTY (Inappropriate ioctl for device) 14:48:20.543310 lseek(3, 0, SEEK_CUR) = 0 14:48:20.543361 fcntl(3, F_DUPFD_CLOEXEC, 0) = 4 14:48:20.543403 fcntl(4, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE) 14:48:20.543445 newfstatat(4, "", {st_mode=S_IFREG|0755, st_size=437, ...}, AT_EMPTY_PATH) = 0 14:48:20.543493 read(4, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n \n# importing os module \nimport os\n \n \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n \n \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(master)\nprint(name)\n \n \n# Get the terminal device\n# name associated with\n# file descriptor slave\nname = os.ttyname(slave)\nprint(name)\n", 4096) = 437 14:48:20.543542 close(4) = 0 14:48:20.543584 lseek(3, 0, SEEK_SET) = 0 14:48:20.543636 read(3, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n \n# importing os module \nimport os\n \n \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n \n \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(master)\nprint(name)\n \n \n# Get the terminal device\n# name associated with\n# file descriptor slave\nname = os.ttyname(slave)\nprint(name)\n", 8192) = 437 14:48:20.543689 close(3) = 0 14:48:20.543735 write(2, " master, slave = os.openpty()\n", 33 master, slave = os.openpty() ) = 33 14:48:20.543794 write(2, "PermissionError: [Errno 13] Permission denied\n", 46PermissionError: [Errno 13] Permission denied ) = 46 ~~~ [4] ~~~ #!/usr/bin/env python import os master, slave = os.openpty() ~~~