2021203 – Symlinking /dev/ptmx to /dev/pts/ptmx prevents regular user from forking terminal

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 2021203 - Symlinking /dev/ptmx to /dev/pts/ptmx prevents regular user from forking terminal

Summary: Symlinking /dev/ptmx to /dev/pts/ptmx prevents regular user from forking term...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 9
Classification:	Red Hat
Component:	crun
Sub Component:
Version:	CentOS Stream
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Giuseppe Scrivano
QA Contact:	Alex Jia
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-11-08 14:52 UTC by David Vallee Delisle
Modified:	2022-05-17 07:27 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2022-05-17 07:27:31 UTC
Type:	Bug
Target Upstream Version:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	systemd systemd issues 20463	0	None	open	devpts improvements	2021-11-09 15:28:06 UTC
Red Hat Issue Tracker	RHELPLAN-101990	0	None	None	None	2021-11-08 14:56:26 UTC

Description David Vallee Delisle 2021-11-08 14:52:21 UTC

Description of problem:

When forking a pty as regular user, we get a failure to read/write on /dev/ptmx since it's just a symlink to /dev/pts/ptmx.

In fedora, /dev/ptmx is its own node.

[1] centos 9
[2] fedora
[3] strace
[4] reproducer script

Version-Release number of selected component (if applicable):
5.14.0-7.el9.x86_64

How reproducible:
All the time

Steps to Reproduce:
1. Run script [4]

Workaround:
rm /dev/ptmx
mknod -m 666 /dev/ptmx c 5 2


Additional info:
[1]
~~~
(undercloud) [stack@undercloud-0 ~]$ ls -tlra /dev/ptmx                                                                                                                                                                                                                                                                                                                                                                                     
lrwxrwxrwx. 1 root root 8 Nov  7 12:34 /dev/ptmx -> pts/ptmx
~~~

[2]
~~~
[dvd@fedora ~]$ ls -tlra /dev/ptmx 
crw-rw-rw-. 1 root tty 5, 2 Nov  8 09:39 /dev/ptmx
[dvd@fedora ~]$ ls -tlra /dev/pts/ptmx
c---------. 1 root root 5, 2 Oct 25 16:50 /dev/pts/ptmx
~~~

[3]
~~~
14:48:20.542970 openat(AT_FDCWD, "/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied)
14:48:20.543061 write(2, "Traceback (most recent call last):\n", 35Traceback (most recent call last):
) = 35
14:48:20.543116 write(2, "  File \"/home/stack/./test.py\", line 10, in <module>\n", 53  File "/home/stack/./test.py", line 10, in <module>
) = 53
14:48:20.543168 openat(AT_FDCWD, "/home/stack/./test.py", O_RDONLY|O_CLOEXEC) = 3
14:48:20.543216 newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=437, ...}, AT_EMPTY_PATH) = 0
14:48:20.543266 ioctl(3, TCGETS, 0x7ffe22bd1f30) = -1 ENOTTY (Inappropriate ioctl for device)
14:48:20.543310 lseek(3, 0, SEEK_CUR)   = 0
14:48:20.543361 fcntl(3, F_DUPFD_CLOEXEC, 0) = 4
14:48:20.543403 fcntl(4, F_GETFL)       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
14:48:20.543445 newfstatat(4, "", {st_mode=S_IFREG|0755, st_size=437, ...}, AT_EMPTY_PATH) = 0
14:48:20.543493 read(4, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n  \n# importing os module \nimport os\n  \n  \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n  \n  \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(master)\nprint(name)\n  \n  \n# Get the terminal device\n# name associated with\n# file descriptor slave\nname = os.ttyname(slave)\nprint(name)\n", 4096) = 437
14:48:20.543542 close(4)                = 0
14:48:20.543584 lseek(3, 0, SEEK_SET)   = 0
14:48:20.543636 read(3, "#!/usr/bin/env python\n# Python program to explain os.openpty() method \n  \n# importing os module \nimport os\n  \n  \n# open new pseudo-terminal pair\n# using os.openpty() method\nmaster, slave = os.openpty()\n  \n  \n# Get the terminal device\n# name associated with\n# file descriptor master \nname = os.ttyname(master)\nprint(name)\n  \n  \n# Get the terminal device\n# name associated with\n# file descriptor slave\nname = os.ttyname(slave)\nprint(name)\n", 8192) = 437
14:48:20.543689 close(3)                = 0
14:48:20.543735 write(2, "    master, slave = os.openpty()\n", 33    master, slave = os.openpty()
) = 33
14:48:20.543794 write(2, "PermissionError: [Errno 13] Permission denied\n", 46PermissionError: [Errno 13] Permission denied
) = 46
~~~

[4]
~~~
#!/usr/bin/env python
import os
master, slave = os.openpty()
~~~

Comment 1 David Vallee Delisle 2021-11-08 14:59:18 UTC

Possibly related:
https://github.com/systemd/systemd/issues/20463

Comment 2 David Vallee Delisle 2021-11-08 15:06:28 UTC

Another workaround is to simply:
chmod 666 /dev/pts/ptmx

Comment 3 David Vallee Delisle 2021-11-08 15:20:51 UTC

After doing further investivation, Openshift product has seen a similar issue in Bug 1950408 that resulted in this article [a] and this doc change [b]

Since this issues starts to happen only after the undercloud installation is completed, we can presume that some openstack containers are having bad mounts [1]

In Bug 1950408 comment 18, it's recommended that, if we need to bind mount /dev, we should also add a mount type=devpts,destination=/dev/pts.

[a] https://access.redhat.com/solutions/6205892
[b] https://github.com/openshift/openshift-docs/pull/34464/commits/b137fa8a5c9f7c715c9c8f2e8264008b048f1b59

[1]
~~~
[root@undercloud-0 ~]# podman ps -q | while read l;do echo "$l";podman inspect $l | jq -r '.[].Mounts[] | [.Source, .Destination] | @tsv' | grep -P "/dev[^\/]";done
229b0319bb29
560fa91d2da3
82bccf849215
fe0cdd41acb8
/dev    /dev/
c7c11279eacb
7f842c077598
c22873137bfe
2ad7c661e2c0
8a703f6f40f9
0546bb29d745
88df372ab2af
8100522a45d4
/dev    /dev
d4725a57676b
9940f74ff548
63a170016303
6d978a2fc69a
0b7824b15fe9
[root@undercloud-0 ~]# podman ps | grep -P "fe0cdd41acb8|8100522a45d4"
fe0cdd41acb8  undercloud-0.ctlplane.home.arpa:8787/tripleo_centos9/openstack-iscsid:latest                     kolla_start           2 days ago  Up 2 days ago (healthy)                iscsid
8100522a45d4  undercloud-0.ctlplane.home.arpa:8787/tripleo_centos9/openstack-ironic-conductor:latest           kolla_start           2 days ago  Up 2 days ago (unhealthy)              ironic_conductor
~~~

Comment 4 David Tardon 2021-11-09 15:28:07 UTC

(In reply to David Vallee Delisle from comment #0)
> Description of problem:
> 
> When forking a pty as regular user, we get a failure to read/write on
> /dev/ptmx since it's just a symlink to /dev/pts/ptmx.

This is only done in nspawn containers. The reason for it is explained in https://github.com/systemd/systemd/issues/20463#issuecomment-900916712 .

> 
> In fedora, /dev/ptmx is its own node.

Not if you run Fedora in a container. There's no difference between Fedora/RHEL/CentOS in this.

> Workaround:
> rm /dev/ptmx
> mknod -m 666 /dev/ptmx c 5 2

This will put any created pty device into the host's devpts namespace.

Comment 5 Yatin Karel 2021-11-16 16:10:57 UTC

I think this issue should be opened against "crun" instead based on https://bugs.launchpad.net/tripleo/+bug/1950176/comments/19

Comment 7 Giuseppe Scrivano 2021-11-17 14:44:59 UTC

the issue was reassigned to crun without any context.

How does the problem happen?  How the container was created?  Is there any reproducer?


If you'd like to have a fresh devpts mount in the container that overrides what is coming from the host, please use:

podman run ... --mount type=devpts,target=/dev/pts ....

Comment 8 Bogdan Dobrelya 2021-11-17 14:51:31 UTC

Sorry for that. The context can be found here: https://bugs.launchpad.net/tripleo/+bug/1950176/comments/19 and looks related to the runc sibling https://github.com/opencontainers/runc/issues/80


Also, cjeanner tried to use the --mount dev=devpts,destination=/dev/pts and it failed for selinux reasons. That's mostly because we bind mount /dev into the container, and selinux rules/types on the host do not match the those in the container. He has more context for that, but currently is on PTO.

Comment 9 Giuseppe Scrivano 2021-11-17 15:25:23 UTC

thanks for the info.

Do you have any pointer to the SELinux failures?  We might need to fix something there, but I still think --mount dev=devpts,destination=/dev/pts is cleaner than expecting the OCI runtime to silently handle it.

Comment 10 Bogdan Dobrelya 2021-11-17 15:53:29 UTC

in the container
crw-rw-rw-. 1 root root system_u:object_r:container_file_t:s0:c810,c818 5, 2 Nov  9 10:49 /dev/pts/ptmx  
crw-rw-rw-. 1 root tty system_u:object_r:ptmx_t:s0 5, 2 Nov  9 11:52 /dev/ptmx
vs on the host
c---------. 1 root root system_u:object_r:devpts_t:s0 5, 2 Nov  9 07:29 /dev/pts/ptmx

and the reproducer should be (have no cs9 at hand to verify though), on the centos9/stream host do:

sudo podman run --runtime=crun -it -v /dev:/dev --rm --privileged  --name foo quay.io/centos/centos:stream9 python3 -c 'import os; _,_ = os.openpty()'

Comment 11 Bogdan Dobrelya 2021-11-17 15:57:16 UTC

forgot to post a failure example:

type=AVC msg=audit(11/09/2021 11:53:48.654:744) : avc:  denied  { getattr } for  pid=16894 comm=ls path=/dev/ptmx dev="devtmpfs" ino=99 scontext=system_u:system_r:container_t:s0:c810,c818 tcontext=system_u:object_r:ptmx_t:s0 tclass=chr_file permissive=0

Comment 12 Cédric Jeanneret 2021-11-29 14:53:56 UTC

Hello there,

I have a cs9 env, and here's what I tested/saw:

- Bogdan's command doesn't affect the /dev/ptmx on the cs9 with crun
- if we edit it and use `-v /dev/:/dev/', we can see the /dev/ptmx being replaced by the symlink

This really looks like the discussions on runc back in 2015:
- https://github.com/opencontainers/runc/pull/96
- https://github.com/opencontainers/runc/pull/742

I tried to find the code in crun responsible for the issue, but unfortunately, I'm not too fluent with that project. I suspect it's nested in there:
https://github.com/containers/crun/blob/main/src/libcrun/linux.c
but finding the exact line is a bit hard for me.

I see there's a "force" flag for the /dev/ptmx symlink creation here:
https://github.com/containers/crun/blob/main/src/libcrun/linux.c#L1270

maybe that's (part of) the issue we're facing?

I'm currently running another OSP deploy with a patch replacing all the "/dev/:/dev/" by /dev:/dev" in order to ensure I'm not crazy (I did run the mentioned commands, but since I had rough nights for the past 2 weeks....)

Stay tuned, I'll come back as soon as I get something more precise from the OSP side.

Cheers,

C.

Comment 13 Giuseppe Scrivano 2021-12-03 16:44:51 UTC

I am not sure the issue is caused by crun.  I've not managed to reproduce it.

When `/dev` is mounted from the host, crun skips any modification there.

If you look at the code you've linked, `create_missing_devs` is called only when `/dev` is not mounted from the host:

  if (! get_private_data (container)->mount_dev_from_host)
    {
      ret = create_missing_devs (container, is_user_ns ? true : false, err);
      if (UNLIKELY (ret < 0))
        return ret;
    }

Could you please share a reproducer?  Could it be the container payload to create that symlink?

Comment 14 Bogdan Dobrelya 2021-12-03 16:48:46 UTC

> When `/dev` is mounted from the host, crun skips any modification there
I think as Cedric commented above, the issue is that

When `/dev/` is mounted from the host, crun *does not* skip modifications there

Comment 15 Giuseppe Scrivano 2021-12-03 16:57:02 UTC

ok thanks, I see the issue now.

I'll open a PR upstream to fix it.

Comment 16 Giuseppe Scrivano 2021-12-03 17:18:05 UTC

PR here: https://github.com/containers/crun/pull/799

Comment 18 RHEL Program Management 2022-05-17 07:27:31 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Note You need to log in before you can comment on or make changes to this bug.