RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2117928 - Error: runc: exec failed: unable to start container process: open /dev/pts/0: operation not permitted: OCI permission denied
Summary: Error: runc: exec failed: unable to start container process: open /dev/pts/0:...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: runc
Version: 8.6
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Alex Jia
URL:
Whiteboard:
: 2124671 2124699 (view as bug list)
Depends On:
Blocks: 2124700 2125648 2137345
TreeView+ depends on / blocked
 
Reported: 2022-08-12 17:49 UTC by Sameer
Modified: 2023-09-18 04:44 UTC (History)
15 users (show)

Fixed In Version: runc-1.1.3-3.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2124699 2124700 2137345 (view as bug list)
Environment:
Last Closed: 2022-11-08 09:16:44 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github opencontainers runc pull 3554 0 None Merged [1.1] Fix failed exec after systemctl daemon-reload (regression in 1.1.3) 2022-08-25 17:06:33 UTC
Github opencontainers runc pull 3559 0 None Merged Fix failed exec after systemctl daemon-reload 2022-08-24 12:15:59 UTC
Red Hat Issue Tracker RHELPLAN-131036 0 None None None 2022-08-12 17:51:14 UTC
Red Hat Knowledge Base (Solution) 6973095 0 None None None 2023-02-01 03:57:59 UTC
Red Hat Product Errata RHSA-2022:7457 0 None None None 2022-11-08 09:17:17 UTC

Comment 1 Alex Jia 2022-08-16 02:08:16 UTC 
I can reproduce this bug on runc-1.1.3-2.module+el8.6.0+15917+093ca6f8 w/ podman-4.1.1-2.module+el8.6.0+15917+093ca6f8.

[root@cloud-qe-05 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

[root@cloud-qe-05 ~]# rpm -q runc podman systemd kernel
runc-1.1.3-2.module+el8.6.0+15917+093ca6f8.x86_64
podman-4.1.1-2.module+el8.6.0+15917+093ca6f8.x86_64
systemd-239-58.el8_6.3.x86_64
kernel-4.18.0-372.23.1.el8_6.x86_64

[root@cloud-qe-05 ~]# podman -v
podman version 4.1.1

[root@cloud-qe-05 ~]# uname -a
Linux cloud-qe-05.idmqe.lab.eng.bos.redhat.com 4.18.0-372.23.1.el8_6.x86_64 #1 SMP Wed Aug 10 11:51:12 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

[root@cloud-qe-05 ~]# podman run -d --name exec-test  -p 8578:80 quay.io/redhattraining/httpd-parent
Trying to pull quay.io/redhattraining/httpd-parent:latest...
Getting image source signatures
Copying blob a3ed95caeb02 done  
Copying blob 6a5240d60dc4 done  
Copying blob 787f47dbeaac done  
Copying blob 08b8c9fdec44 done  
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 408208567b9a done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob a3ed95caeb02 skipped: already exists  
Writing manifest to image destination
Storing signatures
bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142

[root@cloud-qe-05 ~]# podman ps | grep exec-test
bb68e9a2e153  quay.io/redhattraining/httpd-parent:latest  /bin/sh -c /usr/s...  37 seconds ago  Up 37 seconds ago  0.0.0.0:8578->80/tcp  exec-test

[root@cloud-qe-05 ~]# systemctl daemon-reload
[root@cloud-qe-05 ~]# podman --log-level=debug exec -it exec-test sh
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exec.PersistentPreRunE(podman --log-level=debug exec -it exec-test sh) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend file              
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/runc"            
INFO[0000] Setting parallel job count to 13             
DEBU[0000] Handling terminal attach                     
INFO[0000] Created exec session 2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 in container bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142 
INFO[0000] Going to start container bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142 exec session 2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 and attach to it 
DEBU[0000] Sending resize events to exec session 2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 
DEBU[0000] Set user to root                             
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142 -u 2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142/userdata/2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 -p /var/lib/containers/storage/overlay-containers/bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142/userdata/2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205/exec_pid -n exec-test --exit-dir /var/lib/containers/storage/overlay-containers/bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142/userdata/2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205/exit --full-attach -s -l none --log-level debug --syslog -t -i -e --exec-attach --exec-process-spec /var/lib/containers/storage/overlay-containers/bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142/userdata/2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205/exec-process-2068177637 --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg  --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --exec --exit-command-arg 2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 --exit-command-arg bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142.scope 
DEBU[0000] Attaching to container bb68e9a2e153314b370c4d3fac00d1e874a5bb45c28d1e613fbe23ee62743142 exec session 2097f4112f8c6a98ad483a5eaca8715564ee3064cf1e5c456982a3c6799dc205 
DEBU[0000] Received: 0                                  
DEBU[0000] Received: -1                                 
Error: runc: [conmon:d]: exec with attach is waiting for start message from parent
[conmon:d]: exec with attach got start message from parent
time="2022-08-15T22:00:22-04:00" level=error msg="exec failed: unable to start container process: open /dev/pts/0: operation not permitted": OCI permission denied
Comment 2 Tom Sweeney 2022-08-16 17:57:09 UTC 
Giuseppe, could you take a look at this please?
Comment 3 Giuseppe Scrivano 2022-08-17 09:17:48 UTC 
it seems a regression caused by:

commit 343951a22b58c38feb044a5cea501dae92f8540e (HEAD, refs/bisect/bad)
Author: Aleksa Sarai <cyphar>
Date:   Thu Jun 2 12:07:00 2022 +1000

    cgroups: systemd: skip adding device paths that don't exist
    
    systemd emits very loud warnings when the path specified doesn't exist
    (which can be the case for some of our default rules). We don't need the
    ruleset we give systemd to be completely accurate (we discard some kinds
    of wildcard rules anyway) so we can safely skip adding these.
    
    Signed-off-by: Aleksa Sarai <cyphar>


There is a PR already opened upstream to address this issue: https://github.com/opencontainers/runc/pull/3559
Comment 4 Tom Sweeney 2022-08-18 21:22:30 UTC 
Assigning to Jindrich for any further BZ or packaging needs.
Comment 5 Jindrich Novy 2022-08-19 05:29:52 UTC 
Seeing the PR merged, we need to wait for a new runc release with this patch included.
Comment 16 Kir Kolyshkin 2022-08-26 00:43:22 UTC 
Reported upstream as https://github.com/opencontainers/runc/issues/3551

Caused by https://github.com/opencontainers/runc/pull/3498 (backported to 1.1 as https://github.com/opencontainers/runc/pull/3504)

Fixed by https://github.com/opencontainers/runc/pull/3559 (backported to 1.1 as https://github.com/opencontainers/runc/pull/3554)

Fixed in runc 1.1.4, released today.
Comment 17 Kir Kolyshkin 2022-08-26 00:44:29 UTC 
To clarify, this is a regression in runc 1.1.3 (so runc <= 1.1.2 is not affected), which is now fixed in runc 1.1.4.
Comment 25 Alex Jia 2022-08-29 12:58:08 UTC 
This bug has been verified on runc-1.1.4-1.module+el8.7.0+16493+89f82ab8.x86_64.

[root@hpe-dl380pgen8-02-vm-7 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 Beta (Ootpa)

[root@hpe-dl380pgen8-02-vm-7 ~]# rpm -q runc podman systemd kernel
runc-1.1.4-1.module+el8.7.0+16493+89f82ab8.x86_64
podman-4.2.0-1.module+el8.7.0+16493+89f82ab8.x86_64
systemd-239-65.el8.x86_64
kernel-4.18.0-422.el8.x86_64

[root@hpe-dl380pgen8-02-vm-7 ~]# podman -v
podman version 4.2.0

[root@hpe-dl380pgen8-02-vm-7 ~]# uname -a
Linux hpe-dl380pgen8-02-vm-7.hpe2.lab.eng.bos.redhat.com 4.18.0-422.el8.x86_64 #1 SMP Thu Aug 25 21:40:53 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

[root@hpe-dl380pgen8-02-vm-7 ~]# podman run -d --name exec-test  -p 8578:80 quay.io/redhattraining/httpd-parent
Trying to pull quay.io/redhattraining/httpd-parent:latest...
Getting image source signatures
Copying blob a3ed95caeb02 done  
Copying blob 6a5240d60dc4 done  
Copying blob 787f47dbeaac done  
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 done  
Copying blob 08b8c9fdec44 done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 408208567b9a done  
Copying blob a3ed95caeb02 skipped: already exists  
Writing manifest to image destination
Storing signatures
dac442f725a1e1d58448b9906aeb8e74a832881ab1d9dff0706bd105c3b6f956

[root@hpe-dl380pgen8-02-vm-7 ~]# podman ps | grep exec-test
dac442f725a1  quay.io/redhattraining/httpd-parent:latest  /bin/sh -c /usr/s...  4 seconds ago  Up 4 seconds ago  0.0.0.0:8578->80/tcp  exec-test

[root@hpe-dl380pgen8-02-vm-7 ~]# systemctl daemon-reload
[root@hpe-dl380pgen8-02-vm-7 ~]# podman exec -it exec-test sh
sh-4.4# ls
bin  boot  dev	etc  home  lib	lib64  lost+found  media  mnt  opt  proc  root	run  sbin  srv	sys  tmp  usr  var
sh-4.4# exit
exit
[root@hpe-dl380pgen8-02-vm-7 ~]# echo $?
0
Comment 31 Alex Jia 2022-09-01 13:39:12 UTC 
This bug also is verified for podman-4.0.2-5.module+el8.6.0+14672+b2f82327 w/ runc-1.1.3-3.module+el8.7.0+16483+b96ef47f on RHEL 8.6.

[root@kvm-04-guest11 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.6 (Ootpa)

[root@kvm-04-guest11 ~]# rpm -q podman runc systemd kernel
podman-4.0.2-5.module+el8.6.0+14672+b2f82327.x86_64
runc-1.1.3-3.module+el8.7.0+16483+b96ef47f.x86_64
systemd-239-58.el8_6.7.x86_64
kernel-4.18.0-372.25.1.el8_6.x86_64

[root@kvm-04-guest11 ~]# podman run -d --name exec-test -p 8578:80 quay.io/redhattraining/httpd-parent
Trying to pull quay.io/redhattraining/httpd-parent:latest...
Getting image source signatures
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 6a5240d60dc4 done  
Copying blob 787f47dbeaac done  
Copying blob 08b8c9fdec44 done  
Copying blob 408208567b9a done  
Writing manifest to image destination
Storing signatures
43ad94114bcc4a9bb6493f9eedaeb6f590783d151358213abce26d4ab2b84ee3
[root@kvm-04-guest11 ~]# podman ps | grep exec-test
43ad94114bcc  quay.io/redhattraining/httpd-parent:latest  /bin/sh -c /usr/s...  38 seconds ago  Up 38 seconds ago  0.0.0.0:8578->80/tcp  exec-test
[root@kvm-04-guest11 ~]# systemctl daemon-reload
[root@kvm-04-guest11 ~]# podman exec -it exec-test sh
sh-4.4# exit
exit
[root@kvm-04-guest11 ~]# echo $?
0
Comment 34 Tom Sweeney 2022-09-09 20:21:04 UTC 
*** Bug 2124671 has been marked as a duplicate of this bug. ***
Comment 42 Derrick Ornelas 2022-10-24 14:40:37 UTC 
*** Bug 2124699 has been marked as a duplicate of this bug. ***
Comment 43 Alex Jia 2022-10-25 03:55:35 UTC 
This bug has been verified for runc-1.1.3-3.module+el8.7.0+16483+b96ef47f and 
runc-1.1.4-1.module+el8.7.0+16772+33343656 on RHEL 8.7.0.

1. runc-1.1.3-3.module+el8.7.0+16483+b96ef47f

[root@koza-4 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 (Ootpa)

[root@koza-4 ~]# rpm -q podman runc systemd kernel
podman-4.2.0-1.module+el8.7.0+16483+b96ef47f.x86_64
runc-1.1.3-3.module+el8.7.0+16483+b96ef47f.x86_64
systemd-239-68.el8.x86_64
kernel-4.18.0-425.3.1.el8.x86_64

[root@koza-4 ~]# podman run -d --name exec-test -p 8578:80 quay.io/redhattraining/httpd-parent
Trying to pull quay.io/redhattraining/httpd-parent:latest...
Getting image source signatures
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 done  
Copying blob a3ed95caeb02 done  
Copying blob 787f47dbeaac done  
Copying blob 6a5240d60dc4 done  
Copying blob 08b8c9fdec44 done  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob a3ed95caeb02 skipped: already exists  
Copying blob 408208567b9a done  
Copying blob a3ed95caeb02 skipped: already exists  
Writing manifest to image destination
Storing signatures
6f077352b19ee1caadf2564383b95c9508a49365bb9d4339fe77d854e346c4ba

[root@koza-4 ~]# podman ps | grep exec-test
6f077352b19e  quay.io/redhattraining/httpd-parent:latest  /bin/sh -c /usr/s...  3 seconds ago  Up 3 seconds ago  0.0.0.0:8578->80/tcp  exec-test

[root@koza-4 ~]# systemctl daemon-reload
[root@koza-4 ~]# podman exec -it exec-test sh
sh-4.4# pwd
/
sh-4.4# exit
exit

2. runc-1.1.4-1.module+el8.7.0+16772+33343656

[root@koza-4 ~]# rpm -q podman runc systemd kernel
podman-4.2.0-1.module+el8.7.0+16772+33343656.x86_64
runc-1.1.4-1.module+el8.7.0+16772+33343656.x86_64
systemd-239-68.el8.x86_64
kernel-4.18.0-425.3.1.el8.x86_64

[root@koza-4 ~]# podman run -d --name exec-test -p 8578:80 quay.io/redhattraining/httpd-parent
6c27e70169a16510312727492005a7d13174ffb589adfe08ab11978b9241cec2

[root@koza-4 ~]# podman ps --filter name=exec-test
CONTAINER ID  IMAGE                                       COMMAND               CREATED         STATUS             PORTS                 NAMES
6c27e70169a1  quay.io/redhattraining/httpd-parent:latest  /bin/sh -c /usr/s...  29 seconds ago  Up 30 seconds ago  0.0.0.0:8578->80/tcp  exec-test

[root@koza-4 ~]# systemctl daemon-reload
[root@koza-4 ~]# podman exec -it exec-test sh
sh-4.4# whoami
root
sh-4.4# exit
exit
[root@koza-4 ~]# echo $?
0
Comment 46 errata-xmlrpc 2022-11-08 09:16:44 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7457
Comment 48 Red Hat Bugzilla 2023-09-18 04:44:37 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days
Note You need to log in before you can comment on or make changes to this bug.