Bug 1654325

Summary: Can't open any new terminals after running oc cluster up using moby-engine
Product: [Fedora] Fedora Reporter: Jason Montleon <jmontleo>
Component: moby-engineAssignee: Olivier Lemasle <o.lemasle>
Status: CLOSED NEXTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: damichae, fedora.dm0, o.lemasle
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-12 19:47:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jason Montleon 2018-11-28 14:29:22 UTC
Description of problem:
Run oc cluster up using the moby-engine

Version-Release number of selected component (if applicable):
rpm -q moby-engine
moby-engine-18.06.0-1.ce.git0ffa825.fc29.x86_64

$ ~/bin/oc version
oc v3.11.0+0cbc58b
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://172.17.0.1:8443
kubernetes v1.11.0+d4cacc0

How reproducible:
Seems always

Steps to Reproduce:
1. Install moby-engine and start the docker service
2. Bring up an openshift cluster using oc cluster up, something like, "oc cluster up --routing-suffix=172.17.0.1.nip.io --public-hostname=172.17.0.1 --base-dir=/tmp/openshift.local.clusterup --tag=v3.11 --image=docker.io/openshift/origin-\${component}:\${version} --enable=service-catalog,router,registry,web-console,persistent-volumes,sample-templates,rhel-imagestreams"
3. Try to open a new terminal, whether xterm, terminator, whatever. 

Actual results:
Terminal won't open. If you try to run xterm from an already open terminal you'll see

$ xterm
xterm: Error 32, errno 13: Permission denied
Reason: get_pty: not enough ptys

Expected results:
Running oc cluster up doesn't break stuff

Additional info:
This doesn't happen using  docker-2:1.13.1-62.git9cb56fd.fc29.x86_64

As long as I have a terminal open I can do chmod a+rw /dev/pts/ptmx to work around the problem.

Comment 1 Jason Montleon 2018-11-28 15:00:57 UTC
From: https://github.com/opencontainers/runc/issues/80

"When a container is launched with a bind mount between /dev (on the host) and /dev (in the container), all subsequent attempts to open /dev/ptmx fail.

setupPtmx() in libcontainer/rootfs_linux.go will unconditionally remove /dev/ptmx and then symlink it to /dev/pts/ptmx , which is a character file that is untouchable by unprivileged users. This prevents unprivileged programs which rely on getpt from running on the host (such as X terminals, screen, and tmux)."

It looks like this was originally fixed in https://github.com/opencontainers/runc/pull/96.

The again in August of this year with https://github.com/opencontainers/runc/issues/1866

Your runc in the package is from July so doesn't appear to have this change.

I patched the included runc and am running a build now. I'll report back when it's done whether it fixes the problem.

Comment 2 Jason Montleon 2018-11-28 15:14:11 UTC
Applying this patch appears to correct the problem and /dev/ptmx is no longer replaced with a link after running oc cluster up.

Comment 3 Jason Montleon 2018-12-03 15:11:49 UTC
This is still broken in moby-engine-18.06.0-1.ce.git0ffa825.fc29.x86_64.

Retrieving the patch and some minor modifications to the RPM spec should be sufficient to fix this if updating runc to a newer version isn't desired.

wget https://github.com/opencontainers/runc/pull/1867.patch -O revert_libcontainer_rootfs_linux_cleaup.patch

diff --git a/moby-engine.spec.orig b/moby-engine.spec
index 46aeb28..d55b7c7 100644
--- a/moby-engine.spec.orig
+++ b/moby-engine.spec
@@ -50,6 +50,7 @@ Source1: %{git_runc}/archive/%{commit_runc}.tar.gz
 Source2: %{git_containerd}/archive/%{commit_containerd}.tar.gz
 Source3: %{git_libnetwork}/archive/%{commit_libnetwork}.tar.gz
 Source4: %{git_tini}/archive/%{commit_tini}.tar.gz
+Patch0: revert_libcontainer_rootfs_linux_cleaup.patch
 URL: https://www.%{origname}.com
 
 BuildRequires: btrfs-progs-devel
@@ -135,7 +136,7 @@ Requires: nano
 This package installs %{summary}.
 
 %prep
-%autosetup -Sgit -n %{service_name}-ce-%{commit_moby}
+%setup -n %{service_name}-ce-%{commit_moby}
 
 # correct rpmlint errors for bash completion
 pushd components/cli
@@ -144,6 +145,9 @@ popd
 
 # untar runc
 tar zxf %{SOURCE1}
+pushd runc-%{commit_runc}
+%patch0 -p1
+popd
 
 # untar containerd
 tar zxf %{SOURCE2}

Comment 4 David Michael 2018-12-03 17:32:26 UTC
Can you try the 18.06.1 version in rawhide?  That version's runc snapshot looks like it is after the issue was fixed the first time, but before it was broken again.

https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Everything/x86_64/os/Packages/m/moby-engine-18.06.1-1.ce.gite68fc7a.fc30.x86_64.rpm

Comment 5 Jason Montleon 2018-12-03 17:52:19 UTC
Thanks, the Fedora 30 package fixes it!

Sorry if my last comment is confusing. I didn't look closely and it looks like when I did a dnf update this morning it reinstalled the same package I initially reported against. I thought I had picked up a new version.

Comment 6 Olivier Lemasle 2019-07-12 19:47:37 UTC
Closing as the requester wrote the issue is fixed in Fedora 30.