Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1053485

Summary: Changing GID_MIN in login.defs prevents app creation
Product: OpenShift Container Platform Reporter: Josep 'Pep' Turro Mauri <pep>
Component: ContainersAssignee: Brenton Leanhardt <bleanhar>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.0.0CC: cpelland, gpei, libra-onpremise-devel
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-05-15 14:41:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
possible patch none

Description Josep 'Pep' Turro Mauri 2014-01-15 10:29:42 UTC
Description of problem:
If login.defs in the node is modified and GID_MIN is set to a high value app creation fails.

Version-Release number of selected component (if applicable):
rubygem-openshift-origin-node-1.17.5-2.el6op.noarch
rubygem-openshift-origin-container-selinux-0.4.1-2.el6op.noarch

How reproducible:
Always

Steps to Reproduce:
1. edit /etc/login.defs in the node, set GID_MIN to 1001
2. try 'rhc app create ...' from a client

Actual results: (output from 'rhc app create test php'

Creating application 'test' ... 
Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server mynode01.example.com.  This indicates an unexpected problem during the execution of your request..
Reference ID: 340d74edb25352bde139b1054f6165f8

Expected results:
App creation works.

Additional info:

Initially the home dir of the gear seems to be created with the correct gid:
# ls -laZ /var/lib/openshift/5*
drwx------. 52d6614585937af84a000058 1001 system_u:object_r:openshift_var_lib_t:s0 .
drwxr-xr-x. root                     root system_u:object_r:openshift_var_lib_t:s0 ..

but then it's chown'd:

# ls -laZ /var/lib/openshift/5*
drwxr-x---. root 1000 system_u:object_r:openshift_var_lib_t:s0:c0,c1000 .
drwxr-xr-x. root root system_u:object_r:openshift_var_lib_t:s0 ..

# ls -laZ /var/lib/openshift/5*
drwxr-x---. root                     1000 system_u:object_r:openshift_var_lib_t:s0:c0,c1000 .
drwxr-xr-x. root                     root system_u:object_r:openshift_var_lib_t:s0 ..
drwxr-xr-x. root                     root system_u:object_r:openshift_var_lib_t:s0 app-root
drwxr-x---. root                     1000 system_u:object_r:openshift_var_lib_t:s0:c0,c1000 .env
drwxr-x---. 52d6614585937af84a000058 1000 system_u:object_r:openshift_var_lib_t:s0:c0,c1000 .gem
d---------. root                     root system_u:object_r:openshift_var_lib_t:s0 .sandbox
drwxr-x---. root                     1000 system_u:object_r:ssh_home_t:s0:c0,c1000 .ssh
d---------. root                     root system_u:object_r:openshift_var_lib_t:s0 .tmp

which leads to lack of permissions for ssh-keygen, resulting in the failure.

Comment 1 Josep 'Pep' Turro Mauri 2014-03-06 14:35:30 UTC
Created attachment 871493 [details]
possible patch

AFAICT the cause of this problem is that the selinux container assumes that useradd will create a group with gid==uid, which is not necessarily true. e.g. a GID_MIN higher than the lowest available uid would trigger that problem.

Attaching a sample change that first creates the group and specifies the gid in the useradd invocation, which fixes the problem in my test env.

Comment 2 Josep 'Pep' Turro Mauri 2014-03-06 14:40:43 UTC
(In reply to Josep 'Pep' Turro Mauri from comment #0)
> Steps to Reproduce:
> 1. edit /etc/login.defs in the node, set GID_MIN to 1001
> 2. try 'rhc app create ...' from a client

A simpler reproducer, on a fresh node install this should reproduce the problem:

# sed -i.bak -e 's/^GID_MIN.*/GID_MIN 10000/' /etc/login.defs
# oo-devel-node app-create -c testing123 --with-namespace deleteme --with-app-uuid testing123 --with-app-name testing123

results in:

Shell command '/sbin/runuser -s /bin/sh testing123 -c "exec /usr/bin/runcon 'unconfined_u:system_r:openshift_t:s0:c0,c1000' /bin/sh -c \"/usr/bin/ssh-keygen -N '' -f /var/lib/openshift/testing123/.openshift_ssh/id_rsa\""' returned an error. rc=1
Generating public/private rsa key pair.
Saving the key failed: /var/lib/openshift/testing123/.openshift_ssh/id_rsa.

open /var/lib/openshift/testing123/.openshift_ssh/id_rsa failed: Permission denied.

Comment 3 Brenton Leanhardt 2014-03-06 15:36:49 UTC
This change looks reasonable.  I'll sync with upstream to see what their thoughts are and if there was any reason the code assumed uid==gid.

On a related note, it definitely seems like we need an oo-diagnostics check to make sure UID_MIN/MAX and GID_MIN/MAX in /etc/login.defs are compatible with GEAR_MIN/MAX_UID in /etc/openshift/node.conf.  That seems like another way for things to be misconfigured.

Comment 4 Brenton Leanhardt 2014-03-11 14:25:58 UTC
I'm digging into this a bit more.  It seems like there are other places in the code that expect uid==gid.

Comment 5 Josep 'Pep' Turro Mauri 2014-03-11 17:13:03 UTC
Actually the patch proposed in comment #1 just makes sure that the expected gid is applied - so in practice it actually makes sure that uid == gid and the code that assumes this will just work.

However there's a different (but related) problem: 'next_uid' only checks local passwd/groupo files for duplicity so you get into similar problems if the system has external user info configured (e.g. NIS, LDAP). Just reproduced it, will create a separate bug.

Comment 6 Brenton Leanhardt 2014-03-11 18:02:22 UTC
You're correct.  I was confused at first because of a DNS problem in my environment that was causing ssh logins to fail on my patched Node.  I was assuming there was something wrong with the patch.

Comment 7 Brenton Leanhardt 2014-03-11 18:40:12 UTC
I'm submitting this PR upstream @ https://github.com/openshift/origin-server/pull/4933.

Comment 8 openshift-github-bot 2014-03-12 22:10:37 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/2dc7732175c19c7f27362f909fc687127872ce8b
Bug 1053485 - Changing GID_MIN in login.defs prevents app creation

This patch was originally submitted by Josep 'Pep' Turro Mauri <pep>

Comment 10 Gaoyun Pei 2014-03-17 10:04:57 UTC
Retest this bug with rubygem-openshift-origin-container-selinux-0.7.1-1.git.0.4244282.el6op.noarch on puddle 2.1/2014-03-14.1.

It still failed with sshkey permission issue, checked the code, the related patch has not been merged in, so assign back this bug.

Comment 12 Gaoyun Pei 2014-03-18 06:24:57 UTC
Verify this bug on puddle 2.1/2014-03-17.2.

Setup an ose env with htpasswd authentication enabled, modify the GID_MIN to 6000 in login.defs on node.

Could create app and ssh into the app successfully.