Bug 988519

Summary: cron jobs fail due to selinux context mismatch
Product: OpenShift Online Reporter: Andy Grimm <agrimm>
Component: ContainersAssignee: Rob Millner <rmillner>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 2.xCC: bmeng, chunchen, jgoulding, mfisher, qiuzhang, rmillner, xtian
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1003645 (view as bug list) Environment:
Last Closed: 2013-08-07 22:56:44 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1003645    

Description Andy Grimm 2013-07-25 18:40:42 UTC
Description of problem:

Commands which call "oo_spawn" (e.g., "gear" or "ctl_app" commands) do not work inside cron jobs.  The reason is that the "user" part of selinux context is expected to be "unconfined_u", but under cron it is "system_u"

Steps to Reproduce:
1. create an app which has the "cron" cartridge installed
2. add a script called .openshift/cron/minutely/restart.sh to your git repo
with these contents:

#!/bin/sh
gear stop
gear start

Actual results:

The script will fail with a "could not open session" error

Expected results:

The script should succeed

Additional info:

I tracked this down to the oo_spawn function in node/lib/openshift-origin-node/utils/shell_exec.rb

The code runs:

              current_context  = SELinux.getcon
              target_context   = SELinux.context_from_defaults(SELinux.get_mcs_label(options[:uid]))

and compares the two.  if they do not match, then it tries to spawn the specified command under runuser / runcon; with our PAM configuration, this is only allowed for root.  When a command fails in this way under cron, you'll see messages like this in /var/log/secure:

Jul 25 14:35:16 ex-std-nodeXX runuser: pam_unix(runuser:session): session opened for user XXXXXXXXX by (uid=XXXX)
Jul 25 14:35:16 ex-std-nodeXX runuser: pam_namespace(runuser:session): Error opening config file /etc/security/namespace.d/shm.conf
Jul 25 14:35:16 ex-std-nodeXX runuser: PAM audit_open() failed: Permission denied
Jul 25 14:35:16 ex-std-nodeXX runuser: pam_unix(runuser:session): session closed for user XXXXXXXXX

Comment 1 Andy Grimm 2013-07-25 19:21:35 UTC
I don't know whether it's the _correct_ fix, but one possible fix would be to change the runcon invocation in daemon_as_user in oo-scheduled-jobs.  It currently has:

daemon --user="$uuid" runcon -r system_r -t openshift_t -l $mcs_level "$@"

which could be:

daemon --user="$uuid" runcon -u unconfined_u -r system_r -t openshift_t -l $mcs_level "$@"

Comment 2 Rob Millner 2013-07-25 20:42:04 UTC
Implemented the above fix:
https://github.com/openshift/origin-server/pull/3173


Stopping gear...
CLIENT_RESULT: Stopping CRON cart
Waiting for stop to finish
Starting gear...
App container start_gear
CLIENT_RESULT: Starting CRON cart
__________________________________________________________________________
Thu Jul 25 16:40:11 EDT 2013: END minutely cron run - status=0
__________________________________________________________________________

Comment 3 openshift-github-bot 2013-07-26 01:06:08 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/d1f0ecc25c07ef8c1c9aaabc616b3e4353bedd22
Bug 988519 - Ensure that the gear task runs as unconfined_u.

Comment 4 chunchen 2013-07-29 05:45:28 UTC
It's fixed, verified on devenv_3572, please refer to the following results:

1. create an app which has the "cron" cartridge installed

2. add a script called .openshift/cron/minutely/restart.sh to your git repo
with these contents and git push:
#!/bin/sh
gear stop
gear start

3. ssh into this app and check the cron log
\> cat ./cron/log/cron.minutely.log
Stopping gear...
CLIENT_RESULT: Stopping CRON cart
Waiting for stop to finish
Starting gear...
CLIENT_RESULT: Starting CRON cart
__________________________________________________________________________
Mon Jul 29 01:40:14 EDT 2013: END minutely cron run - status=0
__________________________________________________________________________