Bug 963192

This happens because the updates /etc/cron.minutely/openshift-facts loads facter.  If CREATE_APP_SYMLINKS is enabled the following code in /opt/rh/ruby193/root/usr/share/ruby/vendor_ruby/facter/openshift_facts.rb will trigger the denial:

 70 Dir.glob("/var/lib/openshift/*").each do |app_dir|
 71   if File.directory?(app_dir) && !File.symlink?(app_dir)
 72     git_repos_count += Dir.glob(File.join(app_dir, "git/*.git")).count
 73 

The cronjob tries to read the symlink and it can't.  The result is that the gear related facts will always be 0:

[root@broker ~]# mco inventory node1.example.com | grep active
      active_capacity => 0.0
      district_active => true
      gears_active_count => 0
      gears_active_usage_pct => 0.0
      max_active_gears => 100

Putting SELinux into permissive mode results in the gears_active_count

cat audit.log | audit2allow -R

require {
        type openshift_cron_t;
        type openshift_var_lib_t;
        class capability dac_override;
        class lnk_file { read getattr };
}

#============= openshift_cron_t ==============
allow openshift_cron_t openshift_var_lib_t:lnk_file { read getattr };
allow openshift_cron_t self:capability dac_override

I'm setting this to NEEDINFO to get Miroslav's advice.

allow openshift_cron_t self:capability dac_override

should be fixed.

allow openshift_cron_t openshift_var_lib_t:lnk_file { read getattr };

needs to be fixed.

 Please clone this bug for selinux-policy.

From /etc/openshift/node.conf, I set CREATE_APP_SYMLINKS=1 (the default is 0).

CREATE_APP_SYMLINKS=1                                        # If set to 1, creates gear-name symlinks to the UUID directories (debugging only)

Then I create some new apps and get stuff like this:

root@node: ~ # tail -1 /var/log/messages
May 16 11:36:54 node setroubleshoot: SELinux is preventing /opt/rh/ruby193/root/usr/bin/ruby from read access on the lnk_file /var/lib/openshift/prf1832-ns. For complete SELinux messages. run sealert -l a26747d5-082c-442e-9446-718453a33222

root@node: ~ # sealert -l a26747d5-082c-442e-9446-718453a33222
SELinux is preventing /opt/rh/ruby193/root/usr/bin/ruby from read access on the lnk_file /var/lib/openshift/prf0210-ns.

*****  Plugin catchall (100. confidence) suggests  ***************************

If you believe that ruby should be allowed read access on the prf0210-ns lnk_file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# grep ruby /var/log/audit/audit.log | audit2allow -M mypol
# semodule -i mypol.pp


root@node: ~ # grep prf1832-ns /var/log/audit/audit.log
type=AVC msg=audit(1368718567.116:68801): avc:  denied  { read } for  pid=13566 comm="ruby" name="prf1832-ns" dev=dm-0 ino=15736359 scontext=system_u:system_r:openshift_cron_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:openshift_var_lib_t:s0 tclass=lnk_file
type=AVC msg=audit(1368718629.252:71717): avc:  denied  { read } for  pid=16295 comm="ruby" name="prf1832-ns" dev=dm-0 ino=15736359 scontext=system_u:system_r:openshift_cron_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:openshift_var_lib_t:s0 tclass=lnk_file


root@node: /var/lib/openshift/prf1832-ns # ls -alZ
drwxr-x---. root                     5194701e6892dff10c003f5d system_u:object_r:openshift_var_lib_t:s0 .
drwxr-xr-x. root                     root                     system_u:object_r:openshift_var_lib_t:s0 ..
drwxr-xr-x. root                     5194701e6892dff10c003f5d system_u:object_r:openshift_var_lib_t:s0:c5,c103 app-root
drwxr-x---. root                     5194701e6892dff10c003f5d system_u:object_r:openshift_var_lib_t:s0 .env
drwxr-xr-x. root                     root                     unconfined_u:object_r:openshift_var_lib_t:s0 git
-rw-r--r--. root                     root                     unconfined_u:object_r:openshift_var_lib_t:s0 .gitconfig
-rw-r--r--. root                     5194701e6892dff10c003f5d system_u:object_r:openshift_var_lib_t:s0:c5,c103 .pearrc
drwxr-xr-x. 5194701e6892dff10c003f5d 5194701e6892dff10c003f5d system_u:object_r:openshift_var_lib_t:s0:c5,c103 php
d---------. root                     root                     system_u:object_r:openshift_tmp_t:s0 .sandbox
drwxr-x---. root                     5194701e6892dff10c003f5d system_u:object_r:ssh_home_t:s0  .ssh
d---------. root                     root                     system_u:object_r:openshift_tmp_t:s0 .tmp
root@node: /var/lib/openshift/prf1832-ns # cd ../
root@node: /var/lib/openshift # ls -ald /var/lib/openshift/prf1832-ns
lrwxrwxrwx. 1 root root 24 May 16 01:35 /var/lib/openshift/prf1832-ns -> 5194701e6892dff10c003f5d
root@node: /var/lib/openshift # ls -aldZ /var/lib/openshift/prf1832-ns
lrwxrwxrwx. root root unconfined_u:object_r:openshift_var_lib_t:s0 /var/lib/openshift/prf1832-ns -> 5194701e6892dff10c003f5d
root@node: /var/lib/openshift # ls -aldZ /var/lib/openshift/5194701e6892dff10c003f5d
drwxr-x---. root 5194701e6892dff10c003f5d system_u:object_r:openshift_var_lib_t:s0 /var/lib/openshift/5194701e6892dff10c003f5d

root@node: ~/selinux # time grep ruby /var/log/audit/audit.log | audit2allow -R -M lnk_file

root@node: ~/selinux # cat lnk_file.te

policy_module(lnk_file, 1.0)

require {
	type openshift_cron_t;
	type openshift_var_lib_t;
	class lnk_file read;
}

#============= openshift_cron_t ==============
allow openshift_cron_t openshift_var_lib_t:lnk_file read;
root@node: ~/selinux # cat lnk_file.te

policy_module(lnk_file, 1.0)

require {
	type openshift_cron_t;
	type openshift_var_lib_t;
	class lnk_file read;

}

#============= openshift_cron_t ==============
allow openshift_cron_t openshift_var_lib_t:lnk_file read;








Later I found some more selinux errors, and then generated this policy:

root@node: ~/selinux # cat lnk_file2.te

policy_module(lnk_file2, 1.0)

require {
	type openshift_cron_t;
	type openshift_var_lib_t;
	class lnk_file getattr;
}

#============= openshift_cron_t ==============
allow openshift_cron_t openshift_var_lib_t:lnk_file getattr;



Loading both of those seems to have done the trick, no more denials so far.

*** Bug 963830 has been marked as a duplicate of this bug. ***

Yes, we added fixes to Fedora and will have a fix in RHEL6 ASAP.

Retest this bug with selinux-policy-3.7.19-199.el6, but failed.

[root@node1 ~]# rpm -q selinux-policy
selinux-policy-3.7.19-199.el6.noarch
[root@node1 ~]# tailf /var/log/audit/audit.log |grep avc
<--snip-->
type=AVC msg=audit(1370591401.820:115270): avc:  denied  { read } for  pid=19430 comm="ruby" name="jbossews2i25m9pdq-ece03ko19z" dev=dm-0 ino=24531 scontext=system_u:system_r:openshift_cron_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=lnk_file
type=AVC msg=audit(1370591401.821:115271): avc:  denied  { read } for  pid=19430 comm="ruby" name="51b069ca0f5b152695000010-jzu" dev=dm-0 ino=26232 scontext=system_u:system_r:openshift_cron_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=lnk_file
type=AVC msg=audit(1370591401.821:115272): avc:  denied  { read } for  pid=19430 comm="ruby" name="h4qbzitnbg-sun000" dev=dm-0 ino=23985 scontext=system_u:system_r:openshift_cron_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=lnk_file
<--snip-->


Actually this bug is fixed in selinux-policy-3.7.19-195.el6_4.6, after I downgrade packages to selinux-policy-3.7.19-195.el6_4.6, this issue disappeared.

BTW, selinux-policy-3.7.19-195.el6_4.6 is already available in http://cdn.rcm-qa.redhat.com/content/dist/rhel/server/6/6Server/x86_64/os, once selinux-policy-3.7.19-199.el6 is pulled into ose puddle repo, then selinux-policy-3.7.19-199.el6 will be installed, this issue will reproduce.

It should be in this release. 

What does

# rpm -qa |grep selinux

Could you try to re-test it with

selinux-policy-targeted-3.7.19-200.el6

Miroslav,

Did I tag in the correct build?  The RHEL 6.4 bug says it was fixed in selinux-policy-3.7.19-199.el6.

Nevermind, please ignore the NEEDINFO.  I see now you already chimed in. :)

Jianlin you cate get the packages Miroslav mentioned from here:
http://download.devel.redhat.com/brewroot/packages/selinux-policy/3.7.19/200.el6/noarch/

Tagged selinux-policy-3.7.19-195.el6_4.10 into the latest puddle, please verify with this version of selinux-policy

Verify this bug on puddle:
http://buildvm-devops.usersys.redhat.com/puddle/build/OpenShiftEnterprise/1.2/2013-06-07.2/
[root@node2 yum.repos.d]# rpm -qa|grep selinux-policy
selinux-policy-3.7.19-195.el6_4.10.noarch
selinux-policy-targeted-3.7.19-195.el6_4.10.noarch

Modify /etc/openshift/node.conf, set CREATE_APP_SYMLINKS=1
Created a jbosseap-6.0 app, control this app like restart or ssh into it.
No avc denials in audit.log

Closing all bugs introduced, fixed, and verified during 1.2 release work (thus never shipped).