Created attachment 652254 [details]
Output from grep denied /var/log/audit/audit.log

Details of the SELinux errors.

Ok, I see the error in the guide.

restorecon -rv /var/run
restorecon -rv /usr/share/rubygems/gems/passenger-*
restorecon -rv /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
restorecon -rv /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift

The problem is "-R" should be used instead of "-r". So

restorecon -R -v /var/run
restorecon -R -v /usr/share/rubygems/gems/passenger-*
restorecon -R -v /usr/sbin/mcollectived /var/log/mcollective.log /var/run/mcollectived.pid
restorecon -R -v /var/lib/openshift /etc/openshift/node.conf /etc/httpd/conf.d/openshift


should fix this issue. Could you test it?

Does "-R" behave differently than "-r"?  The man page for restorecon doesn't imply that it does.

-R -r  change files and directories file labels recursively

Yes. Forget it.

Peter,
did you run it?

I did a fixfiles onboot and that didn't solve any selinux problems.

What does

# ls -alZ /var/lib/openshift

This is the output with semanage permissive -a quota_t 
Meaning I no longer get a permission denied, but the audit.log still shows the issue.

# ls -aZl /var/lib/openshift/.
total 32
drwxr-xr-x.  3 system_u:object_r:openshift_var_lib_t:s0 root root  4096 Nov 26 14:16 .
drwxr-xr-x. 31 system_u:object_r:var_lib_t:s0   root root  4096 Nov 24 13:58 ..
-rw-------.  1 unconfined_u:object_r:quota_db_t:s0 root root  6144 Nov 26 14:16 aquota.user
drwx------.  2 system_u:object_r:openshift_var_lib_t:s0 root root 16384 Nov 26 14:15 lost+found

What "checkquota -cmug" does is try to create the aquota.user file which due to the selinux policy issues is denied. The SE contexts on the directory etc. are all as they were before checkquota was run.

Miroslav I think 
5d5c6b1669abcecf49e95157d1f342b5b1a995cb

Will fix the problem.

Yes, the restorecon masked the quota_db was created as openshift_var_lib_t.

Peter,
could you test it with the latest policy builds.

https://brewweb.devel.redhat.com/buildinfo?buildID=245393

(In reply to comment #11)
> Peter,
> could you test it with the latest policy builds.
> 
> https://brewweb.devel.redhat.com/buildinfo?buildID=245393

[root@node1 ~]# rpm --upgrade selinux-policy-3.7.19-183.el6.noarch.rpm selinux-policy-targeted-3.7.19-183.el6.noarch.rpm 
libsemanage.get_home_dirs: e2ddef0fc72144e5bc9530bbb0d68496 homedir /var/lib/openshift/e2ddef0fc72144e5bc9530bbb0d68496 or its parent directory conflicts with a file context already specified in the policy.  This usually indicates an incorrectly defined system account.  If it is a system account please make sure its uid is less than 500 or its login shell is /sbin/nologin.
[root@node1 ~]# quotacheck -cmug /mnt/test
quotacheck: Cannot create new quotafile /mnt/test/aquota.user.new: Permission denied
quotacheck: Cannot initialize IO on new quotafile: Permission denied


Still fails. I'm getting the strange error in between - not sure if it means anything.

sealert suggests I do a full relabel. Will do that and comment again.

Btw - /mnt/test is my test using the openshift_var_lib_t context:

# ls -ladZ /mnt/test
drwxr-xr-x. root root system_u:object_r:openshift_var_lib_t:s0 /mnt/test

full relabel (fixfiles onboot) did not resolve the problem either.

Steps followed:
semanage permissive -d quota_t
lvcreate -n lvtest -L 4g vg_broker
mkfs.ext4 -m 0 /dev/vg_broker/lvtest
mkdir /mnt/test
mount -o usrquota /dev/vg_broker/lvtest /mnt/test
rpm --upgrade selinux-policy-3.7.19-183.el6.noarch.rpm selinux-policy-targeted-3.7.19-183.el6.noarch.rpm
# get "strange message"
chcon -t openshift_var_lib_t /mnt/test
quotacheck -cmug /mnt/test
# fails

Ok, the first issue is with a user with homedir in /var/lib/openshift.

"Please make sure its uid is less than 500 or its login shell is /sbin/nologin"

If you execute

# setenforce 0
# quotacheck -cmug /mnt/test
# ausearch -m avc -ts recent

(In reply to comment #17)
> Ok, the first issue is with a user with homedir in /var/lib/openshift.
> 
> "Please make sure its uid is less than 500 or its login shell is
> /sbin/nologin"

I see this entry in /etc/passwd:
e2ddef0fc72144e5bc9530bbb0d68496:x:500:500:OpenShift guest:/var/lib/openshift/e2ddef0fc72144e5bc9530bbb0d68496:/usr/bin/oo-trap-user

Is /etc/login.defs supposed to be changed/modified as part of the install? This user was definitely not created by me (directly).

(In reply to comment #18)
> If you execute
> 
> # setenforce 0
> # quotacheck -cmug /mnt/test
> # ausearch -m avc -ts recent

[root@node1 ~]# quotacheck -cmug /mnt/test
[root@node1 ~]# ausearch -m avc -ts recent
----
time->Thu Nov 29 08:32:31 2012
type=SYSCALL msg=audit(1354195951.886:7041): arch=c000003e syscall=2 success=yes exit=3 a0=7fff2a9a2dc0 a1=c2 a2=180 a3=7fff2a9a2b10 items=0 ppid=4855 pid=6759 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1160 comm="quotacheck" exe="/sbin/quotacheck" subj=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1354195951.886:7041): avc:  denied  { write } for  pid=6759 comm="quotacheck" name="aquota.user.new" dev=dm-2 ino=12 scontext=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file
type=AVC msg=audit(1354195951.886:7041): avc:  denied  { create } for  pid=6759 comm="quotacheck" name="aquota.user.new" scontext=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file
type=AVC msg=audit(1354195951.886:7041): avc:  denied  { add_name } for  pid=6759 comm="quotacheck" name="aquota.user.new" scontext=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=dir
----
time->Thu Nov 29 08:32:31 2012
type=SYSCALL msg=audit(1354195951.888:7042): arch=c000003e syscall=82 success=yes exit=0 a0=7fff2a9a2dc0 a1=7f01fd7ed900 a2=1000 a3=7fff2a9a2aa0 items=0 ppid=4855 pid=6759 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1160 comm="quotacheck" exe="/sbin/quotacheck" subj=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1354195951.888:7042): avc:  denied  { rename } for  pid=6759 comm="quotacheck" name="aquota.user.new" dev=dm-2 ino=12 scontext=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file
type=AVC msg=audit(1354195951.888:7042): avc:  denied  { remove_name } for  pid=6759 comm="quotacheck" name="aquota.user.new" dev=dm-2 ino=12 scontext=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 tcontext=system_u:object_r:file_t:s0 tclass=dir
----
time->Thu Nov 29 08:32:31 2012
type=SYSCALL msg=audit(1354195951.889:7043): arch=c000003e syscall=90 success=yes exit=0 a0=7f01fd7ed900 a1=180 a2=1000 a3=7fff2a9a2aa0 items=0 ppid=4855 pid=6759 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1160 comm="quotacheck" exe="/sbin/quotacheck" subj=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1354195951.889:7043): avc:  denied  { setattr } for  pid=6759 comm="quotacheck" name="aquota.user" dev=dm-2 ino=12 scontext=unconfined_u:unconfined_r:quota_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:file_t:s0 tclass=file

(In reply to comment #19)
> (In reply to comment #17)
> > Ok, the first issue is with a user with homedir in /var/lib/openshift.
> > 
> > "Please make sure its uid is less than 500 or its login shell is
> > /sbin/nologin"
> 
> I see this entry in /etc/passwd:
> e2ddef0fc72144e5bc9530bbb0d68496:x:500:500:OpenShift
> guest:/var/lib/openshift/e2ddef0fc72144e5bc9530bbb0d68496:/usr/bin/oo-trap-
> user

Could you fix uid to see if we can get this working. 

> Is /etc/login.defs supposed to be changed/modified as part of the install?
> This user was definitely not created by me (directly).

The question for OpenShift folks.

How did it work for you?

I've had the broker/node running all weekend with no intervention (this is Christmas month so I don't have much time to do weekend computer stuff).
I'm very puzzled - the user mentioned in comment 19 is no longer defined, and the selinux error is gone. 

I'll retry the patch fix. Instead of testing on a new mount-point, I'm going to remove quota and then add it again to /var/lib/openshift.

I can now add/remove quota with SElinux in enforcing mode on a mounted /var/lib/openshift.  Somehow I'm not able to do that on the /mnt/test - but removing quota and re-creating it for /var/lib/openshift works.

I tested the following packages today:

selinux-policy-targeted-3.7.19-183.el6.noarch
selinux-policy-3.7.19-183.el6.noarch

Here's what I did:

* Created a volume and mounted it at /var/lib/openshift with usrquota turned on
* Tried running 'quotacheck -cmug /var/lib/openshift' before upgrading the packages.  I received permission denied
* I updated the packages.  At this point I saw the same warnings about having an incorrectly defined system account.  In the case of OpenShift I think that warning is incorrect.  /var/lib/openshift is where the Gear ssh account home directories are stored.
* I ran the quotacheck command again and it worked.

At this point I did some general checking of the system and things appear to be working correctly.

Is there any way we could get a build of the policy for RHEL 6.3 that our QE team could perform regression testing on?

You should be able to run the same policy on both RHEL6.3 and 6.4

This policy definitely works on RHEL6.3. I was mostly curious about if we could get a backported policy for RHEL6.3 that we could possibly z-stream.  Would that be possible?

If you want a z-stream, you need to request it.

While we're waiting for the zstream for RHEL 6.3, it's worth noting a workaround. Seems only the quota file creation is blocked.

1. setenforce 0
2. quotacheck -cmug /var/lib/openshift
3. setenforce 1
4. restorecon /var/lib/openshift/aquota.user
5. reboot (there's probably a better way but a simple remount didn't work)
6. New gears will get the quotas. edquota <uuid> for any gears already created, set limits according to gear size.

I'm nominating this bug for 6.3.Z, on behalf of the open shift team.  For questions please email bleanhar

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0314.html