Bug 1349194

*** Bug 1346401 has been marked as a duplicate of this bug. ***

SELinux has been supported on MON and OSD nodes since ceph 1.3.2 according to BZ#1159756 and RHBA-2016:0313.

One important thing to note:  You must install the ceph-selinux rpm.  That rpm contains the policy modules SELinux needs.

ceph-selinux is available from the rhel-7-server-rhceph-1.3-osd-rpms repo on the CDN.

That's correct, SELinux is now supported in enforcing mode in the latest versions of RH Ceph Storage.

Karl, does installing this ceph-selinux package solve all of your issues?

Kurt,

Can you verify that with the ceph-selinux package installed and the proper "ceph_osd_selinux_permissive: false" setting in the Heat template that everything is working as expected.

i.e. SELinux is enabled in enforcing mode and there are no ceph related AVC messages in the audit log.

It appears to work with the following 2 changes:

1) Added the installation of the ceph-selinux package to the custom overcloud image we create

2) Adding ceph_osd_selinux_permissive: false in the dell-environment.yaml file under:

parameter_defaults:
    CephStorageExtraConfig:


After the overcloud was deployed, sestatus now shows:

[root@r7-13g-cephstorage-2 audit]# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

Don't know what I am looking for in the audit.log so I have added as an attachment.  I see some AVS denied messages but will let someone at RH take a look.

Created attachment 1184400 [details]
audit.log for ceph selinux

The audit.log from one of my storage nodes with selinux set to enforcing.

$ grep AVC audit.log | cut -f13- -d' ' | sort | uniq
comm="restorecon" path="/var/lib/ceph/tmp/ceph-disk.activate.lock" dev="sdy2" ino=51383 scontext=system_u:system_r:setfiles_t:s0-s0:c0.c1023 tcontext=system_u:object_r:var_lib_t:s0 tclass=file
comm="restorecon" path="/var/lib/ceph/tmp/ceph-disk.activate.lock" dev="sdy2" ino=51383 scontext=system_u:system_r:setfiles_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=file


So restorecon runs and tries and fails to update the context on the file /var/lib/ceph/tmp/ceph-disk.activate.lock

I don't think that will cause any problems, but it would be nice to have confirmation by an SELinux or Ceph person.

I'd be curious to see the output of:
ls -laZ /var/lib/ceph/tmp/

(In reply to Karl Hastings from comment #8)
> I don't think that will cause any problems, but it would be nice to have
> confirmation by an SELinux or Ceph person.

Boris, is this something new to fix in RHCS 1.3 / 2?

@Karl on another system here's the ouput you asked for:

[root@ceph-osd-01 ~]# ls -laZ /var/lib/ceph/tmp/
drwxr-x---. ceph ceph system_u:object_r:ceph_var_lib_t:s0 .
drwxr-x---. ceph ceph system_u:object_r:ceph_var_lib_t:s0 ..
-rw-r--r--. root root system_u:object_r:ceph_var_lib_t:s0 ceph-disk.activate.lock
-rw-r--r--. root root unconfined_u:object_r:ceph_var_lib_t:s0 ceph-disk.prepare.lock

On this system (CentOS 7.2) ceph-selinux is installed, OSDs are running.
I looked at the audit.log and cannot see the error on /var/lib/ceph/tmp/ceph-disk.activate.lock

Hope that helps.

Worked with Steve Reichard to determine if the AVC messages we are seeing is ok.  He had me run a few commands to see what labels were being placed on particular files.  

semanage fcontext -l | grep ceph  -> This shows no output.  Should there be?


[root@r7-13g-cephstorage-0 tmp]# ls -la -Z
drwxr-xr-x. root root unconfined_u:object_r:var_lib_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:var_lib_t:s0 ..
-rw-r--r--. root root system_u:object_r:var_lib_t:s0   ceph-disk.activate.lock
-rw-r--r--. root root system_u:object_r:var_lib_t:s0   ceph-disk.prepare.lock
-rw-r--r--. root root unconfined_u:object_r:var_lib_t:s0 kurt

Version of ceph-selinux on a storage node:

[root@r7-13g-cephstorage-0 tmp]# rpm -qa | grep ceph-selinux
ceph-selinux-0.94.5-14.el7cp.x86_64

What is the underlying system? Does this fail by any chance on RHEL 7.3? Our re-labelling scripts don't work properly with 7.3 (the SELinux user-space was rebased in 7.3) and this could be related to that. There is the bz1360444 for that.

It is running RHEL 7.2

ceph 1.3.x packages do not pull ceph-selinux as a dependency, while the ceph 2.x packages do

we need to include ceph-selinux explicitly in the list of packages to be installed in the overcloud image for ospd9 (though not in the ospd10 image)

I will move the BZ back into ON_DEV and link the additional submission

Karl, we *can* install ceph-selinux explicitly via Director but, should we add ceph-selinux as a dependency also to the 1.3.x Ceph RPMs, as it happens already for the 2.0 RPMs?

Personally, I hate to muck with rpm dependencies mid release.  My vote would be to add it to the image only.  But I'd defer to someone else if they felt strongly about it.

IIRC this was an explicit decision to not set the hard dependency until RHCS 2.

yes, that would be better.
It can be cloned bug