Bug 1256374 - Compiling Ceph with lttng requires additional selinux policy changes
Compiling Ceph with lttng requires additional selinux policy changes
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: selinux-policy (Show other bugs)
7.2
All Linux
high Severity high
: rc
: ---
Assigned To: Lukas Vrabec
Milos Malik
:
Depends On:
Blocks: 1295396 1375898
  Show dependency treegraph
 
Reported: 2015-08-24 08:36 EDT by Milan Broz
Modified: 2016-11-03 22:20 EDT (History)
15 users (show)

See Also:
Fixed In Version: selinux-policy-3.13.1-87.el7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1375898 (view as bug list)
Environment:
Last Closed: 2016-11-03 22:20:51 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Milan Broz 2015-08-24 08:36:44 EDT
Description of problem:

Recent builds of Ceph uses lttng tracepoing package (see bugs #1207334 and #1207362 etc)

Enabling lttng has some security consequences.

The most visible is requirement to extend prepared selinux policy, according to our simple tests, these are AVCs caused by lttng linkage (this is what produces Fedora build running some of ceph_test with selinux policy active):

type=AVC msg=audit(1439997587.798:67): avc:  denied  { mount } for  pid=800 comm="setfiles" name="/" dev="tracefs" ino=1 scontext=system_u:system_r:setfiles_t:s0 tcontext=system_u:object_r:unlabeled_t:s0 tclass=filesystem permissive=1
type=AVC msg=audit(1439997596.070:79): avc:  denied  { sys_ptrace } for  pid=428 comm="systemd-sysctl" capability=19  scontext=system_u:system_r:systemd_sysctl_t:s0 tcontext=system_u:system_r:systemd_sysctl_t:s0 tclass=capability permissive=1
type=AVC msg=audit(1439997667.898:78): avc:  denied  { sys_ptrace } for  pid=404 comm="systemd-sysctl" capability=19  scontext=system_u:system_r:systemd_sysctl_t:s0 tcontext=system_u:system_r:systemd_sysctl_t:s0 tclass=capability permissive=1
type=AVC msg=audit(1439997842.977:473): avc:  denied  { open } for  pid=953 comm="ceph-mon" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=17363 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=1
type=AVC msg=audit(1439997865.751:817): avc:  denied  { open } for  pid=1802 comm="ceph-osd" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=17363 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=1
type=AVC msg=audit(1439997977.909:994): avc:  denied  { open } for  pid=2200 comm="crushtool" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=17363 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=1
type=AVC msg=audit(1439997986.524:1015): avc:  denied  { open } for  pid=2292 comm="crushtool" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=17363 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=1
type=AVC msg=audit(1439998592.796:1021): avc:  denied  { open } for  pid=4625 comm="crushtool" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=17363 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file permissive=1


(The ptrace could be really dangerous in production systems.)

Could we please collect here some ideas if this is really what should be enabled in el7 build?

Version-Release number of selected component (if applicable):
lttng seems to be enabled in ceph-0.94.2-7.el7cp
Comment 2 Boris Ranto 2015-08-24 08:44:18 EDT
CC'ing Josh.

@Josh: Do you consider this feature practical/important for customers?
Comment 3 Josh Durgin 2015-08-25 20:30:57 EDT
Yes, we want this enabled to be able to collect workload traces.

The ptrace calls are for syscall tracing, which we aren't interested in, so we don't need to allow those. I don't think we care about tracefs either - denying that is fine.

/dev/shm/lttng-ust-wait-5 is used by the lttng cli to communicated with liblttng-ust to enabled/disable tracing, so read-only access to that is needed.

We're most interested in using this with qemu + rbd, which might need some more selinux changes.
Comment 4 Milan Broz 2015-08-26 01:40:07 EDT
I am afraid we cannot selectively allow something there, if lttng generates these AVCs it must be disabled in compile time then to use these.

And generating AVC denials on production system is not acceptable.

I even think that predictable name /dev/shm/lttng-ust-wait-5 could be a security problem per se. Is really compiling-in debugging features for everyone on production systems good idea here?
Comment 5 Boris Ranto 2015-08-26 09:53:15 EDT
I still don't see too much of a use here for _end customers_. I understand that this feature might significantly help with debugging but how may a customer use it in practice? --  i.e. Is there any workload (not debugging workload) where customers might want to use this feature? How can it be used in conjunction with qemu?

I'd be ok with the feature being on in the packages if there were no security concerns related to this feature but:

1.) the feature is pretty much untested (at least on rhel 7, upstream spec file still builds ceph for rhel 7 without lttng support)

2.) based on a fedora report, it creates a world writable fixed path file which is definitely not good:
  https://bugzilla.redhat.com/show_bug.cgi?id=1223319
Comment 6 Josh Durgin 2015-08-27 03:27:24 EDT
(In reply to Milan Broz from comment #4)
> I am afraid we cannot selectively allow something there, if lttng generates
> these AVCs it must be disabled in compile time then to use these.

Ah, that's a pain. What's the usual procedure for things like this then? Fix them/make them configurable upstream? Separate unwanted parts so they can not be packaged?

> And generating AVC denials on production system is not acceptable.

Makes sense, I was hoping there was a way to let regular permissions deny unnecessary accesses.

> I even think that predictable name /dev/shm/lttng-ust-wait-5 could be a
> security problem per se. Is really compiling-in debugging features for
> everyone on production systems good idea here?

(In reply to Boris Ranto from comment #5)
> I still don't see too much of a use here for _end customers_. I understand
> that this feature might significantly help with debugging but how may a
> customer use it in practice? --  i.e. Is there any workload (not debugging
> workload) where customers might want to use this feature? How can it be used
> in conjunction with qemu?

The use case is to enable customers to capture what their workloads actually are, for example with virtual machines using an rbd block device. Qemu links to librbd which links to liblttng-ust for tracing.

librbd has lttng tracepoints that efficiently capture an I/O workload, and
there are tools for analyzing and replaying them later, for tuning.

Rather than saying vms are for 'everything', they can sample what their
workload actually is, and tune for that instead of guessing and approximating.

> I'd be ok with the feature being on in the packages if there were no
> security concerns related to this feature but:
> 
> 1.) the feature is pretty much untested (at least on rhel 7, upstream spec
> file still builds ceph for rhel 7 without lttng support)
> 
> 2.) based on a fedora report, it creates a world writable fixed path file
> which is definitely not good:
>   https://bugzilla.redhat.com/show_bug.cgi?id=1223319

Maybe we should add an option to lttng to ignore the global and per-user 'lttng-ust-wait-5' file, and just use per-vm file (for the qemu use case) that could be properly labelled for selinux by libvirt.
Comment 7 Milan Broz 2015-08-27 03:43:02 EDT
LTTng is excellent tool but enterprise builds usually do not contain debugging features in stable released builds (I think even for lttng there is some small performance overhead. There can be another debug-enabled build similar to kernel-debug in RHEL).

Anyway, there are several unresolved SELinux issues related to lttng even in Fedora, so introducing this to RHEL can easily cause more problems than it solves (there is in fact no testing related to SELinux).

So the first question is if we really want to introduce it in RH build (security and supportability is the major questions here) and if so we have to find how to do it properly.

If we are interested only in UST (userspace tracing) then there probably should be some way how enable just this without messing with syscalls etc...
Comment 8 Milan Broz 2015-08-27 03:45:13 EDT
Mirek, can we somehow support lttng in SELinux RHEL policy in future?
Comment 9 Miroslav Grepl 2015-08-27 09:48:32 EDT
(In reply to Milan Broz from comment #8)
> Mirek, can we somehow support lttng in SELinux RHEL policy in future?

Yes, we have Fedora bug for this issue but not sure if we are able to have working fix in RHEl-7.2.
Comment 10 Federico Lucifredi 2015-08-27 18:10:43 EDT
Hi guys,
  LTTng will start shipping in 1.3.1, and we see value in having tracing as a customer feature out of the box. I do not want to back that out — I want the tracing problem solved in 1.3.1, and the hardened security one in 1.3.2, so to speak. One can of worms at a time methodology :)


  Given that, we need to plan any additional SELinux work to secure these. SELinux currently is targeted at 1.3.2, meaning we have at the very least until the end of October to do this, and possibly longer.

thanks -F
Comment 11 Milan Broz 2015-09-02 08:13:02 EDT
Enabling lttng without basic support in selinux-policy means that customers will get AVCs and I think it will not work in enforcing mode at all.

(And according to comment#9 lttng support we cannot be sure RHEL7.2 will have it. This must be coordinated with RHEL PM otherwise it will cause a lot of noise...
Updating Ceph selinux policy is not enough here.)
Comment 12 Miroslav Grepl 2015-09-02 10:09:27 EDT
(In reply to Milan Broz from comment #11)
> Enabling lttng without basic support in selinux-policy means that customers
> will get AVCs and I think it will not work in enforcing mode at all.
> 
> (And according to comment#9 lttng support we cannot be sure RHEL7.2 will
> have it. This must be coordinated with RHEL PM otherwise it will cause a lot
> of noise...
> Updating Ceph selinux policy is not enough here.)

In this case, there needs to be a request (bug) from you agains selinux-policy component where we can discuss it if we are able to find a solution.
Comment 15 Ken Dreyer (Red Hat) 2015-12-11 16:15:32 EST
(In reply to Miroslav Grepl from comment #12)
> In this case, there needs to be a request (bug) from you agains
> selinux-policy component where we can discuss it if we are able to find a
> solution.

Milan, did you file a bug there?
Comment 16 Milan Broz 2015-12-14 15:29:08 EST
I was under the impression that the decision was to not link to lttng in released version... Siddharth, do you know details here?
If not, I'll fill the bug.
Comment 17 Ken Dreyer (Red Hat) 2015-12-14 16:42:14 EST
We did link lttng in the released version.
Comment 18 Milan Broz 2015-12-15 06:37:55 EST
Hm. ok.

I think lttng SELinux extension is not fixed yet even in Fedora, see bug #1222157.

I'll reassign this to RHEL7 selinux-policy for discussion then, local Ceph SELinux should not solve this problem anyway, it is generic issue with lttng support.

Short summary: Ceph is now linked with linked with lttng, in following release it will fully support SElinux, so users could start to see AVC as mentioned above.

We will need to depend on some global selinux policy that covers lttng issues.
(Or alternatively it can be part of Ceph selinux policy but then we need sync with RHEL selinux maintainers.)
Comment 20 Boris Ranto 2015-12-15 11:06:30 EST
@Milan: The SELinux changes should not be so critical that we should include them in the ceph SELinux policy. Mostly, thanks to

https://github.com/ceph/ceph/pull/6415

that should bring the library split to hammer (it is already in master) -- i.e. afaik, there should be two versions of the libraries, the default one without trace points and the trace point one which has to be enabled manually.

We should probably file a documentation bug though -- i.e. document that if someone wants to debug (use lttng-related stuff), he should disable SELinux support. At least until the SELinux policy changes for lttng-ust will make it into selinux-policy package.
Comment 22 Miroslav Grepl 2015-12-18 04:57:58 EST
(In reply to Boris Ranto from comment #20)
> @Milan: The SELinux changes should not be so critical that we should include
> them in the ceph SELinux policy. Mostly, thanks to
> 
> https://github.com/ceph/ceph/pull/6415
> 
> that should bring the library split to hammer (it is already in master) --
> i.e. afaik, there should be two versions of the libraries, the default one
> without trace points and the trace point one which has to be enabled
> manually.
> 
> We should probably file a documentation bug though -- i.e. document that if
> someone wants to debug (use lttng-related stuff), he should disable SELinux
> support. At least until the SELinux policy changes for lttng-ust will make
> it into selinux-policy package.

You can talk about permissive mode instead of SELinux disabling.
Comment 25 Lukas Vrabec 2016-06-14 09:02:01 EDT
What is state of this BZ? Can I fix these AVCs and create some pull request? Could you point me to right repo? 

Thank you.
Comment 26 Boris Ranto 2016-06-28 02:43:01 EDT
AFAIK, the idea here is that we should create (back-port?) the lttng-ust policy to RHEL 7.3.

While the package (lttng-ust) is not in base, a layered product requires this package and it stops us from fully enabling SELinux on some of the machines (currently, we need to disable SELinux for Ceph to work with the lttng trace-points -- it is a non-default behaviour used mostly for debugging but we still need to disable SELinux if we want to use that feature).
Comment 27 Lukas Vrabec 2016-06-28 03:57:32 EDT
Boris, 
Am I right, (lttng-ust) will be part of selinux-policy package and also part of some of yours packages? (With changes in rhel-7.3 we can make it working, more info: https://mgrepl.wordpress.com/2015/07/31/cil-part2-module-priorities/) ? 

Or this is some package needed by ceph and we also need policy for it? Do you have some policy sources for lttng-ust? 

I'm looking on lttng-ust package and it's just library. I don't think we need policy for this. Could you attach output of 
# ps -efZ | grep unconfined_service_t

To show which processes need policy? 

Thank you.
Comment 28 Boris Ranto 2016-06-28 05:05:44 EDT
Lukas,

you are right, lttng-ust is just a library. I got confused by Comment #9 that suggested that this was being investigated in fedora. @Miroslav: Any idea what bz were you referring to? Maybe, some domain transitions "magic" as lttng-ust always creates/manipulates the files that generate the denial on its own?

If not, then it might be the case that we just need to update the ceph policy to allow these denials.
Comment 29 Milos Malik 2016-06-28 05:18:37 EDT
The lttng-ust package contains some libraries, but there are also 2 daemons:

/usr/bin/lttng-relayd
/usr/bin/lttng-sessiond

# service lttng-sessiond status
● lttng-sessiond.service - LSB: The LTTng session daemon
   Loaded: loaded (/etc/rc.d/init.d/lttng-sessiond; bad; vendor preset: enabled)
   Active: inactive (dead) since Tue 2016-06-28 11:14:37 CEST; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11325 ExecStop=/etc/rc.d/init.d/lttng-sessiond stop (code=exited, status=0/SUCCESS)

Jun 28 11:13:37 rhel71.localdomain systemd[1]: Starting LSB: The LTTng sessi....
Jun 28 11:13:37 rhel71.localdomain lttng-sessiond[11056]: Starting lttng-sess...
Jun 28 11:13:37 rhel71.localdomain systemd[1]: Started LSB: The LTTng sessio....
Jun 28 11:14:37 rhel71.localdomain systemd[1]: Stopping LSB: The LTTng sessi....
Jun 28 11:14:37 rhel71.localdomain lttng-sessiond[11325]: Stopping lttng-sess...
Jun 28 11:14:37 rhel71.localdomain systemd[1]: Stopped LSB: The LTTng sessio....
Hint: Some lines were ellipsized, use -l to show in full.
# service lttng-sessiond start
Starting lttng-sessiond (via systemctl):                   [  OK  ]
# service lttng-sessiond status
● lttng-sessiond.service - LSB: The LTTng session daemon
   Loaded: loaded (/etc/rc.d/init.d/lttng-sessiond; bad; vendor preset: enabled)
   Active: active (running) since Tue 2016-06-28 11:14:44 CEST; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11325 ExecStop=/etc/rc.d/init.d/lttng-sessiond stop (code=exited, status=0/SUCCESS)
  Process: 11450 ExecStart=/etc/rc.d/init.d/lttng-sessiond start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/lttng-sessiond.service
           └─11457 /usr/bin/lttng-sessiond -d

Jun 28 11:14:44 rhel71.localdomain systemd[1]: Starting LSB: The LTTng sessi....
Jun 28 11:14:44 rhel71.localdomain lttng-sessiond[11450]: Starting lttng-sess...
Jun 28 11:14:44 rhel71.localdomain systemd[1]: Started LSB: The LTTng sessio....
Hint: Some lines were ellipsized, use -l to show in full.
# ps -efZ | grep lttng
system_u:system_r:initrc_t:s0   root     11457     1  0 11:14 ?        00:00:00 /usr/bin/lttng-sessiond -d
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 11515 4182  0 11:14 pts/0 00:00:00 grep --color=auto lttng
# rpm -qf /usr/bin/lttng-sessiond
lttng-tools-2.4.1-1.el7cp.x86_64
# 

Unfortunately, there is no systemd unit file for the first one.
Comment 30 Boris Ranto 2016-06-28 05:44:59 EDT
Those daemons are part of lttng-tools and there already is an upstream (fedora) policy for lttng-tools. I'm not sure how helpful it would be for our case.
Comment 31 Lukas Vrabec 2016-06-28 06:09:09 EDT
Boris, 
Do we need back port lttng-tools policy module to rhel-7.3? Could you post link for repo where I can create pull request to fix above denials?
Comment 32 Boris Ranto 2016-06-28 10:37:26 EDT
I'm not sure whether we need the policy or not. Where could I see its policy files? That might give me an idea whether it will help or not. Anyway, the ceph policy is tracked upstream [1] but you don't need to modify it, I will do the upstream PR if that is in fact what we should do here (well, I should do some local testing, first anyway).

[1] https://github.com/ceph/ceph
Comment 34 Lukas Vrabec 2016-06-28 19:05:32 EDT
Yes, we need to back port it.
Comment 35 Lukas Vrabec 2016-06-29 07:01:26 EDT
Following AVCs: 
time->Mon Jan 25 02:55:42 2016
type=PATH msg=audit(1453708542.827:579): item=0 name="/dev/shm/lttng-ust-wait-5" inode=30760 dev=00:11 mode=0100666 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:initrc_state_t:s0 objtype=NORMAL
type=CWD msg=audit(1453708542.827:579):  cwd="/"
type=SYSCALL msg=audit(1453708542.827:579): arch=c000003e syscall=2 success=yes exit=3 a0=7f9d673fc240 a1=a0000 a2=0 a3=7f9d673fbfc0 items=1 ppid=18554 pid=18568 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="radosgw" exe="/usr/bin/radosgw" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1453708542.827:579): avc:  denied  { open } for  pid=18568 comm="radosgw" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
type=AVC msg=audit(1453708542.827:579): avc:  denied  { read } for  pid=18568 comm="radosgw" name="lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
----
time->Mon Jan 25 02:55:46 2016
type=PATH msg=audit(1453708546.199:582): item=0 name="/dev/shm/lttng-ust-wait-5" inode=30760 dev=00:11 mode=0100666 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:initrc_state_t:s0 objtype=NORMAL
type=CWD msg=audit(1453708546.199:582):  cwd="/"
type=SYSCALL msg=audit(1453708546.199:582): arch=c000003e syscall=2 success=yes exit=3 a0=7fc9b0204240 a1=a0000 a2=0 a3=7fc9b0203fc0 items=1 ppid=18588 pid=18609 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="radosgw" exe="/usr/bin/radosgw" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1453708546.199:582): avc:  denied  { open } for  pid=18609 comm="radosgw" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
type=AVC msg=audit(1453708546.199:582): avc:  denied  { read } for  pid=18609 comm="radosgw" name="lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
----
time->Mon Jan 25 02:55:46 2016
type=PATH msg=audit(1453708546.392:589): item=0 name="/dev/shm/lttng-ust-wait-5-0" inode=30913 dev=00:11 mode=0100640 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0 objtype=NORMAL
type=CWD msg=audit(1453708546.392:589):  cwd="/"
type=SYSCALL msg=audit(1453708546.392:589): arch=c000003e syscall=2 success=yes exit=3 a0=7f4332c6b240 a1=a0000 a2=0 a3=7f4332c6b030 items=1 ppid=18618 pid=18624 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="radosgw" exe="/usr/bin/radosgw" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1453708546.392:589): avc:  denied  { open } for  pid=18624 comm="radosgw" path="/dev/shm/lttng-ust-wait-5-0" dev="tmpfs" ino=30913 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file

are fixed by backporting lttng-tools policy to RHEL. 
Remaining AVCs are issues in ceph policy, 
specifically labeling issue. Who will fix this? selinux team or ceph team?
Comment 36 Boris Ranto 2016-06-29 07:16:52 EDT
Great, back-port the lttng-tools policy to RHEL then, please.

I (ceph team) will handle the rest of the denials. Some, if not all, of the remaining denials should have already been fixed upstream.
Comment 37 Lukas Vrabec 2016-06-29 07:18:45 EDT
OK, 
moving to POST state.
Comment 49 errata-xmlrpc 2016-11-03 22:20:51 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2283.html

Note You need to log in before you can comment on or make changes to this bug.