CC'ing Josh.

@Josh: Do you consider this feature practical/important for customers?

Yes, we want this enabled to be able to collect workload traces.

The ptrace calls are for syscall tracing, which we aren't interested in, so we don't need to allow those. I don't think we care about tracefs either - denying that is fine.

/dev/shm/lttng-ust-wait-5 is used by the lttng cli to communicated with liblttng-ust to enabled/disable tracing, so read-only access to that is needed.

We're most interested in using this with qemu + rbd, which might need some more selinux changes.

I am afraid we cannot selectively allow something there, if lttng generates these AVCs it must be disabled in compile time then to use these.

And generating AVC denials on production system is not acceptable.

I even think that predictable name /dev/shm/lttng-ust-wait-5 could be a security problem per se. Is really compiling-in debugging features for everyone on production systems good idea here?

I still don't see too much of a use here for _end customers_. I understand that this feature might significantly help with debugging but how may a customer use it in practice? --  i.e. Is there any workload (not debugging workload) where customers might want to use this feature? How can it be used in conjunction with qemu?

I'd be ok with the feature being on in the packages if there were no security concerns related to this feature but:

1.) the feature is pretty much untested (at least on rhel 7, upstream spec file still builds ceph for rhel 7 without lttng support)

2.) based on a fedora report, it creates a world writable fixed path file which is definitely not good:
  https://bugzilla.redhat.com/show_bug.cgi?id=1223319

(In reply to Milan Broz from comment #4)
> I am afraid we cannot selectively allow something there, if lttng generates
> these AVCs it must be disabled in compile time then to use these.

Ah, that's a pain. What's the usual procedure for things like this then? Fix them/make them configurable upstream? Separate unwanted parts so they can not be packaged?

> And generating AVC denials on production system is not acceptable.

Makes sense, I was hoping there was a way to let regular permissions deny unnecessary accesses.

> I even think that predictable name /dev/shm/lttng-ust-wait-5 could be a
> security problem per se. Is really compiling-in debugging features for
> everyone on production systems good idea here?

(In reply to Boris Ranto from comment #5)
> I still don't see too much of a use here for _end customers_. I understand
> that this feature might significantly help with debugging but how may a
> customer use it in practice? --  i.e. Is there any workload (not debugging
> workload) where customers might want to use this feature? How can it be used
> in conjunction with qemu?

The use case is to enable customers to capture what their workloads actually are, for example with virtual machines using an rbd block device. Qemu links to librbd which links to liblttng-ust for tracing.

librbd has lttng tracepoints that efficiently capture an I/O workload, and
there are tools for analyzing and replaying them later, for tuning.

Rather than saying vms are for 'everything', they can sample what their
workload actually is, and tune for that instead of guessing and approximating.

> I'd be ok with the feature being on in the packages if there were no
> security concerns related to this feature but:
> 
> 1.) the feature is pretty much untested (at least on rhel 7, upstream spec
> file still builds ceph for rhel 7 without lttng support)
> 
> 2.) based on a fedora report, it creates a world writable fixed path file
> which is definitely not good:
>   https://bugzilla.redhat.com/show_bug.cgi?id=1223319

Maybe we should add an option to lttng to ignore the global and per-user 'lttng-ust-wait-5' file, and just use per-vm file (for the qemu use case) that could be properly labelled for selinux by libvirt.

LTTng is excellent tool but enterprise builds usually do not contain debugging features in stable released builds (I think even for lttng there is some small performance overhead. There can be another debug-enabled build similar to kernel-debug in RHEL).

Anyway, there are several unresolved SELinux issues related to lttng even in Fedora, so introducing this to RHEL can easily cause more problems than it solves (there is in fact no testing related to SELinux).

So the first question is if we really want to introduce it in RH build (security and supportability is the major questions here) and if so we have to find how to do it properly.

If we are interested only in UST (userspace tracing) then there probably should be some way how enable just this without messing with syscalls etc...

Mirek, can we somehow support lttng in SELinux RHEL policy in future?

(In reply to Milan Broz from comment #8)
> Mirek, can we somehow support lttng in SELinux RHEL policy in future?

Yes, we have Fedora bug for this issue but not sure if we are able to have working fix in RHEl-7.2.

Hi guys,
  LTTng will start shipping in 1.3.1, and we see value in having tracing as a customer feature out of the box. I do not want to back that out — I want the tracing problem solved in 1.3.1, and the hardened security one in 1.3.2, so to speak. One can of worms at a time methodology :)


  Given that, we need to plan any additional SELinux work to secure these. SELinux currently is targeted at 1.3.2, meaning we have at the very least until the end of October to do this, and possibly longer.

thanks -F

Enabling lttng without basic support in selinux-policy means that customers will get AVCs and I think it will not work in enforcing mode at all.

(And according to comment#9 lttng support we cannot be sure RHEL7.2 will have it. This must be coordinated with RHEL PM otherwise it will cause a lot of noise...
Updating Ceph selinux policy is not enough here.)

(In reply to Milan Broz from comment #11)
> Enabling lttng without basic support in selinux-policy means that customers
> will get AVCs and I think it will not work in enforcing mode at all.
> 
> (And according to comment#9 lttng support we cannot be sure RHEL7.2 will
> have it. This must be coordinated with RHEL PM otherwise it will cause a lot
> of noise...
> Updating Ceph selinux policy is not enough here.)

In this case, there needs to be a request (bug) from you agains selinux-policy component where we can discuss it if we are able to find a solution.

(In reply to Miroslav Grepl from comment #12)
> In this case, there needs to be a request (bug) from you agains
> selinux-policy component where we can discuss it if we are able to find a
> solution.

Milan, did you file a bug there?

I was under the impression that the decision was to not link to lttng in released version... Siddharth, do you know details here?
If not, I'll fill the bug.

We did link lttng in the released version.

Hm. ok.

I think lttng SELinux extension is not fixed yet even in Fedora, see bug #1222157.

I'll reassign this to RHEL7 selinux-policy for discussion then, local Ceph SELinux should not solve this problem anyway, it is generic issue with lttng support.

Short summary: Ceph is now linked with linked with lttng, in following release it will fully support SElinux, so users could start to see AVC as mentioned above.

We will need to depend on some global selinux policy that covers lttng issues.
(Or alternatively it can be part of Ceph selinux policy but then we need sync with RHEL selinux maintainers.)

@Milan: The SELinux changes should not be so critical that we should include them in the ceph SELinux policy. Mostly, thanks to

https://github.com/ceph/ceph/pull/6415

that should bring the library split to hammer (it is already in master) -- i.e. afaik, there should be two versions of the libraries, the default one without trace points and the trace point one which has to be enabled manually.

We should probably file a documentation bug though -- i.e. document that if someone wants to debug (use lttng-related stuff), he should disable SELinux support. At least until the SELinux policy changes for lttng-ust will make it into selinux-policy package.

(In reply to Boris Ranto from comment #20)
> @Milan: The SELinux changes should not be so critical that we should include
> them in the ceph SELinux policy. Mostly, thanks to
> 
> https://github.com/ceph/ceph/pull/6415
> 
> that should bring the library split to hammer (it is already in master) --
> i.e. afaik, there should be two versions of the libraries, the default one
> without trace points and the trace point one which has to be enabled
> manually.
> 
> We should probably file a documentation bug though -- i.e. document that if
> someone wants to debug (use lttng-related stuff), he should disable SELinux
> support. At least until the SELinux policy changes for lttng-ust will make
> it into selinux-policy package.

You can talk about permissive mode instead of SELinux disabling.

What is state of this BZ? Can I fix these AVCs and create some pull request? Could you point me to right repo? 

Thank you.

AFAIK, the idea here is that we should create (back-port?) the lttng-ust policy to RHEL 7.3.

While the package (lttng-ust) is not in base, a layered product requires this package and it stops us from fully enabling SELinux on some of the machines (currently, we need to disable SELinux for Ceph to work with the lttng trace-points -- it is a non-default behaviour used mostly for debugging but we still need to disable SELinux if we want to use that feature).

Boris, 
Am I right, (lttng-ust) will be part of selinux-policy package and also part of some of yours packages? (With changes in rhel-7.3 we can make it working, more info: https://mgrepl.wordpress.com/2015/07/31/cil-part2-module-priorities/) ? 

Or this is some package needed by ceph and we also need policy for it? Do you have some policy sources for lttng-ust? 

I'm looking on lttng-ust package and it's just library. I don't think we need policy for this. Could you attach output of 
# ps -efZ | grep unconfined_service_t

To show which processes need policy? 

Thank you.

Lukas,

you are right, lttng-ust is just a library. I got confused by Comment #9 that suggested that this was being investigated in fedora. @Miroslav: Any idea what bz were you referring to? Maybe, some domain transitions "magic" as lttng-ust always creates/manipulates the files that generate the denial on its own?

If not, then it might be the case that we just need to update the ceph policy to allow these denials.

The lttng-ust package contains some libraries, but there are also 2 daemons:

/usr/bin/lttng-relayd
/usr/bin/lttng-sessiond

# service lttng-sessiond status
● lttng-sessiond.service - LSB: The LTTng session daemon
   Loaded: loaded (/etc/rc.d/init.d/lttng-sessiond; bad; vendor preset: enabled)
   Active: inactive (dead) since Tue 2016-06-28 11:14:37 CEST; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11325 ExecStop=/etc/rc.d/init.d/lttng-sessiond stop (code=exited, status=0/SUCCESS)

Jun 28 11:13:37 rhel71.localdomain systemd[1]: Starting LSB: The LTTng sessi....
Jun 28 11:13:37 rhel71.localdomain lttng-sessiond[11056]: Starting lttng-sess...
Jun 28 11:13:37 rhel71.localdomain systemd[1]: Started LSB: The LTTng sessio....
Jun 28 11:14:37 rhel71.localdomain systemd[1]: Stopping LSB: The LTTng sessi....
Jun 28 11:14:37 rhel71.localdomain lttng-sessiond[11325]: Stopping lttng-sess...
Jun 28 11:14:37 rhel71.localdomain systemd[1]: Stopped LSB: The LTTng sessio....
Hint: Some lines were ellipsized, use -l to show in full.
# service lttng-sessiond start
Starting lttng-sessiond (via systemctl):                   [  OK  ]
# service lttng-sessiond status
● lttng-sessiond.service - LSB: The LTTng session daemon
   Loaded: loaded (/etc/rc.d/init.d/lttng-sessiond; bad; vendor preset: enabled)
   Active: active (running) since Tue 2016-06-28 11:14:44 CEST; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 11325 ExecStop=/etc/rc.d/init.d/lttng-sessiond stop (code=exited, status=0/SUCCESS)
  Process: 11450 ExecStart=/etc/rc.d/init.d/lttng-sessiond start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/lttng-sessiond.service
           └─11457 /usr/bin/lttng-sessiond -d

Jun 28 11:14:44 rhel71.localdomain systemd[1]: Starting LSB: The LTTng sessi....
Jun 28 11:14:44 rhel71.localdomain lttng-sessiond[11450]: Starting lttng-sess...
Jun 28 11:14:44 rhel71.localdomain systemd[1]: Started LSB: The LTTng sessio....
Hint: Some lines were ellipsized, use -l to show in full.
# ps -efZ | grep lttng
system_u:system_r:initrc_t:s0   root     11457     1  0 11:14 ?        00:00:00 /usr/bin/lttng-sessiond -d
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 11515 4182  0 11:14 pts/0 00:00:00 grep --color=auto lttng
# rpm -qf /usr/bin/lttng-sessiond
lttng-tools-2.4.1-1.el7cp.x86_64
# 

Unfortunately, there is no systemd unit file for the first one.

Those daemons are part of lttng-tools and there already is an upstream (fedora) policy for lttng-tools. I'm not sure how helpful it would be for our case.

Boris, 
Do we need back port lttng-tools policy module to rhel-7.3? Could you post link for repo where I can create pull request to fix above denials?

I'm not sure whether we need the policy or not. Where could I see its policy files? That might give me an idea whether it will help or not. Anyway, the ceph policy is tracked upstream [1] but you don't need to modify it, I will do the upstream PR if that is in fact what we should do here (well, I should do some local testing, first anyway).

[1] https://github.com/ceph/ceph

Hi Boris, 
Attaching lttng-tools policy:
https://github.com/fedora-selinux/selinux-policy/blob/rawhide-contrib/lttng-tools.fc
https://github.com/fedora-selinux/selinux-policy/blob/rawhide-contrib/lttng-tools.te

Let me know if I need back port it.

Yes, we need to back port it.

Following AVCs: 
time->Mon Jan 25 02:55:42 2016
type=PATH msg=audit(1453708542.827:579): item=0 name="/dev/shm/lttng-ust-wait-5" inode=30760 dev=00:11 mode=0100666 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:initrc_state_t:s0 objtype=NORMAL
type=CWD msg=audit(1453708542.827:579):  cwd="/"
type=SYSCALL msg=audit(1453708542.827:579): arch=c000003e syscall=2 success=yes exit=3 a0=7f9d673fc240 a1=a0000 a2=0 a3=7f9d673fbfc0 items=1 ppid=18554 pid=18568 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="radosgw" exe="/usr/bin/radosgw" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1453708542.827:579): avc:  denied  { open } for  pid=18568 comm="radosgw" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
type=AVC msg=audit(1453708542.827:579): avc:  denied  { read } for  pid=18568 comm="radosgw" name="lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
----
time->Mon Jan 25 02:55:46 2016
type=PATH msg=audit(1453708546.199:582): item=0 name="/dev/shm/lttng-ust-wait-5" inode=30760 dev=00:11 mode=0100666 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:initrc_state_t:s0 objtype=NORMAL
type=CWD msg=audit(1453708546.199:582):  cwd="/"
type=SYSCALL msg=audit(1453708546.199:582): arch=c000003e syscall=2 success=yes exit=3 a0=7fc9b0204240 a1=a0000 a2=0 a3=7fc9b0203fc0 items=1 ppid=18588 pid=18609 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="radosgw" exe="/usr/bin/radosgw" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1453708546.199:582): avc:  denied  { open } for  pid=18609 comm="radosgw" path="/dev/shm/lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
type=AVC msg=audit(1453708546.199:582): avc:  denied  { read } for  pid=18609 comm="radosgw" name="lttng-ust-wait-5" dev="tmpfs" ino=30760 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:initrc_state_t:s0 tclass=file
----
time->Mon Jan 25 02:55:46 2016
type=PATH msg=audit(1453708546.392:589): item=0 name="/dev/shm/lttng-ust-wait-5-0" inode=30913 dev=00:11 mode=0100640 ouid=0 ogid=0 rdev=00:00 obj=unconfined_u:object_r:user_tmp_t:s0 objtype=NORMAL
type=CWD msg=audit(1453708546.392:589):  cwd="/"
type=SYSCALL msg=audit(1453708546.392:589): arch=c000003e syscall=2 success=yes exit=3 a0=7f4332c6b240 a1=a0000 a2=0 a3=7f4332c6b030 items=1 ppid=18618 pid=18624 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="radosgw" exe="/usr/bin/radosgw" subj=system_u:system_r:ceph_t:s0 key=(null)
type=AVC msg=audit(1453708546.392:589): avc:  denied  { open } for  pid=18624 comm="radosgw" path="/dev/shm/lttng-ust-wait-5-0" dev="tmpfs" ino=30913 scontext=system_u:system_r:ceph_t:s0 tcontext=unconfined_u:object_r:user_tmp_t:s0 tclass=file

are fixed by backporting lttng-tools policy to RHEL. 
Remaining AVCs are issues in ceph policy, 
specifically labeling issue. Who will fix this? selinux team or ceph team?

Great, back-port the lttng-tools policy to RHEL then, please.

I (ceph team) will handle the rest of the denials. Some, if not all, of the remaining denials should have already been fixed upstream.

OK, 
moving to POST state.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2283.html