Bug 1223319

Summary: librados2 1:0.94.1-2.fc23.x86_64 causes librados2 linked subjects to maintain world-writable predictable-named /dev/shm/lttng-ust-wait-5
Product: [Fedora] Fedora Reporter: dac.override
Component: cephAssignee: Boris Ranto <branto>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: autarch, branto, crobinso, david, devin, fedora, jdurgin, jfrieben, kdreyer, loic, marianne, mikhail.v.gavrilov, nathanael, quintela, steve
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-0.94.1-3.fc22 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-04 20:19:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186650    

Description dac.override 2015-05-20 10:26:06 UTC
Description of problem:
qemu started to maintain a world writable file with a predictable name in /dev/shm (lttng-ust-wait-5)

Version-Release number of selected component (if applicable):

    Upgraded librados2-1:0.94.1-1.fc23.x86_64                         @rawhide
    Upgrade            1:0.94.1-2.fc23.x86_64                         @rawhide

How reproducible:

update to 1:0.94.1-2.fc23.x86_64 , restart your qemu guests

Steps to Reproduce:
1.
2.
3.

Actual results:

denied  { create } for  pid=1457 comm="qemu-system-x86" name="lttng-ust-wait-5" scontext=system_u:system_r:fedtest1_subject_t tcontext=system_u:object_r:tmpfs_fs_t tclass=file permissive=1

libust[3436/3436]: Error: Error cancelling global ust listener thread: No such process (in lttng_ust_exit() at lttng-ust-comm.c:1592)

libust[3433/3433]: Warning: HOME environment variable not set. Disabling LTTng-UST per-user tracing. (in setup_local_apps() at lttng-ust-comm.c:375)
ls -alZ /dev/shm
total 4
drwxrwxrwt.  2 root root system_u:object_r:tmpfs_fs_t                60 May 20 10:53 .
drwxr-xr-x. 22 root root system_u:object_r:devtmpfs_fs_t           3780 May 20 10:36 ..
-rw-rw-rw-.  1 root root system_u:object_r:fedtest1_tmpfs_object_t 4096 May 20 10:53 lttng-ust-wait-5

Comment 1 Boris Ranto 2015-05-20 18:24:03 UTC
I guess I could compile Ceph without lttng-ust support but I need to investigate more on this to see if this is indeed a buggy behaviour.

btw: Are you sure this happened on update from -1 to -2? I really doubt that as the two releases differ only in a very tiny patch that does not touch the lttng-related code and is non-x86_64-specific.

Comment 2 Boris Ranto 2015-05-20 18:27:47 UTC
btw2: Judging from the discussion, here

https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1432644

this is somewhat expected (you can either have this behaviour or the one that is in ubuntu at the moment). I guess I could disable lttng-ust trace points support for fedora builds but I'll need to search a bit more to make a more qualified decision.

Comment 3 dac.override 2015-05-20 18:33:55 UTC
I am not 100% sure but pretty sure nonetheless.

I pretty much update rawhide daily (although in this case i skipped a few days due to a kernel bug in rawhide). Anyhow after each update i restart my qemu guests.

This only started today, and today i updated to latest librados2. So even though i am not willing to bet my life on it, i am pretty sure.

A minimal qemu implementation shouldnt be doing this i hope? Also the messages i pasted above also signal that it would'nt work anyway?

Comment 4 Boris Ranto 2015-05-21 08:49:22 UTC
Yeah, the AVC denial message should in fact stop qemu from creating the file but you seem to be running in Permissive mode so it does not. Did you by any chance set SELinux to Permissive mode between the librados updates? That would explain the change in the behaviour. (but the bug was there all along)

Apparently, other people are running into similar problems:

https://bugzilla.redhat.com/show_bug.cgi?id=1190461

Judging from these reports (this bz, launchpad, the other bz), the lttng-ust support currently looks experimental to me and I'll disable it for now (judging from Sage's comment in the launchpad, it is also the solution they chose for ubuntu).

Comment 5 dac.override 2015-05-21 08:53:21 UTC
Good observation. You are right about permissive mode.

That, however, does not change the fact that this behavior is probably unwanted and likely due to experimental code.

I agree that disabling this functionality for now is probably the best course of action.

Comment 6 Fedora Update System 2015-05-21 12:42:39 UTC
ceph-0.94.1-3.fc22 has been submitted as an update for Fedora 22.
https://admin.fedoraproject.org/updates/FEDORA-2015-8189/ceph-0.94.1-3.fc22

Comment 7 Boris Ranto 2015-05-21 12:45:33 UTC
I've submitted f22 update that should include patches for this. I've also pushed the change to f23 (rawhide):

http://koji.fedoraproject.org/koji/taskinfo?taskID=9814905

Feel free to test to see if it fixes your issue.

Comment 8 dac.override 2015-05-21 12:51:27 UTC
Yes, looks like that fixed the issue.

May 21 14:49:21 d30 systemd[1]: Stopping FedoraTest1 QEMU guest...
May 21 14:49:23 d30 kernel: br0: port 1(tap0) entered disabled state
May 21 14:49:23 d30 systemd-networkd[989]: tap0            : lost carrier
May 21 14:49:23 d30 systemd-timesyncd[979]: Network configuration changed, trying to establish connection.
May 21 14:49:23 d30 systemd[1]: Stopped FedoraTest1 QEMU guest.
May 21 14:49:23 d30 kernel: audit_printk_skb: 3 callbacks suppressed
May 21 14:49:23 d30 kernel: audit: type=1131 audit(1432212563.391:3365): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:syst
em_r:systemd_t msg='unit=fedoratest1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 21 14:49:23 d30 audit[1]: <audit-1131> pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:systemd_t msg='unit=fedor
atest1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 21 14:49:23 d30 polkitd[1419]: Unregistered Authentication Agent for unix-process:10749:10160277 (system bus name :1.144, object p
ath /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
May 21 14:49:28 d30 polkitd[1419]: Registered Authentication Agent for unix-process:10764:10161029 (system bus name :1.145 [/usr/bin/p
kttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
May 21 14:49:28 d30 systemd[1]: Starting FedoraTest1 QEMU guest...
May 21 14:49:28 d30 kernel: br0: port 1(tap0) entered forwarding state
May 21 14:49:28 d30 kernel: br0: port 1(tap0) entered forwarding state
May 21 14:49:28 d30 systemd-networkd[989]: tap0            : gained carrier
May 21 14:49:28 d30 systemd-timesyncd[979]: Network configuration changed, trying to establish connection.
May 21 14:49:28 d30 myqemu_runner_tap[10770]: char device redirected to /dev/pts/2 (label serial0)
May 21 14:49:29 d30 systemd[1]: Started FedoraTest1 QEMU guest.
May 21 14:49:29 d30 kernel: audit: type=1130 audit(1432212569.013:3366): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:syst
em_r:systemd_t msg='unit=fedoratest1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 21 14:49:29 d30 audit[1]: <audit-1130> pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:systemd_t msg='unit=fedor
atest1 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
May 21 14:49:29 d30 polkitd[1419]: Unregistered Authentication Agent for unix-process:10764:10161029 (system bus name :1.145, object p
ath /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
May 21 14:49:29 d30 kernel: kvm: zapping shadow pages for mmio generation wraparound
May 21 14:49:38 d30 kernel: kvm [10778]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
May 21 14:49:38 d30 kernel: kvm [10778]: vcpu0 unhandled rdmsr: 0x570
May 21 14:49:38 d30 kernel: kvm [10778]: vcpu1 unhandled rdmsr: 0x570
May 21 14:49:43 d30 kernel: br0: port 1(tap0) entered forwarding state

Comment 9 dac.override 2015-05-21 13:00:34 UTC
 ... thanks!

Comment 10 Boris Ranto 2015-05-21 13:02:05 UTC
No problem, thanks for pointing this out. :)

Comment 11 Fedora Update System 2015-05-22 02:31:23 UTC
Package ceph-0.94.1-3.fc22:
* should fix your issue,
* was pushed to the Fedora 22 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing ceph-0.94.1-3.fc22'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2015-8189/ceph-0.94.1-3.fc22
then log in and leave karma (feedback).

Comment 12 Cole Robinson 2015-05-25 23:45:42 UTC
Fixes selinux errors we were seeing via launching qemu VMs (which link to ceph): https://bugzilla.redhat.com/show_bug.cgi?id=1221945

FWIW in this other ubuntu bug, a dev suggests he worked around it in his code by making the tracepoints conditional on an env variable. Maybe that's an option for ceph: https://bugs.launchpad.net/ubuntu/+source/ust/+bug/1404302

Comment 13 Cole Robinson 2015-05-25 23:47:09 UTC
*** Bug 1221945 has been marked as a duplicate of this bug. ***

Comment 14 marianne@tuxette.fr 2015-05-26 10:58:21 UTC
Description of problem:
Lauching a VM 

Version-Release number of selected component:
selinux-policy-3.13.1-126.fc22.noarch

Additional info:
reporter:       libreport-2.5.1
hashmarkername: setroubleshoot
kernel:         4.0.4-300.fc22.x86_64
type:           libreport

Comment 15 Nathanael Noblet 2015-05-28 15:20:45 UTC
Description of problem:
Rebooted and received the warning. I use libvirtd/qemu/virt-manager for some vms

Version-Release number of selected component:
selinux-policy-3.13.1-126.fc22.noarch

Additional info:
reporter:       libreport-2.5.1
hashmarkername: setroubleshoot
kernel:         4.0.4-301.fc22.x86_64
type:           libreport

Comment 16 Fedora Update System 2015-06-04 20:19:51 UTC
ceph-0.94.1-3.fc22 has been pushed to the Fedora 22 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 17 Ken Dreyer (Red Hat) 2015-06-29 18:47:57 UTC
Boris do we need a new bug to track the SELinux issue?

Comment 18 Boris Ranto 2015-06-30 11:52:07 UTC
@Ken: Hmm, maybe but an upstream one. Although, I suspect/hope that the SELinux issue will get fixed once the shared device will have better permissions.

Comment 19 Ken Dreyer (Red Hat) 2015-06-30 16:31:55 UTC
@Josh, was this the bug that we'd discussed yesterday re: SELinux and lttng-ust?

Comment 20 Josh Durgin 2015-06-30 16:51:43 UTC
Yes, this is the lttng-ust issue we were talking about.