Bug 1365435

Summary: Regression: SELinux prevents systemd from setting rlimits and breaks core dumping
Product: [Fedora] Fedora Reporter: Stef Walter <stefw>
Component: selinux-policy-targetedAssignee: Lukas Vrabec <lvrabec>
Status: CLOSED DUPLICATE QA Contact: Ben Levenson <benl>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 25CC: awilliam, dwalsh, fedora, fweimer, goeran, lvrabec, mcatanzaro+wrong-account-do-not-cc, mvollmer, robatino, robert.hancock, sgallagh, vpodzime
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: RejectedBlocker
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-16 15:34:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1341829    

Description Stef Walter 2016-08-09 09:30:31 UTC
Description of problem:

SELinux prevents systemd from setting up rlimits as configured, when in enforcing mode. Among other things this prevents the systemd-coredump from working correctly.

Version-Release number of selected component (if applicable):

systemd-229-8.fc24.x86_64

How reproducible:

Every time.

Steps to Reproduce:

# setenforce 1
# systemctl show -p LimitCORE crond
LimitCORE=18446744073709551615
# systemctl stop crond
# systemctl start crond
# cat /proc/$(pgrep crond)/limits | grep core
Max core file size        0                    unlimited            bytes     
# setenforce 0
# systemctl stop crond
# systemctl start crond
# cat /proc/$(pgrep crond)/limits | grep core
Max core file size        unlimited             unlimited            bytes     

Actual results:

core file size is set to 0

Expected results:

core file size should be set to unlimited

Additional info:

This is distinct from the denied mounton SELinux issue tracked in bug #1317927

Comment 1 Stef Walter 2016-09-10 06:14:45 UTC
Same problem exists on Fedora 25.

Comment 2 Stef Walter 2016-09-10 06:16:47 UTC
This issue was discovered by the Cockpit integration tests. Here's our upstream tracker for this: https://github.com/cockpit-project/cockpit/issues/5027

Comment 3 Stef Walter 2016-11-11 11:50:53 UTC
This happens regularly in Fedora 25 and breaks core dumping on the system. This is a regression.

It prevents us from making progress debugging other issues in components like storaged.

For example here's a segfault that we didn't get a core dump for, and are unable to make progress solving:

https://fedorapeople.org/groups/cockpit/logs/pull-4565-d063ad01-verify-fedora-25/TestStorage-testLvm-10.111.120.109-FAIL.log

The reproducer for this above is simple and consistent. In addition here's another trivial consistent reproducer:

# systemctl start crond
# killall -SEGV crond
# ls /var/lib/systemd/coredump/
# setenforce 0
# systemctl start crond
# killall -SEGV crond
# ls /var/lib/systemd/coredump/
core.crond.0.f0be89cae1e34090ab645453bdd4f1c4.1348.1478864770000000000000.lz4

After "setenforce 0" it works. There are no denied messages in the journal:

# journalctl -b | grep denied
<no output>

The above system is a clean install of Fedora 25. Built on October 22nd. It was built with the following packages (ie: no ABRT):

%packages
@core
%end

In addition if you wish to have an actual image you can get it here:

https://fedorapeople.org/groups/cockpit/images/fedora-25-c6ac0dbaf59ea4ecc5b07aaa0bcd7f307a0c1e31.qcow2.xz

Once again:

# rpm -qa | grep selinux
rpm -qa | grep selinux
selinux-policy-targeted-3.13.1-220.fc25.noarch
selinux-policy-3.13.1-220.fc25.noarch
libselinux-python3-2.5-12.fc25.x86_64
rpm-plugin-selinux-4.13.0-0.rc1.46.fc25.x86_64
container-selinux-1.12.2-3.git15c82b8.fc25.x86_64
libselinux-2.5-12.fc25.x86_64
libselinux-utils-2.5-12.fc25.x86_64

# rpm -qa | grep systemd
systemd-231-10.fc25.x86_64
systemd-udev-231-10.fc25.x86_64
systemd-libs-231-10.fc25.x86_64
systemd-container-231-10.fc25.x86_64
systemd-pam-231-10.fc25.x86_64
python3-systemd-232-1.fc25.x86_64
python-systemd-doc-232-1.fc25.x86_64
rpm-plugin-systemd-inhibit-4.13.0-0.rc1.46.fc25.x86_64
oci-systemd-hook-0.1.4-4.git15c2f48.fc25.x86_64

Comment 4 Fedora Blocker Bugs Application 2016-11-11 13:13:21 UTC
Proposed as a Blocker for 25-final by Fedora user sgallagh using the blocker tracking app because:

 "All applications that can be launched using the standard graphical mechanism of a release-blocking desktop after a default installation of that desktop must start successfully and withstand a basic functionality test." (Final criterion)

It might be arguable, but I'd claim that ABRT wouldn't be withstanding the basic functionality test if SELinux is preventing coredumps from being generated and therefore displayed in ABRT.

Comment 5 Adam Williamson 2016-11-14 07:08:25 UTC
I don't think it is. We've reported dozens of F25 crashes via abrt just fine. I just tested - with the same selinux-policy you have, 220.fc25, though in fact what's going into Final is -224 - and if I pkill gedit I get an abrt crash report, including coredump, just fine.

On that basis, -1 blocker, I don't see any criterion violation here.

Comment 6 Stef Walter 2016-11-14 07:34:56 UTC
I don't think this is a blocker ... but it is hindering us from fixing real bugs in Fedora 25.

In particular this bug keeps coming up sporadically during integration testing ... we'd like to find a real fix for it. But since the integration tests (for good reason) have to run with SELinux enabled, we can't make progress.

Nov 11 16:59:34 localhost.localdomain kernel: udisksd[1390]: segfault at 40 ip 00007fbd40c2154f sp 00007ffe9327b680 error 4 in libudisks2_lvm2.so[7fbd40c02000+2b000]
Nov 11 16:59:34 localhost.localdomain kernel: audit: type=1701 audit(1478901574.843:262): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:devicekit_t:s0 pid=1390 comm="udisksd" exe="/usr/libexec/udisks2/udisksd" sig=11
Nov 11 16:59:34 localhost.localdomain audit[1390]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:devicekit_t:s0 pid=1390 comm="udisksd" exe="/usr/libexec/udisks2/udisksd" sig=11
Nov 11 16:59:35 localhost.localdomain systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Nov 11 16:59:35 localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-2132-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 11 16:59:35 localhost.localdomain systemd[1]: Started Process Core Dump (PID 2132/UID 0).
Nov 11 16:59:35 localhost.localdomain kernel: audit: type=1130 audit(1478901575.836:263): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-coredump@0-2132-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 11 16:59:35 localhost.localdomain systemd-coredump[2137]: Core Dumping has been disabled for process 1390 (udisksd).
Nov 11 16:59:35 localhost.localdomain systemd[1]: udisks2.service: Main process exited, code=dumped, status=11/SEGV
Nov 11 16:59:35 localhost.localdomain systemd[1]: udisks2.service: Unit entered failed state.

Comment 7 Lukas Vrabec 2016-11-14 09:18:50 UTC
Here is the workaround: 

# cat crond_rhlimith.cil 
(allow init_t crond_t (process (rlimitinh)))

# semodule -i crond_rhlimith.cil

I'll fix it in all supported fedoras.

Comment 8 Stef Walter 2016-11-14 09:42:51 UTC
(In reply to Lukas Vrabec from comment #7)
> Here is the workaround: 
> 
> # cat crond_rhlimith.cil 
> (allow init_t crond_t (process (rlimitinh)))

That does fix it for a segfault in the crond process. What's an equivalent line that works for any other process? I'd like to put this workaround into our integration tests.

Comment 9 Michael Catanzaro 2016-11-14 13:00:59 UTC
Isn't this a duplicate of bug #1341829? It has been broken since F24. If Lukas is going to fix it, that saves us all some effort as it's derailing my coredumpctl by default effort and we were planning to escalate this issue to FESCo. :)

(In reply to Fedora Blocker Bugs Application from comment #4)
> It might be arguable, but I'd claim that ABRT wouldn't be withstanding the
> basic functionality test if SELinux is preventing coredumps from being
> generated and therefore displayed in ABRT.

Do you have the abrt-ccpp plugin disabled on Server? If not, ABRT should still work fine. This bug is causing no issues with ABRT on Workstation. It would definitely break ABRT in combination with our desired switch to enable coredumpctl by default in F26, but we're not there yet.

Comment 10 Adam Williamson 2016-11-14 19:40:40 UTC
Discussed at 2016-11-14 blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2016-11-14/f25-blocker-review.2016-11-14-17.00.html . Rejected as a blocker on the grounds that this does not seem to violate any criteria: as discussed above, it does not in fact prevent abrt working as intended on Workstation.

Comment 11 Lukas Vrabec 2016-11-16 12:18:04 UTC
(In reply to Stef Walter from comment #8)
> (In reply to Lukas Vrabec from comment #7)
> > Here is the workaround: 
> > 
> > # cat crond_rhlimith.cil 
> > (allow init_t crond_t (process (rlimitinh)))
> 
> That does fix it for a segfault in the crond process. What's an equivalent
> line that works for any other process? I'd like to put this workaround into
> our integration tests.

Stef, 
Just change allow rule:
(allow domain domain (process (rlimitinh)))

Comment 12 Stephen Gallagher 2016-12-01 13:35:26 UTC
(In reply to Lukas Vrabec from comment #11)
> (In reply to Stef Walter from comment #8)
> > (In reply to Lukas Vrabec from comment #7)
> > > Here is the workaround: 
> > > 
> > > # cat crond_rhlimith.cil 
> > > (allow init_t crond_t (process (rlimitinh)))
> > 
> > That does fix it for a segfault in the crond process. What's an equivalent
> > line that works for any other process? I'd like to put this workaround into
> > our integration tests.
> 
> Stef, 
> Just change allow rule:
> (allow domain domain (process (rlimitinh)))

Lukas, we need the SELinux policy to be updated to work generally, because right now it prevents coredumpctl from working on Fedora, which is a very important for debugging and improving Fedora quality.

Comment 13 Lukas Vrabec 2016-12-01 14:25:50 UTC
Stephen, 
Do we need to allow this for every domain on the system? Probably we should create a boolean for this.

Comment 14 Stephen Gallagher 2016-12-01 14:33:50 UTC
(In reply to Lukas Vrabec from comment #13)
> Stephen, 
> Do we need to allow this for every domain on the system?

I think so, yes. Since any process on the system could crash and need to be captured.

> Probably we should create a boolean for this.

Doesn't make sense to me. By the time you realize you need to set the boolean, you would have already missed the crash you wanted to capture.

Comment 15 Stef Walter 2016-12-01 14:37:14 UTC
(In reply to Stephen Gallagher from comment #14)
> (In reply to Lukas Vrabec from comment #13)
> > Stephen, 
> > Do we need to allow this for every domain on the system?
> 
> I think so, yes. Since any process on the system could crash and need to be
> captured.
> 
> > Probably we should create a boolean for this.
> 
> Doesn't make sense to me. By the time you realize you need to set the
> boolean, you would have already missed the crash you wanted to capture.

I agree. The goal should be that systemd and systemd-coredump just works. Why do we need to set a boolean to unbreak this feature of systemd?

Comment 16 Michael Catanzaro 2016-12-01 15:10:47 UTC
Lukas, please do what you need to do to make the feature work as designed without any modifications to SELinux policy. If the problem is really just that SELinux is preventing systemd from raising the rlimit, then I'd say we don't need to allow every process on the system to do that..

Also: is this bug a duplicate of bug #1341829?

Comment 17 Robert Hancock 2016-12-01 23:28:42 UTC
(In reply to Michael Catanzaro from comment #9)
> (In reply to Fedora Blocker Bugs Application from comment #4)
> > It might be arguable, but I'd claim that ABRT wouldn't be withstanding the
> > basic functionality test if SELinux is preventing coredumps from being
> > generated and therefore displayed in ABRT.
> 
> Do you have the abrt-ccpp plugin disabled on Server? If not, ABRT should
> still work fine. This bug is causing no issues with ABRT on Workstation. It
> would definitely break ABRT in combination with our desired switch to enable
> coredumpctl by default in F26, but we're not there yet.

I've seen references in a few reports to the fact that systemd-coredump isn't supposed to be enabled on F25 Workstation, but it is on my system despite never taking any action to do so. Even though abrt-ccpp.service is enabled, systemd-coredump appears to set its core pattern into the kernel after ABRT does, thus overriding it, so it ends up trying (and failing) to handle the coredump instead. If I restart abrt-ccpp.service after bootup, ABRT starts working.

I don't know how systemd-coredump is supposed to be disabled, since all of the settings in systemd config files seem to be at default (rpm -V systemd shows no output). Sure looks like it's enabled by default in F25 to me, unless the installer has some magic to modify the settings (this system was upgraded several times from previous Fedora versions).

Comment 18 Michael Catanzaro 2016-12-02 00:36:06 UTC
(In reply to Robert Hancock from comment #17)
> I've seen references in a few reports to the fact that systemd-coredump
> isn't supposed to be enabled on F25 Workstation, but it is on my system
> despite never taking any action to do so. Even though abrt-ccpp.service is
> enabled, systemd-coredump appears to set its core pattern into the kernel
> after ABRT does, thus overriding it, so it ends up trying (and failing) to
> handle the coredump instead. If I restart abrt-ccpp.service after bootup,
> ABRT starts working.

It's a bug, abrt-ccpp.service should be overriding it (in F25; the plan for F26 is to disable abrt-ccpp.service). This is the first I've heard of that happening before. Please file a different bug report as that's not this issue, against abrt I guess (the developers should eventually move it to the right place).

Comment 19 Lukas Vrabec 2016-12-02 11:40:12 UTC
Fixed in F25. Moving to POST state. 

commit 76b2c6650ae485f93646650be372f694f07ae587
Author: Lukas Vrabec <lvrabec>
Date:   Fri Dec 2 12:36:16 2016 +0100

    Allow systemd to raise rlimit to all domains.BZ(1365435)

Comment 20 Fedora Update System 2016-12-05 17:01:45 UTC
selinux-policy-3.13.1-225.1.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2016-e3864b8972

Comment 21 Fedora Update System 2016-12-07 02:25:19 UTC
selinux-policy-3.13.1-225.1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-e3864b8972

Comment 22 Fedora Update System 2016-12-08 18:22:21 UTC
selinux-policy-3.13.1-225.1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 23 Göran Uddeborg 2016-12-16 22:22:56 UTC
Unless I'm missing something, 3.13.1-225.1.fc25 hasn't done anything to fix this.  If I repeat the test case in comment 0, I still get the limit 0 in enforcing mode.  Trying the command "sesearch -A -p rlimitinh -s init_t" I I only get an allow rule when transitioning to virtd_t, and one when transitioning to a type I've added in a local module.

Comment 24 Michael Catanzaro 2016-12-16 22:33:22 UTC
I guess I'll ask a third time... is this a duplicate of bug #1341829?

Comment 25 Göran Uddeborg 2016-12-16 22:37:59 UTC
If you ask me, I BELIEVE it's the same reason.  But there COULD be something more involved, and I'm not sure enough that I dare close it as a duplicate.

Comment 26 Stef Walter 2016-12-19 16:31:05 UTC
Now also a problem on Fedora Atomic since it has updated to a Fedora 25.

Comment 27 Michael Catanzaro 2017-02-16 15:34:16 UTC
OK this is obviously a duplicate of bug #1341829, I'm going to mark it accordingly.

*** This bug has been marked as a duplicate of bug 1341829 ***