This service will be undergoing maintenance at 00:00 UTC, 2017-10-23 It is expected to last about 30 minutes
Bug 1309172 - 'ulimit -c' is unlimited by default
'ulimit -c' is unlimited by default
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: systemd (Show other bugs)
24
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: systemd-maint
Fedora Extras Quality Assurance
:
Depends On: 1341829
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-16 23:40 EST by Sebastian Keller
Modified: 2016-08-11 17:54 EDT (History)
26 users (show)

See Also:
Fixed In Version: systemd-229-12.fc24
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-08-11 17:54:05 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Sebastian Keller 2016-02-16 23:40:43 EST
I'm on rawhide with abrt 2.8.0-2 and every time when a program crashes a file called core.PID (where PID is the pid of the crashed program) is created in the PWD of the crashed program. This usually means the home directory but it is also affecting system daemons such as gdm. This machine for example has several core.PID files in /var/lib/gdm.

I've checked if it is abrt that is handling those crashes using this command:
$ cat /proc/sys/kernel/core_pattern
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I

Here are the journal entries from one such crash:
Feb 16 12:23:54 t420 audit[770]: ANOM_ABEND auid=4294967295 uid=1000 gid=1000 ses=4294967295 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=770 comm="nautilus" exe="/usr/bin/nautilus" sig=11
Feb 16 12:23:54 t420 kernel: traps: nautilus[770] general protection ip:7fd286898355 sp:7fd2637fd9f8 error:0 in libglib-2.0.so.0.4705.0[7fd28680a000+10d000]
Feb 16 12:23:55 t420 abrt-hook-ccpp[1463]: Process 739 (nautilus) of user 1000 killed by SIGSEGV - dumping core
Feb 16 12:23:57 t420 abrt-hook-ccpp[1463]: Failed to create core_backtrace: waitpid failed: No child processes
Feb 16 12:24:13 t420 abrt-server[1464]: Deleting problem directory ccpp-2016-02-16-12:23:55-739 (dup of ccpp-2016-02-15-08:09:19-15481)
Feb 16 12:24:14 t420 dbus[687]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Feb 16 12:24:14 t420 dbus[687]: [system] Successfully activated service 'org.freedesktop.problems'
Comment 1 Jakub Filak 2016-02-17 01:05:35 EST
Thank you for the report. I cannot reproduce the problem you have. I double checked abrt source code to make sure that the core.PID is produced only if 'ulimit -c' is not 0 and MakeCompatCore=yes in the /etc/abrt/plugins/CCpp.conf file [1]. If I set MakeCompatCore=no or 'ulimit -c 0', then no core.PID file is produced. My 'ulimit -c' is 0 by default and MakeCompatCore=yes by default.

Do you have 'ulimit -c' = 0?

1: https://github.com/abrt/abrt/blob/master/src/hooks/abrt-hook-ccpp.c#L762
Comment 2 Jakub Filak 2016-02-17 02:12:56 EST
$ uname -r; ulimit -c                                                            
4.5.0-0.rc3.git1.2.fc24.x86_64                                                   
0                                                                                
                                                                                 
$ uname -r; ulimit -c
4.5.0-0.rc4.git0.1.fc24.x86_64
unlimited

I set Priority and Severity to High because this issue causes creation of core files all around user file systems.
Comment 3 Sebastian Keller 2016-02-17 02:22:02 EST
I'm running the 4.3.5-300.fc23.x86_64 kernel on rawhide, and I'm seeing "unlimited" here too. So I would assume the stable release might be affected as well.
Comment 4 Sebastian Keller 2016-02-17 03:36:18 EST
The more I look at it, the more it seems like something systemd related. I've installed kernel 4.5.0-0.rc3.git1.1.fc24.x86_64 now which is older than the one you were still getting 0 for and I'm getting "unlimited".

Then I've tried to downgrade systemd from 229-1 to 228-10. Reboot, still "unlimited".

Then I ran "dracut -f" to regenerate the initramfs with the downgraded systemd, rebooted and then "ulimit -c" finally was back to 0.

I've also tried the same kernel with systemd 228-10 in the initramfs and 229-1 installed on the system which resulted in another "unlimited". Seems like both have to be 228-10.
Comment 5 Jakub Filak 2016-02-17 04:10:18 EST
Indeed, I found this in systemd NEWS [1]:

* The RLIMIT_CORE resource limit now defaults to "unlimited" for PID 1
  and all forked processes by default. ...

1: https://github.com/systemd/systemd/blob/2c45295e47642812ea417efba7f2f3339c6e2912/NEWS

Systemd maintainers, can you please disable this feature for Fedora? It confuses ABRT which creates a core file as it could be done if ABRT would not be installed.
Comment 6 Jan Synacek 2016-02-17 05:10:31 EST
https://github.com/systemd/systemd/commit/15a900327aba7dc4dc886affe1ae22d3b759b193

According to the log message, it sounds like ABRT logic was taken into account.

https://github.com/systemd/systemd/issues/2643
Comment 7 Jóhann B. Guðmundsson 2016-02-17 05:46:46 EST
(In reply to Jakub Filak from comment #5)
> Indeed, I found this in systemd NEWS [1]:
> 
> * The RLIMIT_CORE resource limit now defaults to "unlimited" for PID 1
>   and all forked processes by default. ...
> 
> 1:
> https://github.com/systemd/systemd/blob/
> 2c45295e47642812ea417efba7f2f3339c6e2912/NEWS
> 
> Systemd maintainers, can you please disable this feature for Fedora? It
> confuses ABRT which creates a core file as it could be done if ABRT would
> not be installed.

Not everybody want or need abrt installed on their system or have it as part of their products so upstream systemd should be the default in the distribution and ABRT should make whatever changes it needs if installed ( and revert those if uninstalled ). 

And Jan whats the point of that upstream report when it's clear the systemd-coredump is not being used in the kernel core patter but some abrt hook  ( |/usr/libexec/abrt-hook-ccpp ) instead?
Comment 8 Jakub Filak 2016-02-17 06:18:19 EST
(In reply to Jóhann B. Guðmundsson from comment #7)
> (In reply to Jakub Filak from comment #5)
> > Indeed, I found this in systemd NEWS [1]:
> > 
> > * The RLIMIT_CORE resource limit now defaults to "unlimited" for PID 1
> >   and all forked processes by default. ...
> > 
> > 1:
> > https://github.com/systemd/systemd/blob/
> > 2c45295e47642812ea417efba7f2f3339c6e2912/NEWS
> > 
> > Systemd maintainers, can you please disable this feature for Fedora? It
> > confuses ABRT which creates a core file as it could be done if ABRT would
> > not be installed.
> 
> Not everybody want or need abrt installed on their system or have it as part
> of their products so upstream systemd should be the default in the
> distribution and ABRT should make whatever changes it needs if installed (
> and revert those if uninstalled ). 
> 

Agreed, I didn't want systemd to drop it, just to disable it for Fedora. However, it looks like that we can reset RTLIMIT_CORE to 0 via system.conf  (comment #6) and that is what I was asking for. I just need to figure out how to do that from the abrt services.
Comment 9 Lennart Poettering 2016-02-17 06:29:28 EST
Ahum, it appears as if abrt writes those coredumps ignoring RLIMIT_CORE.

I am pretty sure systemd should not be changed, but abrt should honour RLIMIT_CORE and not write coredumps everywhere.
Comment 10 Lennart Poettering 2016-02-17 06:32:19 EST
Also, if core_pattern is used to turn off coredumping in working directories, then abrt really shouldn't do that anyway. The coredumping in working directories is just wrong in pretty much all cases.
Comment 11 Lennart Poettering 2016-02-17 06:35:38 EST
And no, turning this off in systemd is really wrong, as not everybody uses abrt, and it's really only abrt that causes this. People who do no crash dumper at all, or use systemd-coredumper won't get any coredumps littered anywhere.

Please, make sure abrt honours RLIMIT_CORE. The kernel explicitly has provisions for core pattern hooks to handle RLIMIT_CORE via "%c".
Comment 12 Jakub Filak 2016-02-17 09:24:42 EST
(In reply to Lennart Poettering from comment #9)
> Ahum, it appears as if abrt writes those coredumps ignoring RLIMIT_CORE.

I thought that RLIMIT_CORE is passed by kernel to a core_pattern helper through %c argument. If this argument is non-zero, the ABRT core_pattern helper writes a core file as it could be done if ABRT would not be installed. Requested by our users - bug #541707.

> 
> I am pretty sure systemd should not be changed, but abrt should honour
> RLIMIT_CORE and not write coredumps everywhere.

Of course, I do not want to change upstream systemd, I just want to restore the old behavior if ABRT is employed.

I believe ABRT honor RLIMIT_CORE.

> Also, if core_pattern is used to turn off coredumping in working directories,
> then abrt really shouldn't do that anyway.

I have never thought of core_pattern as a tool to turn off coredumping. I was always turning coredumping on/off by changing 'ulimit -c' and "man 5 core" does not say that one should use core_pattern to disable coredumping.

> The coredumping in working directories is just wrong in pretty much all cases.

I can understand this but I have different opinion. ABRT writes a core file if and only if '%c' argument passed by kernel is non zero. Prior systemd 229, the most common use case was that user issued 'ulimit -c unlimited'.

> And no, turning this off in systemd is really wrong

I wanted to turn this off only in Fedora systemd, but it was too demanding. We just need to turn it off only if ABRT is employed.

> , as not everybody uses abrt

Sure, I know, so it would be the best to find a way how to make the default RTLIMIT_CORE=0 only if ABRT is employed.

> , and it's really only abrt that causes this.

I think there is no need to blame anybody.

> People who do no crash dumper at all, or use systemd-coredumper won't get any
> coredumps littered anywhere.

People with systemd >= 229 and without systemd-coredumps will have "core" in kernel.core_pattern and "ulimit -c"=unlimited, so core files will be crated in working directories.

> Please, make sure abrt honours RLIMIT_CORE. The kernel explicitly has provisions for core pattern hooks to handle RLIMIT_CORE via "%c".

ABRT honours RLIMIT_CORE. 

# cat /proc/sys/kernel/core_pattern 
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I

$ grep argv\\[2\\] src/hooks/abrt-hook-ccpp.c 
     * argv[2] = "CORE_SIZE_LIMIT PID ..."
    off_t ulimit_c = strtoull(argv[2], NULL, 10);


Is there a way how to restore RLIMIT_CORE=0 only if ABRT is employed?
Comment 13 Jan Kurik 2016-02-24 10:47:20 EST
This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle.
Changing version to '24'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase
Comment 14 Tom Horsley 2016-08-02 15:12:33 EDT
I go to a lot of trouble to disable all the abrt and core pattern nonsense on my system, but now, on fedora 24, I'm getting core files spewing all over the place.

It was previously the default to have the "soft" limit for core files set to zero so users could get a core file if they wanted one by setting "ulimit -c unlimited".

I want it to work that way again.

What can I do to make the default soft limit be zero?
Comment 15 Tom Horsley 2016-08-02 15:55:20 EDT
(In reply to Tom Horsley from comment #14)

> What can I do to make the default soft limit be zero?

OK, I poked around quite a lot and finally found DefaultLimitCORE=
in /etc/systemd/system.conf. Setting it to

DefaultLimitCORE=0:infinity

seems to restore the default behavior linux has had for the last several millenium (after a reboot).
Comment 16 Michael Catanzaro 2016-08-07 19:38:51 EDT
(In reply to Jakub Filak from comment #12)
> Is there a way how to restore RLIMIT_CORE=0 only if ABRT is employed?

The abrt-addon-ccpp package could drop a config file into /usr/lib/systemd/system.conf.d to set the limit to 0.

(It should go in the subpackage so that we can drop it from comps to get the unlimited limit without removing the rest of ABRT if we decide to get coredumpctl can work as expected out-of-the-box in the future.)
Comment 17 Zbigniew Jędrzejewski-Szmek 2016-08-07 20:12:07 EDT
(In reply to Michael Catanzaro from comment #16)
> (In reply to Jakub Filak from comment #12)
> > Is there a way how to restore RLIMIT_CORE=0 only if ABRT is employed?
> 
> The abrt-addon-ccpp package could drop a config file into
> /usr/lib/systemd/system.conf.d to set the limit to 0.

No, please don't. This would change the default for everyone who has abrt package installed, even if they disable abrt.

> it would be the best to find a way how to make the default RTLIMIT_CORE=0 only if ABRT is employed.

This similarly is impossible / inadvisable. Processes get started during boot with various RLIMIT_CORE values, and abrt starts later during the boot process, and we cannot go back and change the limits on those processes. Also it's perfectly legal to change kernel.core_pattern at runtime, e.g. by stopping abrt, and with this proposal, RLIMIT_CORE would somehow have to be reverted to previous values.

Instead, abrt should take %c value into account and not create the coredump in that case, comment #c11.
Comment 18 Michael Catanzaro 2016-08-07 21:49:12 EDT
(In reply to Zbigniew Jędrzejewski-Szmek from comment #17)
> Instead, abrt should take %c value into account and not create the coredump
> in that case, comment #c11.

So to be clear: per comment #12, it already does. That's the reason we have an issue here. Positive %c tells ABRT that the user wants a core dump file left in the cwd (exactly how traditional coredump creation works, same problem exists if you uninstall ABRT as Tom from comment #14 has done!), and with this systemd change that causes ABRT to dump core files to be dumped all over the disk, unless MakeCompatCore is set to no in /etc/abrt/plugins/CCpp.conf. The ABRT developers changed MakeCompatCore to default to no just to work around this bug.

(In reply to Zbigniew Jędrzejewski-Szmek from comment #17)
> This similarly is impossible / inadvisable. Processes get started during
> boot with various RLIMIT_CORE values, and abrt starts later during the boot
> process, and we cannot go back and change the limits on those processes.
> Also it's perfectly legal to change kernel.core_pattern at runtime, e.g. by
> stopping abrt, and with this proposal, RLIMIT_CORE would somehow have to be
> reverted to previous values.

Good point.

I can think of two ways out of this: (a) coredumpctl could go back to ignoring ulimit, and we turn MakeCompatCore back on, and systemd developers are sad; or (b) do nothing, close this bug WONTFIX, and give up on ever having ABRT's MakeCompatCore feature on by default again.

I prefer (b) as a long term solution. MakeCompatCore requires ABRT's CCpp plugin, which conflicts with having coredumpctl functional, functional coredumpctl would be a much better default, and users who want the old-style cwd cores and choose to manually turn on MakeCompatCore (or disable ABRT) can simply adjust RLIMIT_CORE at the same time.

However, in the short term, coredumpctl is not enabled by default, the CCpp plugin is. coredumpctl is totally borked in F24 anyway, so we couldn't enable it even if we wanted to (bug #1341829, bug #1317927). And with MakeCompatCore off, users have no way to get core dumps asides from learning how to use Fedora-specific abrt-cli or messing with system services, which is a serious problem. (I'm really confused why we haven't had more complaints about this, beyond this bug?) Until we can get coredumpctl functioning out-of-the-box (that means disabling abrt-ccpp.service!), we seem to be stuck with option (a).

Did I miss any options...?
Comment 19 Jakub Filak 2016-08-08 00:44:50 EDT
> Instead, abrt should take %c value into account and not create the coredump in that case, comment #c11.

As explained in comment #12, ABRT takes %c value into account but interprets it in a different way than systemd-coredump.
 

If the reason why systemd has changed the default RLIMIT_CORE to 'unliminted' is to allow users to be able to turn off coredumping by setting the option to '0', I propose to revert the systemd change and tell those users to set ulimit to 1. The 1 value tells kernel to abort coredumping in case of the "|/" string in core_pattern [1] and the value cannot meet the minimal coreddump size limit of any binary format [2]. I believe that 90% of users do not need something more than to be able to turn coredumping on or off.

1: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/coredump.c#n602
2: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/fs/coredump.c#n660
Comment 20 Zbigniew Jędrzejewski-Szmek 2016-08-09 00:28:54 EDT
(In reply to Michael Catanzaro from comment #18)
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #17)
> > Instead, abrt should take %c value into account and not create the coredump
> > in that case, comment #c11.
> 
> So to be clear: per comment #12, it already does. That's the reason we have
> an issue here. Positive %c tells ABRT that the user wants a core dump file
> left in the cwd (exactly how traditional coredump creation works, same
> problem exists if you uninstall ABRT as Tom from comment #14 has done!), and
> with this systemd change that causes ABRT to dump core files to be dumped
> all over the disk, unless MakeCompatCore is set to no in
> /etc/abrt/plugins/CCpp.conf. The ABRT developers changed MakeCompatCore to
> default to no just to work around this bug.
> 
> (In reply to Zbigniew Jędrzejewski-Szmek from comment #17)
> > This similarly is impossible / inadvisable. Processes get started during
> > boot with various RLIMIT_CORE values, and abrt starts later during the boot
> > process, and we cannot go back and change the limits on those processes.
> > Also it's perfectly legal to change kernel.core_pattern at runtime, e.g. by
> > stopping abrt, and with this proposal, RLIMIT_CORE would somehow have to be
> > reverted to previous values.
> 
> Good point.
> 
> I can think of two ways out of this: (a) coredumpctl could go back to
> ignoring ulimit, and we turn MakeCompatCore back on, and systemd developers
> are sad; or (b) do nothing, close this bug WONTFIX, and give up on ever
> having ABRT's MakeCompatCore feature on by default again.
> 
> I prefer (b) as a long term solution. MakeCompatCore requires ABRT's CCpp
> plugin, which conflicts with having coredumpctl functional, functional
> coredumpctl would be a much better default, and users who want the old-style
> cwd cores and choose to manually turn on MakeCompatCore (or disable ABRT)
> can simply adjust RLIMIT_CORE at the same time.

For 24, let's see if we can fix the selinux issue. If this doesn't happen quickly.
I'll do a) for F24, i.e. patch systemd to ignore RLIMIT_CORE. Long term,
I vote for b) too.

> (I'm really confused why we haven't had more complaints about this, beyond this bug?)

Things work with selinux permissive or off. This is especially common on development
systems where you needs coredumps the most.

(In reply to Jakub Filak from comment #19)
> > Instead, abrt should take %c value into account and not create the coredump in that case, comment #c11.
> 
> As explained in comment #12, ABRT takes %c value into account but interprets
> it in a different way than systemd-coredump.
>  
> 
> If the reason why systemd has changed the default RLIMIT_CORE to
> 'unliminted' is to allow users to be able to turn off coredumping by setting
> the option to '0', I propose to revert the systemd change and tell those
> users to set ulimit to 1. The 1 value tells kernel to abort coredumping in
> case of the "|/" string in core_pattern [1] and the value cannot meet the
> minimal coreddump size limit of any binary format [2]. I believe that 90% of
> users do not need something more than to be able to turn coredumping on or
> off.

MakeCompatCore was added to solve the issue of coredumps for unpackaged programs
being lost (#541707 that you linked). But systemd-coredump + coredumpctl is a
much nicer solution to this problem: we get corefiles, we get nice logs
including a traceback, users have access only to their core files, we don't
get corefiles in random directories, coredumps are subject to automatic cleanup.
I don't think there's much point in trying to "rescue" MakeCompatCore.

Letting '1' mean disable, and '0' mean enable (with some potentially disruptive
behaviour) is bad user UI.

Thank you for the links to the kernel source code. Current systemd behaviour of
trying to dump the core with RLIMIT=1 does not make much sense, since we fail anyway.
I filed [1] to raise the limit.

[1] https://github.com/systemd/systemd/pull/3932


OK, to summarize, currently I think 1341829 needs to be solved. If this happens,
this can be closed as NOTABUG, otherwise, I'll patch out the RLIMIT_CORE check for F24.
Comment 21 Michael Catanzaro 2016-08-09 09:47:04 EDT
(In reply to Zbigniew Jędrzejewski-Szmek from comment #20)
> For 24, let's see if we can fix the selinux issue. If this doesn't happen
> quickly.
> I'll do a) for F24, i.e. patch systemd to ignore RLIMIT_CORE. Long term,
> I vote for b) too.

I think we should do (a) in F24 regardless. What's important to me is that users have some way to get core dumps WITHOUT mucking with system services. The only way that will ever be possible in F24 is to reenable ABRT's MakeCompatCore setting. We're definitely not going to change our vendor presets in F24, so abrt-ccpp will remain enabled by default and coredumpctl will not work, regardless of whether bug #1341829 is resolved, unless the user knows which Fedora-specific ABRT service to disable.

Arguably it's too late to try squeezing a preset change into F25 as well, as the deadline for F25 features has come and gone, so coredumpctl by default should be an F26 feature IMO. I really do want to make this happen for all the reasons you mention (once those SELinux bugs get fixed), but for future Fedoras. Let's do it properly, advertise the change as a nice feature, and restore the historic behavior in the meantime.
Comment 22 Fedora Update System 2016-08-09 13:03:18 EDT
systemd-229-11.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-10e709e870
Comment 23 Fedora Update System 2016-08-10 08:25:04 EDT
systemd-229-11.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-10e709e870
Comment 24 Fedora Update System 2016-08-10 11:21:09 EDT
systemd-229-12.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-10e709e870
Comment 25 Fedora Update System 2016-08-10 23:24:49 EDT
systemd-229-12.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-10e709e870
Comment 26 Fedora Update System 2016-08-11 17:53:45 EDT
systemd-229-12.fc24 has been pushed to the Fedora 24 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.