Bug 1657158

Summary: In ABRT + systemd-coredump environment, "sysctl --system" disables abrt-hook-ccpp
Product: Red Hat Enterprise Linux 8 Reporter: Kazuo Moriwaka <kmoriwak>
Component: redhat-releaseAssignee: Djordje Todorovic <dtodorov>
Status: CLOSED ERRATA QA Contact: Release Test Team <release-test-team-automation>
Severity: low Docs Contact:
Priority: high    
Version: 8.0CC: abrt-devel-list, dtodorov, geovannisantosgmc, jwright, lisas, msobczyk, pkotvan, pzatko, sgoveas, stalexan
Target Milestone: rcFlags: pm-rhel: mirror+
Target Release: 8.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-04 01:46:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1812825, 1812830, 1812833, 1842946    

Description Kazuo Moriwaka 2018-12-07 09:51:12 UTC
Description of problem:

In RHEL 8 Beta, abrt-hook-ccpp is disabled by sysctl reapplying.

Version-Release number of selected component (if applicable):

abrt-2.10.9-7.el8.x86_64
systemd-239-8.el8.x86_64
tuned-2.10.0-7.el8.noarch


How reproducible:

Mostly (I tried 3 times and reproduce in all times, but it may depend on timing)

Steps to Reproduce:
1. enable systemd-coredump, abrt, and tuned
2. reboot system
3. check /proc/sys/kernel/core_pattern  

Actual results:

|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

Expected results:

|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e 


Additional info:

I found a candidate reason for this symptoms. It's tuned.
Restarting tuned cause overwrite of core_pattern

# abrt-install-ccpp-hook install 
# cat /proc/sys/kernel/core_pattern 
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e 
# systemctl stop tuned 
# systemctl start tuned 
# cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

I don't check tuned's details, but any programs which call "sysctl --system" can cause same issue. See following log:

# abrt-install-ccpp-hook install
# cat /proc/sys/kernel/core_pattern 
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e
# sysctl --system
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-coredump.conf ...
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.promote_secondaries = 1
net.core.default_qdisc = fq_codel
net.ipv4.tcp_ecn = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ...
net.core.optmem_max = 81920
* Applying /usr/lib/sysctl.d/60-libvirtd.conf ...
fs.aio-max-nr = 1048576
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...
# cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e


/usr/lib/sysctl.d/50-coredump.conf set up core_pattern for systemd-coredump, and this break abrt-hook-ccpp setting.

I propose a solution to extend abrt-install-ccpp-hook.
- "abrt-install-ccpp-hook install" put /etc/sysctl.d/60-abrt-coredump.conf 
- "abrt-install-ccpp-hook uninstall" delete /etc/sysctl.d/60-abrt-coredump.conf

Comment 3 ekulik 2020-02-19 13:57:11 UTC
This is a bit late, but what is the actual issue? Do you *want* to use abrt-hook-ccpp?

Comment 4 Kazuo Moriwaka 2020-02-20 00:54:43 UTC
At first, I just want to use abrt-ccpp.  And I was very confused by this conflict.
This could be fixed by KCS solution about a workaround.

After I understand what was happen, I wanted is to improve the user experience. 
Configure enable/disable abrt-ccpp by just one abrt-install-ccpp-hook command.

My proposed solution is for later one.

Thanks,

Comment 5 ekulik 2020-02-20 08:40:15 UTC
Okay, I just wanted to make sure whether you simply wanted ABRT to work or to use abrt-hook-ccpp specifically, since ABRT can also monitor the system journal and pick up crashes from there.

Comment 6 ekulik 2020-02-24 09:00:49 UTC
I’m going to do it as you suggested, but the recommendation is to move to the abrt-journal-core service.

(And for some reason the link to the knowledge base article doesn’t work)

Comment 7 ekulik 2020-02-24 12:54:25 UTC
Right, so sysctl configs cannot have arbitrary commands, so this is not going to work at all. Moreover, it would unconditionally set kernel.core_pattern, which is not what you would want if you disable abrt-ccpp.service or enable abrt-journal-core.service.

Comment 8 ekulik 2020-02-24 13:16:29 UTC
What could be a solution, though, is changing the way abrt-install-ccpp-hook works: instead of poking /proc/sys/kernel/core_pattern directly, we install a sysctl configuration file with contents like such:

#kernel.core_pattern=|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e

And the install/uninstall commands would comment and uncomment the line accordingly. No need to stash the previous core_pattern value, either.

Comment 10 ekulik 2020-02-26 13:48:21 UTC
Reassigning to redhat-release. Can we please have abrt-journal-core.service in the preset instead of abrt-ccpp.service? That was the change implemented in Fedora and it works better with systemd. That would take care of new installations, but we can also add some release notes to document the situation.

Should anyone actually want to continue using abrt-hook-ccpp, then we will consider fixing this issue the proposed way.

Comment 12 ekulik 2020-03-13 17:02:23 UTC
The Jira issue has a label set. Yes, 8.3.0.

Comment 13 Marcin Sobczyk 2020-03-16 12:55:20 UTC
> Should anyone actually want to continue using abrt-hook-ccpp, then we will
> consider fixing this issue the proposed way.

'abrt-hook-ccpp' is still relevant for oVirt/RHV use case - currently the 'systemd-coredump'
approach works in a way that you end up with 2 copies of the same core dump which is very
problematic for VMs since the core files they generate are very big. It would be very nice
to have this issue resolved and to prioritize a feature that avoids the duplication when
using 'systemd-coredump'.

Comment 14 Petr Zatko 2020-05-15 10:15:32 UTC
Hello Ernestas, it seems that swaping abrt-journal-core.service in the preset instead of abrt-ccpp.service would have some possible impact on some teams considering comment 13. 
From qe point of view I would grant qa_ack for the solutions as it is covered by automated test, but I would like to know whether the change is really intended or if tou consider fixing this issue the proposed way.

Comment 15 ekulik 2020-05-15 10:24:38 UTC
(In reply to Petr Zatko from comment #14)
> Hello Ernestas, it seems that swaping abrt-journal-core.service in the
> preset instead of abrt-ccpp.service would have some possible impact on some
> teams considering comment 13. 
> From qe point of view I would grant qa_ack for the solutions as it is
> covered by automated test, but I would like to know whether the change is
> really intended or if tou consider fixing this issue the proposed way.

The above was discussed in private already and the impact is minimal, because the ability to use our custom hook for core dump analysis is still there. The proposed solution is still pending implementation upstream and is going to be more invasive, so there is greater potential for breakage should we decide to backport it, but it’s still an open question.

The agreement in the team was that this should be fine for new users if it is documented in the release notes. For most cases, it does not matter, because we have solutions in place to limit the impact of large core dumps being copied. The cases where it does matter are specialized (usually it’s a developer dealing with specific technologies, like WebKit).

Comment 16 ekulik 2020-05-15 10:26:24 UTC
In fact, my understanding is that switching the preset should have been done prior to RHEL 8.0, as was done in the Fedora version it’s based on.

Comment 17 Petr Zatko 2020-05-15 11:47:52 UTC
Thank you for the clarification, granting qa_ack and updating our test to reflect that.

Comment 25 errata-xmlrpc 2020-11-04 01:46:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-release bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4495

Comment 27 Red Hat Bugzilla 2023-09-15 01:27:58 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days