RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1657158 - In ABRT + systemd-coredump environment, "sysctl --system" disables abrt-hook-ccpp
Summary: In ABRT + systemd-coredump environment, "sysctl --system" disables abrt-hook-...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: redhat-release
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: rc
: 8.3
Assignee: Djordje Todorovic
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks: 1812825 1812830 1812833 1842946
TreeView+ depends on / blocked
 
Reported: 2018-12-07 09:51 UTC by Kazuo Moriwaka
Modified: 2023-09-15 01:27 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 01:46:28 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELBLD-1953 0 None None None 2022-08-03 18:34:38 UTC
Red Hat Knowledge Base (Article) 3757811 0 None None None 2018-12-15 14:13:01 UTC

Internal Links: 2076267

Description Kazuo Moriwaka 2018-12-07 09:51:12 UTC 
Description of problem:

In RHEL 8 Beta, abrt-hook-ccpp is disabled by sysctl reapplying.

Version-Release number of selected component (if applicable):

abrt-2.10.9-7.el8.x86_64
systemd-239-8.el8.x86_64
tuned-2.10.0-7.el8.noarch


How reproducible:

Mostly (I tried 3 times and reproduce in all times, but it may depend on timing)

Steps to Reproduce:
1. enable systemd-coredump, abrt, and tuned
2. reboot system
3. check /proc/sys/kernel/core_pattern  

Actual results:

|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

Expected results:

|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e 


Additional info:

I found a candidate reason for this symptoms. It's tuned.
Restarting tuned cause overwrite of core_pattern

# abrt-install-ccpp-hook install 
# cat /proc/sys/kernel/core_pattern 
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e 
# systemctl stop tuned 
# systemctl start tuned 
# cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e

I don't check tuned's details, but any programs which call "sysctl --system" can cause same issue. See following log:

# abrt-install-ccpp-hook install
# cat /proc/sys/kernel/core_pattern 
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e
# sysctl --system
* Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ...
kernel.yama.ptrace_scope = 0
* Applying /usr/lib/sysctl.d/50-coredump.conf ...
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
* Applying /usr/lib/sysctl.d/50-default.conf ...
kernel.sysrq = 16
kernel.core_uses_pid = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.promote_secondaries = 1
net.core.default_qdisc = fq_codel
net.ipv4.tcp_ecn = 1
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
* Applying /usr/lib/sysctl.d/50-libkcapi-optmem_max.conf ...
net.core.optmem_max = 81920
* Applying /usr/lib/sysctl.d/60-libvirtd.conf ...
fs.aio-max-nr = 1048576
* Applying /etc/sysctl.d/99-sysctl.conf ...
* Applying /etc/sysctl.conf ...
# cat /proc/sys/kernel/core_pattern 
|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e


/usr/lib/sysctl.d/50-coredump.conf set up core_pattern for systemd-coredump, and this break abrt-hook-ccpp setting.

I propose a solution to extend abrt-install-ccpp-hook.
- "abrt-install-ccpp-hook install" put /etc/sysctl.d/60-abrt-coredump.conf 
- "abrt-install-ccpp-hook uninstall" delete /etc/sysctl.d/60-abrt-coredump.conf
Comment 3 ekulik 2020-02-19 13:57:11 UTC 
This is a bit late, but what is the actual issue? Do you *want* to use abrt-hook-ccpp?
Comment 4 Kazuo Moriwaka 2020-02-20 00:54:43 UTC 
At first, I just want to use abrt-ccpp.  And I was very confused by this conflict.
This could be fixed by KCS solution about a workaround.

After I understand what was happen, I wanted is to improve the user experience. 
Configure enable/disable abrt-ccpp by just one abrt-install-ccpp-hook command.

My proposed solution is for later one.

Thanks,
Comment 5 ekulik 2020-02-20 08:40:15 UTC 
Okay, I just wanted to make sure whether you simply wanted ABRT to work or to use abrt-hook-ccpp specifically, since ABRT can also monitor the system journal and pick up crashes from there.
Comment 6 ekulik 2020-02-24 09:00:49 UTC 
I’m going to do it as you suggested, but the recommendation is to move to the abrt-journal-core service.

(And for some reason the link to the knowledge base article doesn’t work)
Comment 7 ekulik 2020-02-24 12:54:25 UTC 
Right, so sysctl configs cannot have arbitrary commands, so this is not going to work at all. Moreover, it would unconditionally set kernel.core_pattern, which is not what you would want if you disable abrt-ccpp.service or enable abrt-journal-core.service.
Comment 8 ekulik 2020-02-24 13:16:29 UTC 
What could be a solution, though, is changing the way abrt-install-ccpp-hook works: instead of poking /proc/sys/kernel/core_pattern directly, we install a sysctl configuration file with contents like such:

#kernel.core_pattern=|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %P %I %h %e

And the install/uninstall commands would comment and uncomment the line accordingly. No need to stash the previous core_pattern value, either.
Comment 10 ekulik 2020-02-26 13:48:21 UTC 
Reassigning to redhat-release. Can we please have abrt-journal-core.service in the preset instead of abrt-ccpp.service? That was the change implemented in Fedora and it works better with systemd. That would take care of new installations, but we can also add some release notes to document the situation.

Should anyone actually want to continue using abrt-hook-ccpp, then we will consider fixing this issue the proposed way.
Comment 12 ekulik 2020-03-13 17:02:23 UTC 
The Jira issue has a label set. Yes, 8.3.0.
Comment 13 Marcin Sobczyk 2020-03-16 12:55:20 UTC 
> Should anyone actually want to continue using abrt-hook-ccpp, then we will
> consider fixing this issue the proposed way.

'abrt-hook-ccpp' is still relevant for oVirt/RHV use case - currently the 'systemd-coredump'
approach works in a way that you end up with 2 copies of the same core dump which is very
problematic for VMs since the core files they generate are very big. It would be very nice
to have this issue resolved and to prioritize a feature that avoids the duplication when
using 'systemd-coredump'.
Comment 14 Petr Zatko 2020-05-15 10:15:32 UTC 
Hello Ernestas, it seems that swaping abrt-journal-core.service in the preset instead of abrt-ccpp.service would have some possible impact on some teams considering comment 13. 
From qe point of view I would grant qa_ack for the solutions as it is covered by automated test, but I would like to know whether the change is really intended or if tou consider fixing this issue the proposed way.
Comment 15 ekulik 2020-05-15 10:24:38 UTC 
(In reply to Petr Zatko from comment #14)
> Hello Ernestas, it seems that swaping abrt-journal-core.service in the
> preset instead of abrt-ccpp.service would have some possible impact on some
> teams considering comment 13. 
> From qe point of view I would grant qa_ack for the solutions as it is
> covered by automated test, but I would like to know whether the change is
> really intended or if tou consider fixing this issue the proposed way.

The above was discussed in private already and the impact is minimal, because the ability to use our custom hook for core dump analysis is still there. The proposed solution is still pending implementation upstream and is going to be more invasive, so there is greater potential for breakage should we decide to backport it, but it’s still an open question.

The agreement in the team was that this should be fine for new users if it is documented in the release notes. For most cases, it does not matter, because we have solutions in place to limit the impact of large core dumps being copied. The cases where it does matter are specialized (usually it’s a developer dealing with specific technologies, like WebKit).
Comment 16 ekulik 2020-05-15 10:26:24 UTC 
In fact, my understanding is that switching the preset should have been done prior to RHEL 8.0, as was done in the Fedora version it’s based on.
Comment 17 Petr Zatko 2020-05-15 11:47:52 UTC 
Thank you for the clarification, granting qa_ack and updating our test to reflect that.
Comment 25 errata-xmlrpc 2020-11-04 01:46:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (redhat-release bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4495
Comment 27 Red Hat Bugzilla 2023-09-15 01:27:58 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days
Note You need to log in before you can comment on or make changes to this bug.