RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2047373 - [RHEL-8.4] systemd still crashes in version systemd-239-45.el8_4.6.x86_64 with coredump
Summary: [RHEL-8.4] systemd still crashes in version systemd-239-45.el8_4.6.x86_64 wi...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: systemd
Version: 8.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Michal Sekletar
QA Contact: Frantisek Sumsal
URL:
Whiteboard:
Depends On: 2089089
Blocks: 2084052 2084053 2087933
TreeView+ depends on / blocked
 
Reported: 2022-01-27 17:58 UTC by Stephanie
Modified: 2023-01-20 10:36 UTC (History)
29 users (show)

Fixed In Version: systemd-239-59.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2084052 2084053 2087933 (view as bug list)
Environment:
Last Closed: 2022-11-08 10:49:54 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-110073 0 None None None 2022-01-27 18:02:51 UTC
Red Hat Knowledge Base (Solution) 6369201 0 None None None 2022-02-11 14:01:18 UTC
Red Hat Product Errata RHBA-2022:7727 0 None None None 2022-11-08 10:50:26 UTC

Description Stephanie 2022-01-27 17:58:34 UTC 
Description of problem:

Customer applied updates on systemd in Rhel 8.4 EUS but was still affected by a bug.

Systemd still crashes in version systemd-239-45.el8_4.6.x86_64 generating coredump.

It should be solved by Bugzilla rhbz#2026244 per errata : https://access.redhat.com/errata/RHBA-2021:5099


Version-Release number of selected component (if applicable):

Rhel 8.4 EUS

systemd-239-45.el8_4.6.x86_64


How reproducible:

(Note we've only seen this once so far, so there is no way to easily reproduce this)


Additional info:

[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# cat installed-rpms | grep -i systemd
python3-systemd-234-8.el8.x86_64                            Thu May 13 14:53:43 2021
rpm-plugin-systemd-inhibit-4.14.3-14.el8_4.x86_64           Thu Jul 15 19:42:01 2021
systemd-239-45.el8_4.6.x86_64                               Fri Jan 14 19:23:40 2022      <-- updated at Jan 14Th
systemd-libs-239-45.el8_4.6.x86_64                          Fri Jan 14 19:23:39 2022
systemd-pam-239-45.el8_4.6.x86_64                           Fri Jan 14 19:23:40 2022
systemd-udev-239-45.el8_4.6.x86_64                          Fri Jan 14 19:23:41 2022
[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# 



	[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# sed -n 37,45p sos_commands/abrt/abrt-cli_list
	id dbd03a18e86b17ad8f9eaccc99ad2fc9aa773d83
	reason:         realloc(): systemd killed by SIGABRT
	time:           Tue Jan 25 16:21:01 2022                                 <--- Crash happened after using the version that should have the fixes
	cmdline:        /usr/lib/systemd/systemd --switched-root --system --deserialize 18
	package:        systemd-239-45.el8_4.6
	uid:            0 (root)
	count:          1
	Directory:      /var/spool/abrt/ccpp-2022-01-25-16:21:01-2137021
	Run 'abrt-cli report /var/spool/abrt/ccpp-2022-01-25-16:21:01-2137021' for creating a case in Red Hat Customer Portal
	[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# 


(gdb) bt
#0  0x00007ff8860c067b in kill () at ../sysdeps/unix/syscall-template.S:78
#1  0x000055b56f246f7a in crash (sig=6) at ../src/core/main.c:194
#2  <signal handler called>
#3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#4  0x00007ff8860aadb5 in __GI_abort () at abort.c:79
#5  0x00007ff8861034e7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ff886212a0e "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#6  0x00007ff88610a5ec in malloc_printerr (str=str@entry=0x7ff886214a88 "malloc(): smallbin double linked list corrupted") at malloc.c:5374
#7  0x00007ff88610d8b4 in _int_malloc (av=av@entry=0x7ff886448bc0 <main_arena>, bytes=bytes@entry=113) at malloc.c:3656
#8  0x00007ff88610e3af in _int_realloc (av=av@entry=0x7ff886448bc0 <main_arena>, oldp=oldp@entry=0x55b56fbb54b0, oldsize=oldsize@entry=112, nb=nb@entry=128) at malloc.c:4612
#9  0x00007ff88610f67b in __GI___libc_realloc (oldmem=0x55b56fbb54c0, bytes=bytes@entry=112) at malloc.c:3238
#10 0x00007ff88789d1f6 in message_extend_fields (m=m@entry=0x55b56fbfff40, align=align@entry=8, sz=sz@entry=7, add_offset=add_offset@entry=false) at ../src/libsystemd/sd-bus/bus-message.c:163
#11 0x00007ff8878a57fa in message_append_field_signature (h=8, ret=0x0, s=0x55b56fc124a0 "s", m=0x55b56fbfff40) at ../src/libsystemd/sd-bus/bus-message.c:299
#12 sd_bus_message_seal (m=m@entry=0x55b56fbfff40, cookie=4070336, timeout_usec=timeout_usec@entry=25000000) at ../src/libsystemd/sd-bus/bus-message.c:2922
#13 0x00007ff8878b767e in bus_seal_message (b=b@entry=0x55b56fbcffd0, m=0x55b56fbfff40, timeout=25000000, timeout@entry=0) at ../src/libsystemd/sd-bus/sd-bus.c:1667
#14 0x00007ff8878be7e9 in sd_bus_send (bus=0x55b56fbcffd0, _m=<optimized out>, cookie=cookie@entry=0x0) at ../src/libsystemd/sd-bus/sd-bus.c:1860
#15 0x00007ff887890179 in sd_bus_reply_method_error (call=0x55b56fc67840, e=<optimized out>) at ../src/libsystemd/sd-bus/bus-convenience.c:185
#16 0x00007ff887890673 in sd_bus_reply_method_errno (call=0x55b56fc67840, error=-3, p=0x7ffc4df230d0) at ../src/libsystemd/sd-bus/bus-convenience.c:236
#17 0x00007ff88789a09d in bus_maybe_reply_error (m=m@entry=0x55b56fc67840, r=r@entry=-3, error=error@entry=0x7ffc4df230d0) at ../src/libsystemd/sd-bus/bus-internal.c:341
#18 0x00007ff8878ad4c0 in method_callbacks_run (found_object=0x7ffc4df23187, require_fallback=false, c=<optimized out>, m=0x55b56fc67840, bus=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/bus-objects.c:409
#19 object_find_and_run (bus=0x55b56fbcffd0, m=0x55b56fc67840, p=0x55b56fc61198 "/org/freedesktop/systemd1", require_fallback=false, found_object=0x7ffc4df23187) at ../src/libsystemd/sd-bus/bus-objects.c:1268
#20 0x00007ff8878aec7c in bus_process_object (bus=bus@entry=0x55b56fbcffd0, m=m@entry=0x55b56fc67840) at ../src/libsystemd/sd-bus/bus-objects.c:1388
#21 0x00007ff8878bdbaa in process_message (m=0x55b56fc67840, bus=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/sd-bus.c:2705
#22 process_running (ret=0x0, priority=<optimized out>, hint_priority=<optimized out>, bus=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/sd-bus.c:2747
#23 bus_process_internal (bus=0x55b56fbcffd0, hint_priority=<optimized out>, priority=<optimized out>, ret=0x0) at ../src/libsystemd/sd-bus/sd-bus.c:2966
#24 0x00007ff8878bf89c in io_callback (s=<optimized out>, fd=<optimized out>, revents=<optimized out>, userdata=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/sd-bus.c:3346
#25 0x00007ff8878e6f19 in source_dispatch (s=0x55b56fc1b790) at ../src/libsystemd/sd-event/sd-event.c:3344
#26 0x00007ff8878e8d0d in sd_event_dispatch (e=0x55b56f9fb520) at ../src/libsystemd/sd-event/sd-event.c:3763
#27 0x00007ff8878e8f18 in sd_event_run (e=0x55b56f9fb520, timeout=18446744073709551615) at ../src/libsystemd/sd-event/sd-event.c:3823
#28 0x000055b56f28bab2 in manager_loop (m=m@entry=0x55b56f9fd560) at ../src/core/manager.c:2884
#29 0x000055b56f242471 in invoke_main_loop (ret_error_message=0x7ffc4df23548, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffc4df23558, ret_shutdown_verb=<synthetic pointer>, 
    ret_retval=<synthetic pointer>, ret_reexecute=<synthetic pointer>, saved_rlimit_memlock=0x7ffc4df235c0, saved_rlimit_nofile=0x7ffc4df235b0, m=0x55b56f9fd560) at ../src/core/main.c:1665
#30 main (argc=<optimized out>, argv=0x7ffc4df23828) at ../src/core/main.c:2606
(gdb)
Comment 2 Renaud Métrich 2022-02-11 14:21:52 UTC 
Instructions for installing the ASAN build updated in the KCS.
Comment 14 Lukáš Nykrýn 2022-04-28 16:45:18 UTC 
Is that sosreport from the affected machine? I don't see any crashes in the logs.
Comment 15 Lukáš Nykrýn 2022-04-28 17:39:24 UTC 
If the customer wants to help us, here is a new test build:
https://people.redhat.com/~lnykryn/systemd-239-46.el8_4.9.1.asan/

It is built with asan, based on the latest 8.4 z-stream and I've added the patch Frantisek found. What we got in the log is either a bug and then the build could fix it or a false positive and then it should remove it.
Comment 33 Michal Sekletar 2022-05-10 17:51:56 UTC 
Test build is here, 
https://msekleta.fedorapeople.org/systemd-test-build-bz2047373/

You can install it by running following command,
yum --nogpgcheck --repofrompath=bz2047373,https://msekleta.fedorapeople.org/systemd-test-build-bz2047373 update -y systemd

Issues addressed in the test build,
- undefined behaviour reported by UBSAN (in previous debug build) in src/shared/acpi-fpdt.c
  - https://github.com/systemd/systemd/pull/15543/commits/49490c1d353bc920cbf73f4c71e9c35d2e3eb8b1

- systemd-coredump may get stuck in the infinite loop (backport from RHEL-8.6)
  - https://bugzilla.redhat.com/show_bug.cgi?id=1977569

- incorrect reference count update for sd_bus_track object,
  - https://github.com/systemd/systemd/issues/23097
  - https://github.com/systemd/systemd/pull/23099 

- avoid use-after-free after unit_free() when reloading,
  - https://github.com/systemd/systemd/issues/23312
  - https://github.com/systemd/systemd/pull/23313

- skip unnecessary freezer operation on slice unit and avoid assertion,
  - https://github.com/systemd/systemd/issues/23278
  - https://github.com/systemd/systemd/pull/23283
Comment 36 Plumber Bot 2022-05-11 13:20:33 UTC 
fix merged to github master branch -> https://github.com/redhat-plumbers/systemd-rhel8/pull/279
Comment 52 errata-xmlrpc 2022-11-08 10:49:54 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7727
Note You need to log in before you can comment on or make changes to this bug.