Bug 2047373

Summary: [RHEL-8.4] systemd still crashes in version systemd-239-45.el8_4.6.x86_64 with coredump
Product: Red Hat Enterprise Linux 8 Reporter: Stephanie <sleolaso>
Component: systemdAssignee: Michal Sekletar <msekleta>
Status: CLOSED ERRATA QA Contact: Frantisek Sumsal <fsumsal>
Severity: high Docs Contact:
Priority: high    
Version: 8.4CC: bkulawia, ccheney, dchong, ddutile, dhildenb, dseals, dtardon, ecrosby, fgrosjea, fkrska, fweimer, igreen, lnykryn, mgokhool, mmatsuya, msekleta, peter.vreman, qguo, rmetrich, saime, sbroz, seyi, skrenger, sleolaso, systemd-maint-list, toneata, tparsons, tstaudt, zpytela
Target Milestone: rcKeywords: Triaged, ZStream
Target Release: ---Flags: pm-rhel: mirror+
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: systemd-239-59.el8 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2084052 2084053 2087933 (view as bug list) Environment:
Last Closed: 2022-11-08 10:49:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2089089    
Bug Blocks: 2084052, 2084053, 2087933    

Description Stephanie 2022-01-27 17:58:34 UTC
Description of problem:

Customer applied updates on systemd in Rhel 8.4 EUS but was still affected by a bug.

Systemd still crashes in version systemd-239-45.el8_4.6.x86_64 generating coredump.

It should be solved by Bugzilla rhbz#2026244 per errata : https://access.redhat.com/errata/RHBA-2021:5099


Version-Release number of selected component (if applicable):

Rhel 8.4 EUS

systemd-239-45.el8_4.6.x86_64


How reproducible:

(Note we've only seen this once so far, so there is no way to easily reproduce this)


Additional info:

[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# cat installed-rpms | grep -i systemd
python3-systemd-234-8.el8.x86_64                            Thu May 13 14:53:43 2021
rpm-plugin-systemd-inhibit-4.14.3-14.el8_4.x86_64           Thu Jul 15 19:42:01 2021
systemd-239-45.el8_4.6.x86_64                               Fri Jan 14 19:23:40 2022      <-- updated at Jan 14Th
systemd-libs-239-45.el8_4.6.x86_64                          Fri Jan 14 19:23:39 2022
systemd-pam-239-45.el8_4.6.x86_64                           Fri Jan 14 19:23:40 2022
systemd-udev-239-45.el8_4.6.x86_64                          Fri Jan 14 19:23:41 2022
[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# 



	[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# sed -n 37,45p sos_commands/abrt/abrt-cli_list
	id dbd03a18e86b17ad8f9eaccc99ad2fc9aa773d83
	reason:         realloc(): systemd killed by SIGABRT
	time:           Tue Jan 25 16:21:01 2022                                 <--- Crash happened after using the version that should have the fixes
	cmdline:        /usr/lib/systemd/systemd --switched-root --system --deserialize 18
	package:        systemd-239-45.el8_4.6
	uid:            0 (root)
	count:          1
	Directory:      /var/spool/abrt/ccpp-2022-01-25-16:21:01-2137021
	Run 'abrt-cli report /var/spool/abrt/ccpp-2022-01-25-16:21:01-2137021' for creating a case in Red Hat Customer Portal
	[root@sleola sosreport-srv009201-2022-01-25-xjnnjeu]# 


(gdb) bt
#0  0x00007ff8860c067b in kill () at ../sysdeps/unix/syscall-template.S:78
#1  0x000055b56f246f7a in crash (sig=6) at ../src/core/main.c:194
#2  <signal handler called>
#3  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#4  0x00007ff8860aadb5 in __GI_abort () at abort.c:79
#5  0x00007ff8861034e7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ff886212a0e "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#6  0x00007ff88610a5ec in malloc_printerr (str=str@entry=0x7ff886214a88 "malloc(): smallbin double linked list corrupted") at malloc.c:5374
#7  0x00007ff88610d8b4 in _int_malloc (av=av@entry=0x7ff886448bc0 <main_arena>, bytes=bytes@entry=113) at malloc.c:3656
#8  0x00007ff88610e3af in _int_realloc (av=av@entry=0x7ff886448bc0 <main_arena>, oldp=oldp@entry=0x55b56fbb54b0, oldsize=oldsize@entry=112, nb=nb@entry=128) at malloc.c:4612
#9  0x00007ff88610f67b in __GI___libc_realloc (oldmem=0x55b56fbb54c0, bytes=bytes@entry=112) at malloc.c:3238
#10 0x00007ff88789d1f6 in message_extend_fields (m=m@entry=0x55b56fbfff40, align=align@entry=8, sz=sz@entry=7, add_offset=add_offset@entry=false) at ../src/libsystemd/sd-bus/bus-message.c:163
#11 0x00007ff8878a57fa in message_append_field_signature (h=8, ret=0x0, s=0x55b56fc124a0 "s", m=0x55b56fbfff40) at ../src/libsystemd/sd-bus/bus-message.c:299
#12 sd_bus_message_seal (m=m@entry=0x55b56fbfff40, cookie=4070336, timeout_usec=timeout_usec@entry=25000000) at ../src/libsystemd/sd-bus/bus-message.c:2922
#13 0x00007ff8878b767e in bus_seal_message (b=b@entry=0x55b56fbcffd0, m=0x55b56fbfff40, timeout=25000000, timeout@entry=0) at ../src/libsystemd/sd-bus/sd-bus.c:1667
#14 0x00007ff8878be7e9 in sd_bus_send (bus=0x55b56fbcffd0, _m=<optimized out>, cookie=cookie@entry=0x0) at ../src/libsystemd/sd-bus/sd-bus.c:1860
#15 0x00007ff887890179 in sd_bus_reply_method_error (call=0x55b56fc67840, e=<optimized out>) at ../src/libsystemd/sd-bus/bus-convenience.c:185
#16 0x00007ff887890673 in sd_bus_reply_method_errno (call=0x55b56fc67840, error=-3, p=0x7ffc4df230d0) at ../src/libsystemd/sd-bus/bus-convenience.c:236
#17 0x00007ff88789a09d in bus_maybe_reply_error (m=m@entry=0x55b56fc67840, r=r@entry=-3, error=error@entry=0x7ffc4df230d0) at ../src/libsystemd/sd-bus/bus-internal.c:341
#18 0x00007ff8878ad4c0 in method_callbacks_run (found_object=0x7ffc4df23187, require_fallback=false, c=<optimized out>, m=0x55b56fc67840, bus=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/bus-objects.c:409
#19 object_find_and_run (bus=0x55b56fbcffd0, m=0x55b56fc67840, p=0x55b56fc61198 "/org/freedesktop/systemd1", require_fallback=false, found_object=0x7ffc4df23187) at ../src/libsystemd/sd-bus/bus-objects.c:1268
#20 0x00007ff8878aec7c in bus_process_object (bus=bus@entry=0x55b56fbcffd0, m=m@entry=0x55b56fc67840) at ../src/libsystemd/sd-bus/bus-objects.c:1388
#21 0x00007ff8878bdbaa in process_message (m=0x55b56fc67840, bus=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/sd-bus.c:2705
#22 process_running (ret=0x0, priority=<optimized out>, hint_priority=<optimized out>, bus=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/sd-bus.c:2747
#23 bus_process_internal (bus=0x55b56fbcffd0, hint_priority=<optimized out>, priority=<optimized out>, ret=0x0) at ../src/libsystemd/sd-bus/sd-bus.c:2966
#24 0x00007ff8878bf89c in io_callback (s=<optimized out>, fd=<optimized out>, revents=<optimized out>, userdata=0x55b56fbcffd0) at ../src/libsystemd/sd-bus/sd-bus.c:3346
#25 0x00007ff8878e6f19 in source_dispatch (s=0x55b56fc1b790) at ../src/libsystemd/sd-event/sd-event.c:3344
#26 0x00007ff8878e8d0d in sd_event_dispatch (e=0x55b56f9fb520) at ../src/libsystemd/sd-event/sd-event.c:3763
#27 0x00007ff8878e8f18 in sd_event_run (e=0x55b56f9fb520, timeout=18446744073709551615) at ../src/libsystemd/sd-event/sd-event.c:3823
#28 0x000055b56f28bab2 in manager_loop (m=m@entry=0x55b56f9fd560) at ../src/core/manager.c:2884
#29 0x000055b56f242471 in invoke_main_loop (ret_error_message=0x7ffc4df23548, ret_switch_root_init=<synthetic pointer>, ret_switch_root_dir=<synthetic pointer>, ret_fds=0x7ffc4df23558, ret_shutdown_verb=<synthetic pointer>, 
    ret_retval=<synthetic pointer>, ret_reexecute=<synthetic pointer>, saved_rlimit_memlock=0x7ffc4df235c0, saved_rlimit_nofile=0x7ffc4df235b0, m=0x55b56f9fd560) at ../src/core/main.c:1665
#30 main (argc=<optimized out>, argv=0x7ffc4df23828) at ../src/core/main.c:2606
(gdb)

Comment 2 Renaud Métrich 2022-02-11 14:21:52 UTC
Instructions for installing the ASAN build updated in the KCS.

Comment 14 Lukáš Nykrýn 2022-04-28 16:45:18 UTC
Is that sosreport from the affected machine? I don't see any crashes in the logs.

Comment 15 Lukáš Nykrýn 2022-04-28 17:39:24 UTC
If the customer wants to help us, here is a new test build:
https://people.redhat.com/~lnykryn/systemd-239-46.el8_4.9.1.asan/

It is built with asan, based on the latest 8.4 z-stream and I've added the patch Frantisek found. What we got in the log is either a bug and then the build could fix it or a false positive and then it should remove it.

Comment 33 Michal Sekletar 2022-05-10 17:51:56 UTC
Test build is here, 
https://msekleta.fedorapeople.org/systemd-test-build-bz2047373/

You can install it by running following command,
yum --nogpgcheck --repofrompath=bz2047373,https://msekleta.fedorapeople.org/systemd-test-build-bz2047373 update -y systemd

Issues addressed in the test build,
- undefined behaviour reported by UBSAN (in previous debug build) in src/shared/acpi-fpdt.c
  - https://github.com/systemd/systemd/pull/15543/commits/49490c1d353bc920cbf73f4c71e9c35d2e3eb8b1

- systemd-coredump may get stuck in the infinite loop (backport from RHEL-8.6)
  - https://bugzilla.redhat.com/show_bug.cgi?id=1977569

- incorrect reference count update for sd_bus_track object,
  - https://github.com/systemd/systemd/issues/23097
  - https://github.com/systemd/systemd/pull/23099 

- avoid use-after-free after unit_free() when reloading,
  - https://github.com/systemd/systemd/issues/23312
  - https://github.com/systemd/systemd/pull/23313

- skip unnecessary freezer operation on slice unit and avoid assertion,
  - https://github.com/systemd/systemd/issues/23278
  - https://github.com/systemd/systemd/pull/23283

Comment 36 Plumber Bot 2022-05-11 13:20:33 UTC
fix merged to github master branch -> https://github.com/redhat-plumbers/systemd-rhel8/pull/279

Comment 52 errata-xmlrpc 2022-11-08 10:49:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (systemd bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7727