Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
A customer reported systemd freezing execution after it aborted due to memory corruption on IBM Z:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
#0 0x000003ff8b33ec22 in kill () at ../sysdeps/unix/syscall-template.S:78
#1 0x000002aa0c33c730 in crash (sig=<optimized out>) at ../src/core/main.c:194
#2 <signal handler called>
#3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#4 0x000003ff8b323218 in __GI_abort () at abort.c:79
#5 0x000003ff8b38279c in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x3ff8b4678d0 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#6 0x000003ff8b38a348 in malloc_printerr (str=str@entry=0x3ff8b465e06 "malloc(): smallbin double linked list corrupted") at malloc.c:5374
#7 0x000003ff8b38d99e in _int_malloc (av=av@entry=0x3ff8b49e960 <main_arena>, bytes=bytes@entry=33) at malloc.c:3656
#8 0x000003ff8b38e67e in _int_realloc (av=av@entry=0x3ff8b49e960 <main_arena>, oldp=oldp@entry=0x2aa3460a7e0, oldsize=oldsize@entry=32, nb=nb@entry=48) at malloc.c:4612
#9 0x000003ff8b38f956 in __GI___libc_realloc (oldmem=0x2aa3460a7f0, bytes=28) at malloc.c:3238
#10 0x000003ff8bafc0a2 in strextend_with_separator (x=0x3ffd9a7def8, x@entry=<error reading variable: value has been optimized out>, separator=separator@entry=0x0) at ../src/basic/string-util.c:921
#11 0x000003ff8bad0222 in chase_symlinks (path=<optimized out>, path@entry=0x3ffd9a7e110 "/run/systemd/generator.late/session-71355.scope.wants", original_root=original_root@entry=0x0, flags=flags@entry=0, ret=<optimized out>,
ret@entry=0x3ffd9a7e178) at ../src/basic/fs-util.c:900
#12 0x000003ff8ba7c2b4 in unit_file_find_dir (dirs=0x3ffd9a7e250, path=0x3ffd9a7e110 "/run/systemd/generator.late/session-71355.scope.wants", original_root=0x0) at ../src/shared/dropin.c:112
#13 unit_file_find_dirs (original_root=original_root@entry=0x0, unit_path_cache=<optimized out>, unit_path_cache@entry=0x0, unit_path=0x2aa3440b900 "/run/systemd/generator.late", name=0x2aa345f0b60 "session-71355.scope",
suffix=suffix@entry=0x2aa0c41d8de ".wants", dirs=<optimized out>) at ../src/shared/dropin.c:154
#14 0x000003ff8ba7cd92 in unit_file_find_dropin_paths (original_root=original_root@entry=0x0, lookup_path=0x2aa34407de0, unit_path_cache=0x0, dir_suffix=dir_suffix@entry=0x2aa0c41d8de ".wants", file_suffix=<optimized out>,
file_suffix@entry=0x0, names=0x2aa345840d8, ret=0x3ffd9a7e380) at ../src/shared/dropin.c:243
#15 0x000002aa0c3e0850 in process_deps (u=u@entry=0x2aa345e91d0, dependency=dependency@entry=UNIT_WANTS, dir_suffix=dir_suffix@entry=0x2aa0c41d8de ".wants") at ../src/core/load-dropin.c:37
#16 0x000002aa0c3e0e78 in unit_load_dropin (u=0x2aa345e91d0) at ../src/core/load-dropin.c:112
[...]
#22 0x000002aa0c343f24 in method_start_transient_unit (error=0x3ffd9a7e9f0, userdata=0x2aa3440c330, message=0x2aa34622a00) at ../src/core/dbus-manager.c:992
[...]
#37 main (argc=<optimized out>, argv=0x3ffd9a7f638) at ../src/core/main.c:2606
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
This happened while systemd was starting a scope for an incoming user session.
This was seen on IBM Z but I don't see why it wouldn't happen on other hardware as well.
The error happened while extending string "/run/systemd" with "/generator.late":
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
(gdb) f 11
#11 0x000003ff8bad0222 in chase_symlinks (path=<optimized out>, path@entry=0x3ffd9a7e110 "/run/systemd/generator.late/session-71355.scope.wants", original_root=original_root@entry=0x0, flags=flags@entry=0, ret=<optimized out>,
ret@entry=0x3ffd9a7e178) at ../src/basic/fs-util.c:900
900 if (!strextend(&done, first, NULL))
(gdb) p first
$1 = 0x2aa3445fc90 "/generator.late"
(gdb) p done
$2 = 0x2aa3460a7f0 "/run/systemd"
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Version-Release number of selected component (if applicable):
systemd-239-45.el8.s390x
How reproducible:
No idea, didn't reproduce yet.
Closeing this as a dup of BZ #1970945 since root cause may be identical, even though backtraces are different:
since it's a memory corruption, it could have happen a long time before in the past and just show later doing any operation involving malloc() (i.e. for systemd, almost everything since malloc() is used widely).
*** This bug has been marked as a duplicate of bug 1970945 ***
Description of problem: A customer reported systemd freezing execution after it aborted due to memory corruption on IBM Z: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- #0 0x000003ff8b33ec22 in kill () at ../sysdeps/unix/syscall-template.S:78 #1 0x000002aa0c33c730 in crash (sig=<optimized out>) at ../src/core/main.c:194 #2 <signal handler called> #3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #4 0x000003ff8b323218 in __GI_abort () at abort.c:79 #5 0x000003ff8b38279c in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x3ff8b4678d0 "%s\n") at ../sysdeps/posix/libc_fatal.c:181 #6 0x000003ff8b38a348 in malloc_printerr (str=str@entry=0x3ff8b465e06 "malloc(): smallbin double linked list corrupted") at malloc.c:5374 #7 0x000003ff8b38d99e in _int_malloc (av=av@entry=0x3ff8b49e960 <main_arena>, bytes=bytes@entry=33) at malloc.c:3656 #8 0x000003ff8b38e67e in _int_realloc (av=av@entry=0x3ff8b49e960 <main_arena>, oldp=oldp@entry=0x2aa3460a7e0, oldsize=oldsize@entry=32, nb=nb@entry=48) at malloc.c:4612 #9 0x000003ff8b38f956 in __GI___libc_realloc (oldmem=0x2aa3460a7f0, bytes=28) at malloc.c:3238 #10 0x000003ff8bafc0a2 in strextend_with_separator (x=0x3ffd9a7def8, x@entry=<error reading variable: value has been optimized out>, separator=separator@entry=0x0) at ../src/basic/string-util.c:921 #11 0x000003ff8bad0222 in chase_symlinks (path=<optimized out>, path@entry=0x3ffd9a7e110 "/run/systemd/generator.late/session-71355.scope.wants", original_root=original_root@entry=0x0, flags=flags@entry=0, ret=<optimized out>, ret@entry=0x3ffd9a7e178) at ../src/basic/fs-util.c:900 #12 0x000003ff8ba7c2b4 in unit_file_find_dir (dirs=0x3ffd9a7e250, path=0x3ffd9a7e110 "/run/systemd/generator.late/session-71355.scope.wants", original_root=0x0) at ../src/shared/dropin.c:112 #13 unit_file_find_dirs (original_root=original_root@entry=0x0, unit_path_cache=<optimized out>, unit_path_cache@entry=0x0, unit_path=0x2aa3440b900 "/run/systemd/generator.late", name=0x2aa345f0b60 "session-71355.scope", suffix=suffix@entry=0x2aa0c41d8de ".wants", dirs=<optimized out>) at ../src/shared/dropin.c:154 #14 0x000003ff8ba7cd92 in unit_file_find_dropin_paths (original_root=original_root@entry=0x0, lookup_path=0x2aa34407de0, unit_path_cache=0x0, dir_suffix=dir_suffix@entry=0x2aa0c41d8de ".wants", file_suffix=<optimized out>, file_suffix@entry=0x0, names=0x2aa345840d8, ret=0x3ffd9a7e380) at ../src/shared/dropin.c:243 #15 0x000002aa0c3e0850 in process_deps (u=u@entry=0x2aa345e91d0, dependency=dependency@entry=UNIT_WANTS, dir_suffix=dir_suffix@entry=0x2aa0c41d8de ".wants") at ../src/core/load-dropin.c:37 #16 0x000002aa0c3e0e78 in unit_load_dropin (u=0x2aa345e91d0) at ../src/core/load-dropin.c:112 [...] #22 0x000002aa0c343f24 in method_start_transient_unit (error=0x3ffd9a7e9f0, userdata=0x2aa3440c330, message=0x2aa34622a00) at ../src/core/dbus-manager.c:992 [...] #37 main (argc=<optimized out>, argv=0x3ffd9a7f638) at ../src/core/main.c:2606 -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- This happened while systemd was starting a scope for an incoming user session. This was seen on IBM Z but I don't see why it wouldn't happen on other hardware as well. The error happened while extending string "/run/systemd" with "/generator.late": -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- (gdb) f 11 #11 0x000003ff8bad0222 in chase_symlinks (path=<optimized out>, path@entry=0x3ffd9a7e110 "/run/systemd/generator.late/session-71355.scope.wants", original_root=original_root@entry=0x0, flags=flags@entry=0, ret=<optimized out>, ret@entry=0x3ffd9a7e178) at ../src/basic/fs-util.c:900 900 if (!strextend(&done, first, NULL)) (gdb) p first $1 = 0x2aa3445fc90 "/generator.late" (gdb) p done $2 = 0x2aa3460a7f0 "/run/systemd" -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Version-Release number of selected component (if applicable): systemd-239-45.el8.s390x How reproducible: No idea, didn't reproduce yet.