Description of problem:
I ran sudo dnf upgrade --refresh in a Fedora 34 KDE Plasma installation on 2021-2-19. The update included systemd-247.3-3.fc34.x86_64. While dnf was upgrading systemd, an error message "systemd[1]: Caught <ABRT>, dumped core as pid 5419." was shown.
While dnf was cleaning up packages, kwin_wayland segmentation faulted. The error that systemd had aborted was shown in the VT.
Broadcast message from systemd-journald...
systemd[1]: Caught <ABRT>, dumped core as pid 5419.
Broadcast message from systemd-journald...
systemd[1]: Freezing execution.
After several minutes of the system remaining on that message, I used sysrq+alt+r,e,i,s,u,b to kill all processes, sync, remount read-only and reboot.
The trace of the systemd crash showed a "corrupted double-linked list" in malloc_printerr at malloc.c:5626 in glibc-2.32.9000-29.fc34.x86_64 in frame 6.
I reported that crash at https://bugzilla.redhat.com/show_bug.cgi?id=1930793
I rebooted and logged into Plasma on Wayland. sudo dnf history info showed that the previous transaction hadn't completed properly. I ran sudo rpm --rebuilddb. I logged out of Plasma and logged into a VT. I reinstalled the roughly 80 packages in the previous update except systemd with sudo dnf reinstall. I reinstalled systemd with sudo dnf reinstall systemd*247.3* While the systemd-247.3-3 post-install scriptlet ran, systemd aborted with a message like
Broadcast message from systemd-journald...
systemd[1]: Caught <ABRT>, dumped core as pid 14024.
Broadcast message from systemd-journald...
systemd[1]: Freezing execution.
The journal at the time of the systemd crash showed the error 'Assertion 'p->n_ref > 0' failed at src/shared/varlink.c:386, function varlink_unref(). Aborting.'
Feb 19 11:09:35 sudo[13983]: matt : TTY=tty3 ; PWD=/home/matt ; USER=root ; COMMAND=/usr/bin/dnf reinstall systemd*247.3*
Feb 19 11:09:35 audit[13983]: CRED_REFR pid=13983 uid=1000 auid=1000 ses=7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_fprintd acct="root" exe="/usr/bin/sudo" hostname=fedora addr=? terminal=/dev/tty3 res=success'
Feb 19 11:09:35 sudo[13983]: pam_unix(sudo:session): session opened for user root(uid=0) by matt(uid=1000)
Feb 19 11:09:35 audit[13983]: USER_START pid=13983 uid=1000 auid=1000 ses=7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=fedora addr=? terminal=/dev/tty3 res=success'
Feb 19 11:09:43 dbus-broker-launch[811]: Noticed file-system modification, trigger reload.
Feb 19 11:09:43 polkitd[834]: Reloading rules
Feb 19 11:09:43 polkitd[834]: Collecting garbage unconditionally...
Feb 19 11:09:43 polkitd[834]: Loading rules from directory /etc/polkit-1/rules.d
Feb 19 11:09:43 polkitd[834]: Loading rules from directory /usr/share/polkit-1/rules.d
Feb 19 11:09:43 polkitd[834]: Finished loading, compiling and executing 7 rules
Feb 19 11:09:43 polkitd[834]: Reloading rules
Feb 19 11:09:43 polkitd[834]: Collecting garbage unconditionally...
Feb 19 11:09:43 polkitd[834]: Loading rules from directory /etc/polkit-1/rules.d
Feb 19 11:09:43 polkitd[834]: Loading rules from directory /usr/share/polkit-1/rules.d
Feb 19 11:09:43 polkitd[834]: Finished loading, compiling and executing 7 rules
Feb 19 11:09:43 dbus-broker-launch[811]: Noticed file-system modification, trigger reload.
Feb 19 11:09:44 dbus-broker-launch[811]: Noticed file-system modification, trigger reload.
Feb 19 11:09:44 dbus-broker-launch[4094]: Noticed file-system modification, trigger reload.
Feb 19 11:09:44 dbus-broker-launch[4094]: Service file '/usr/share//dbus-1/services/sealert.service' is not named after the D-Bus name 'org.fedoraproject.Setroubleshootd'.
Feb 19 11:09:44 dbus-broker-launch[4094]: Service file '/usr/share//dbus-1/services/org.kde.kscreen.service' is not named after the D-Bus name 'org.kde.KScreen'.
Feb 19 11:09:44 dbus-broker-launch[4094]: Service file '/usr/share//dbus-1/services/org.kde.plasma.Notifications.service' is not named after the D-Bus name 'org.freedesktop.Notifications'.
Feb 19 11:09:44 dbus-broker-launch[4094]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
Feb 19 11:09:44 dbus-broker-launch[4094]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
Feb 19 11:09:44 dbus-broker-launch[811]: Noticed file-system modification, trigger reload.
Feb 19 11:09:45 systemd[1]: Reexecuting.
Feb 19 11:09:45 audit: BPF prog-id=106 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=105 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=104 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=127 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=126 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=109 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=108 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=107 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=125 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=124 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=123 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=110 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=113 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=112 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=111 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=116 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=115 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=114 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=117 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=120 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=119 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=118 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=122 op=UNLOAD
Feb 19 11:09:45 audit: BPF prog-id=121 op=UNLOAD
Feb 19 11:09:45 systemd[1]: Assertion 'p->n_ref > 0' failed at src/shared/varlink.c:386, function varlink_unref(). Aborting.
Feb 19 11:09:45 audit[14024]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:init_t:s0 pid=14024 comm="systemd" exe=2F7573722F6C69622F73797374656D642F73797374656D64202864656C6574656429 sig=6 res=1
Feb 19 11:09:45 systemd-coredump[14025]: Due to PID 1 having crashed coredump collection will now be turned off.
Feb 19 11:09:45 systemd-coredump[14025]: Process 14024 (systemd) of user 0 dumped core.
Stack trace of thread 14024:
#0 0x00007f43f86ec58b kill (libc.so.6 + 0x3d58b)
#1 0x000055d40915348e crash (systemd + 0x4848e)
#2 0x00007f43f8892960 __restore_rt (libpthread.so.0 + 0x13960)
#3 0x00007f43f86ec292 __GI_raise (libc.so.6 + 0x3d292)
#4 0x00007f43f86d58a4 __GI_abort (libc.so.6 + 0x268a4)
#5 0x00007f43f8a698d2 log_assert_failed_realm (libsystemd-shared-247.so + 0x728d2)
#6 0x00007f43f8b063c1 varlink_unref (libsystemd-shared-247.so + 0x10f3c1)
#7 0x000055d40919b0d8 manager_free.part.0 (systemd + 0x900d8)
#8 0x000055d409150572 main (systemd + 0x45572)
#9 0x00007f43f86d6b75 __libc_start_main (libc.so.6 + 0x27b75)
#10 0x000055d40915301e _start (systemd + 0x4801e)
Feb 19 11:09:45 systemd[1]: Caught <ABRT>, dumped core as pid 14024.
Feb 19 11:09:45 systemd[1]: Freezing execution.
Feb 19 11:09:45 abrt-dump-journal-core[872]: Failed to obtain all required information from journald
Feb 19 11:09:46 dbus-broker-launch[811]: Noticed file-system modification, trigger reload.
Feb 19 11:09:46 dbus-broker-launch[811]: ERROR launcher_reload_config @ ../src/launch/launcher.c +1164: Connection refused
Feb 19 11:09:46 dbus-broker-launch[811]: launcher_on_dirwatch @ ../src/launch/launcher.c +154
Feb 19 11:09:46 systemd-oomd[741]: Failed to connect to /run/systemd/io.system.ManagedOOM: Connection refused
Feb 19 11:09:46 systemd-oomd[741]: Failed to acquire varlink connection
Feb 19 11:09:46 systemd-oomd[741]: Event loop failed: Connection refused
Feb 19 11:10:37 abrt-notification[14088]: Process 14024 (systemd) crashed in in_addr_prefix_to_string.cold()
The trace of the systemd crash show the failed assertion in frame 6 in varlink_unref at ../src/shared/varlink.c:386 in systemd-247.3-3.fc34.
Core was generated by `/usr/lib/systemd/systemd --switched-root --system --deserialize 31'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f43f86ec58b in kill () at ../sysdeps/unix/syscall-template.S:120
120 ../sysdeps/unix/syscall-template.S: No such file or directory.
[Current thread is 1 (LWP 14024)]
(gdb) bt
#0 0x00007f43f86ec58b in kill () at ../sysdeps/unix/syscall-template.S:120
#1 0x000055d40915348e in crash (sig=6) at ../src/core/main.c:224
#2 <signal handler called>
#3 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#4 0x00007f43f86d58a4 in __GI_abort () at abort.c:79
#5 0x00007f43f8a698d2 in log_assert_failed_realm (realm=<optimized out>, text=<optimized out>,
file=<optimized out>, line=<optimized out>, func=<optimized out>) at ../src/basic/log.c:864
#6 0x00007f43f8b063c1 in varlink_unref (p=<optimized out>) at ../src/shared/varlink.c:386
#7 varlink_unref (p=0x55d4099e6a70) at ../src/shared/varlink.c:386
#8 0x00007f43f8b0af7b in varlink_close_unref (v=<optimized out>) at ../src/shared/varlink.c:1212
#9 0x000055d40919b0d8 in manager_varlink_done (m=0x55d4099e26b0) at ../src/core/core-varlink.c:479
#10 manager_free (m=0x55d4099e26b0) at ../src/core/manager.c:1367
#11 0x000055d409150572 in manager_free (m=<optimized out>) at ../src/core/manager.c:1352
#12 main (argc=<optimized out>, argv=<optimized out>) at ../src/core/main.c:2878
Many systemd services failed to restart because their connection timed out while the systemd-247.3-3.fc34 post-uninstall scriptlet ran.
Failed to try-restart systemd-udevd.service: Connection timed out
See system logs and 'systemctl status systemd-udevd.service' for details.
Failed to try-restart systemd-timesyncd.service: Connection timed out
See system logs and 'systemctl status systemd-timesyncd.service' for details.
Failed to try-restart systemd-timedated.service: Connection timed out
See system logs and 'systemctl status systemd-timedated.service' for details.
Failed to try-restart systemd-portabled.service: Connection timed out
See system logs and 'systemctl status systemd-portabled.service' for details.
Failed to try-restart systemd-homed.service: Connection timed out
See system logs and 'systemctl status systemd-homed.service' for details.
Failed to try-restart systemd-hostnamed.service: Connection timed out
See system logs and 'systemctl status systemd-hostnamed.service' for details.
Failed to try-restart systemd-journald.service: Connection timed out
See system logs and 'systemctl status systemd-journald.service' for details.
Failed to try-restart systemd-localed.service: Connection timed out
See system logs and 'systemctl status systemd-localed.service' for details.
Failed to try-restart systemd-userdbd.service: Connection timed out
See system logs and 'systemctl status systemd-userdbd.service' for details.
Failed to try-restart systemd-oomd.service: Connection timed out
See system logs and 'systemctl status systemd-oomd.service' for details.
The systemd-247.3-3.fc34.x86_64 post-transaction scriptlet failed with the following error messages.
Failed to reload daemon: Connection timed out
warning: %transfiletriggerin(systemd-247.3-3.fc34.x86_64) scriptlet failed, exit status 1
The transaction completed after several minutes, whereas it would normally take a few seconds. When I logged out of the VT, the log out didn't seem to complete. I switched back to SDDM and pressed Restart, but nothing happened. I tried to log in to Plasma again but sddm froze. I tried to switch to another VT, but the system became unresponsive. I used sysrq+alt+r,e,i,s,u,b to kill all processes, sync, remount read-only and reboot.
Version-Release number of selected component:
systemd-247.3-3.fc34
Additional info:
reporter: libreport-2.14.0
backtrace_rating: 4
cgroup: 0::/init.scope
cmdline: /usr/lib/systemd/systemd --switched-root --system --deserialize 31
crash_function: log_assert_failed_realm
executable: /usr/lib/systemd/systemd
journald_cursor: s=c26fcd8a329a4568939359e3e9030ac0;i=2147f3;b=42dfc5a20baa446b8c343fe2a67f8d11;m=12cc181b0;t=5bbb2ae408e23;x=ec979b0ca530df88
kernel: 5.11.0-156.fc34.x86_64
rootdir: /
runlevel: N 5
type: CCpp
uid: 0
Truncated backtrace:
Thread no. 1 (5 frames)
#5 log_assert_failed_realm at ../src/basic/log.c:864
#6 varlink_unref at ../src/shared/varlink.c:386
#8 varlink_close_unref at ../src/shared/varlink.c:1212
#9 manager_varlink_done at ../src/core/core-varlink.c:479
#10 manager_free at ../src/core/manager.c:1367