We also started to see this in cockpit's CI in e.g. this test run [1]. Full journal at [2].

[1] https://logs-https-frontdoor.apps.ocp.ci.centos.org/logs/pull-16124-20210722-043022-771379a1-fedora-34/log.html#151
[2] https://logs-https-frontdoor.apps.ocp.ci.centos.org/logs/pull-16124-20210722-043022-771379a1-fedora-34/TestSOS-testWithUrlRoot-fedora-34-127.0.0.2-2201-FAIL.log.gz

I'm also hitting this on my ARM home server using systemd-248.4-1.fc34.aarch64 on Fedora 34 IoT. My personal laptop is running systemd-248.5-1.fc34.x86_64 and is fine, however.

In my specific case, it looks like the problem is with a specific podman container. (It's an often-badly-behaving Minecraft Java server I've been running for my nephews.) I removed the Minecraft container (which will probably make my nephews and their friends sad, but they can play Fortnite instead 😉), pulled the plug (as rebooting doesn't even work without systemd operating properly), and did a fresh boot. So far, everything is working... and I even have a few other services in Podman containers that still work.

I know Cockpit's CI uses containers and VMs... could this be something triggered by resource-hungry containers?

The particular test where we see this does not use containers in the sense of "podman". (Of course every systemd service is also a container of some kind). The whole VM qemu runs inside a container on the host, but the QEMU layer should sufficiently shield systemd from knowing that.

*** Bug 1985126 has been marked as a duplicate of this bug. ***

Just diagnosed this issue for several of my fc34 servers after having looked at lots of https://bugzilla.redhat.com/show_bug.cgi?id=1463745 -like issues in the last days (the symptoms are similar: systemctl commands are not responding (incl. "reboot" without the -f) and timeouts after 30s, ssh login is very slow (waits for 90s) ) - however solutions like kill -9 1 or systemctl daemon-reexec will not do anything here (reboot helps for a few hours to a few days)


same journalctl as reporter : 
Jul 23 16:57:46 lame15.enst.fr systemd[1]: Assertion 'a <= b' failed at src/libsystemd/sd-event/sd-event.c:2903, function sleep_between(). Aborting.
Jul 23 16:57:46 lame15.enst.fr audit[15757]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 pid=15757 comm="systemd" exe="/usr/lib/systemd/systemd" sig=6 res=1
Jul 23 16:57:46 lame15.enst.fr systemd-coredump[15758]: Due to PID 1 having crashed coredump collection will now be turned off.
Jul 23 16:57:46 lame15.enst.fr systemd-coredump[15758]: [🡕] Process 15757 (systemd) of user 0 dumped core.
                                                        
                                                        Stack trace of thread 15757:
                                                        #0  0x00007fac997a159b kill (libc.so.6 + 0x3d59b)
                                                        #1  0x0000556f87ffebc8 crash (systemd + 0x45bc8)
                                                        #2  0x00007fac99946a20 __restore_rt (libpthread.so.0 + 0x13a20)
                                                        #3  0x00007fac997a12a2 raise (libc.so.6 + 0x3d2a2)
                                                        #4  0x00007fac9978a8a4 abort (libc.so.6 + 0x268a4)
                                                        #5  0x00007fac99aeca42 log_assert_failed.cold (libsystemd-shared-248.so + 0x78a42)
                                                        #6  0x00007fac99c85d6f sleep_between (libsystemd-shared-248.so + 0x211d6f)
                                                        #7  0x00007fac99c8e2b9 event_arm_timer (libsystemd-shared-248.so + 0x21a2b9)
                                                        #8  0x00007fac99c8ecf5 sd_event_prepare (libsystemd-shared-248.so + 0x21acf5)
                                                        #9  0x00007fac99c91d50 sd_event_run (libsystemd-shared-248.so + 0x21dd50)
                                                        #10 0x0000556f880437d4 manager_loop (systemd + 0x8a7d4)
                                                        #11 0x0000556f87ffbc5f main (systemd + 0x42c5f)
                                                        #12 0x00007fac9978bb75 __libc_start_main (libc.so.6 + 0x27b75)
                                                        #13 0x0000556f87ffe77e _start (systemd + 0x4577e)
Jul 23 16:57:46 lame15.enst.fr systemd[1]: Caught <ABRT>, dumped core as pid 15757.
Jul 23 16:57:46 lame15.enst.fr systemd[1]: Freezing execution.
Jul 23 16:57:47 lame15.enst.fr systemd-oomd[826]: Failed to connect to /run/systemd/io.system.ManagedOOM: Connection refused
Jul 23 16:57:47 lame15.enst.fr systemd-oomd[826]: Failed to acquire varlink connection: Connection refused
Jul 23 16:57:47 lame15.enst.fr systemd-oomd[826]: Event loop failed: Connection refused

(System fully up to date, package info below:)
Name         : systemd
Version      : 248.5
Release      : 1.fc34
Architecture : x86_64
Size         : 14 M
Source       : systemd-248.5-1.fc34.src.rpm
Repository   : @System
From repo    : updates

FEDORA-2021-3141f0eff1 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-3141f0eff1

FEDORA-2021-3141f0eff1 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-3141f0eff1`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-3141f0eff1

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2021-3141f0eff1 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

*** Bug 1985690 has been marked as a duplicate of this bug. ***