Description of problem: After upgrading from F40 to F41 production system no longer booting due to frozen systemd-udevd.service ### booting in rescue mode ### # systemctl status systemd-udevd.service ○ systemd-udevd.service - Rule-based Manager for Device Events and Files Loaded: loaded (/usr/lib/systemd/system/systemd-udevd.service; static) Drop-In: /usr/lib/systemd/system/service.d └─10-timeout-abort.conf, 50-keep-warm.conf Active: inactive (dead) (thawing) since Thu 2025-05-22 11:37:19 CEST; 15min ago Duration: 1.376s Invocation: 6e390cac004f4811bc51d40634535167 TriggeredBy: ● systemd-udevd-kernel.socket ● systemd-udevd-control.socket Docs: man:systemd-udevd.service(8) man:udev(7) Main PID: 362 (code=exited, status=0/SUCCESS) Status: "Shutting down..." CPU: 499ms Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:37:24 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. Mai 22 11:47:18 <hostname> systemd[1]: Cannot start frozen unit systemd-udevd.service - Rule-based Manager for Device Events and Files. BUT: You can't stop/start/restart the service # systemctl restart systemd-udevd.service Cannot perform operation on frozen unit systemd-udevd.service. because it's frozen. But it is IMPOSSIBLE to freeze/unfreeze it, because the unit does not support freeze! [root@ ~]# systemctl freeze systemd-udevd.service Failed to freeze unit systemd-udevd.service: Unit does not support freeze/thaw [root@ ~]# systemctl thaw systemd-udevd.service Failed to thaw unit systemd-udevd.service: Unit does not support freeze/thaw How is that even possible?!?!?!?! Version-Release number of selected component (if applicable): systemd-libs-256.12-1.fc41.x86_64 systemd-pam-256.12-1.fc41.x86_64 systemd-resolved-256.12-1.fc41.x86_64 systemd-networkd-256.12-1.fc41.x86_64 systemd-256.12-1.fc41.x86_64 systemd-bootchart-235-1.fc41.x86_64 systemd-udev-256.12-1.fc41.x86_64 python3-systemd-235-11.fc41.x86_64 systemd-container-256.12-1.fc41.x86_64 systemd-rpm-macros-256.12-1.fc41.noarch systemd-boot-unsigned-256.12-1.fc41.x86_64 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results:
$ systemctl show systemd-udevd brings this: ... LoadState=loaded ActiveState=inactive FreezerState=thawing SubState=dead ... CanStart=yes CanStop=yes CanReload=yes CanIsolate=no CanFreeze=yes ... but IF thats true, it could freeze, which "systemctl freeze systemd-udevd" denied to be possible (s.a.). tried : reinstalling systemd* systemctl clean systemd-udevd no change.. it's stuck.
(In reply to customercare from comment #0) > [root@ ~]# systemctl freeze systemd-udevd.service > Failed to freeze unit systemd-udevd.service: Unit does not support > freeze/thaw > [root@ ~]# systemctl thaw systemd-udevd.service > Failed to thaw unit systemd-udevd.service: Unit does not support freeze/thaw > > How is that even possible?!?!?!?! It's because the error message may be misleading. systemd itself only returns EOPNOTSUPP error code for an unit that doesn't support freeze/thaw, but it could also be returned by the kernel when the operation is actually being performed. The code that check the error code and prints the message wouldn't be able to tell the difference in that case. You can try to run strace on PID 1 to get an idea where the EOPNOTSUPP comes from.
Also, what does /sys/fs/cgroup/system.slice/systemd-udevd.service/cgroup.freeze contain and what happens if you try to write 0 there?
File not found: # cat /sys/fs/cgroup/system.slice/systemd-udevd.service/cgroup.freeze cat: /sys/fs/cgroup/system.slice/systemd-udevd.service/cgroup.freeze: Datei oder Verzeichnis nicht gefunden # strace -f -p 1 2>&1 | grep EOPNOTSUPP did not give out ANY result while trying to start systemd-udevd.service, but a few seconds later it logged something unrelated removexattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.oomd_avoid") = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) removexattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.oomd_omit") = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) removexattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.journald_log_filter_pattern"...) = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) removexattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.coredump_receive") = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) setxattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.invocation_id", "3c00b1c331b74d5e960dbdf6f2577173", 32, 0) = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) removexattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.delegate") = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) removexattr("/sys/fs/cgroup/system.slice/sysstat-collect.service", "user.survive_final_kill_signal") = -1 EOPNOTSUPP (Die Operation wird nicht unterstützt) ... etc.etc.... But i have to give an update what i tried in the meantime: because i could not find the place where the false state was stored persistently, I just cloned systemd-udevd.service to systemd-udevd-test.service and I cloned rndg.service in the same way. Both services start without any problems. I moved the 2 frozen unit files to / and was able to boot the system. Ofcourse the references to systemd-udevd.service in all other files fail, because there is none. ### Issues atm: - Network has to be started in rescue shell with manual start of dhclient I think, that - i do not have a login context on the monitor is due to the udevd sockets pointing to the wrong service file. - same cause for unit dev-xvda2.swap failing, but executing "swapon /dev/xvda2" in rescue mode works flawless. IMHO: because the udevd-test clone starts flawless, there must be a persistent corrupted state file on disk, but i can't find it.
Downgrading to F40 systemd-255 ... **WORKS** booting straight up. Now we have to wait for systemd devs to respond ( ticket open )
Today i upgraded another similar system.. at the end of the upgrade process a disturbing amount of errors show up: ... LAST PACKAGE TO BE UPGRADED .... Aufräumen : libgcc-14.2.1-3.fc40.x86_64 2871/2871 Ausgeführtes Scriptlet: libgcc-14.2.1-3.fc40.x86_64 2871/2871 Ausgeführtes Scriptlet: filesystem-3.18-23.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: glibc-all-langpacks-2.40-25.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: grub2-common-1:2.12-21.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: GeoIP-GeoLite-data-2018.06-17.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: crypto-policies-scripts-20250124-1.git4d262e7.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: fontconfig-2.15.0-8.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: ca-certificates-2024.2.69_v8.0.401-1.0.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: authselect-libs-1.5.0-8.fc41.x86_64 2871/2871 /usr/bin/sed: /etc/authselect/authselect.conf kann nicht gelesen werden: Datei oder Verzeichnis nicht gefunden Ausgeführtes Scriptlet: systemd-resolved-256.12-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: dconf-0.40.0-14.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: dovecot-1:2.3.21.1-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: appstream-1.0.4-2.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: httpd-2.4.63-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: urw-base35-bookman-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-c059-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-d050000l-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-gothic-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-nimbus-mono-ps-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-nimbus-roman-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-nimbus-sans-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-p052-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-standard-symbols-ps-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: urw-base35-z003-fonts-20200910-23.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: rpm-4.20.1-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: sssd-common-2.10.2-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: selinux-policy-targeted-41.39-1.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: selinux-policy-minimum-41.39-1.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: container-selinux-4:2.237.0-1.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: mysql-selinux-1.0.11-1.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: freeipa-selinux-4.12.2-8.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: java-21-openjdk-1:21.0.7.0.6-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: java-11-openjdk-1:11.0.27.0.6-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: java-1.8.0-openjdk-1:1.8.0.452.b06-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: java-11-openjdk-devel-1:11.0.27.0.6-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: java-17-openjdk-1:17.0.15.0.6-1.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: sphinx-2.2.11-30.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: copy-jdk-configs-4.1-6.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: plocate-1.1.22-4.fc41.x86_64 2871/2871 Ausgeführtes Scriptlet: rootfiles-8.1-37.fc41.noarch 2871/2871 Ausgeführtes Scriptlet: libgcc-14.2.1-3.fc40.x86_64 2871/2871 Failed to start bus client: No such file or directory Reload daemon failed: Der Socket ist nicht verbunden Failed to start bus client: No such file or directory Failed to start jobs: Der Socket ist nicht verbunden "/home" already exists and is not a directory. Failed to start bus client: No such file or directory Reload daemon failed: Der Socket ist nicht verbunden Failed to start bus client: No such file or directory Failed to start jobs: Der Socket ist nicht verbunden Another shell showed, that systemd-udevd was not running, so a lot of rpms do not care to start theire service up once finished. luckily this time, the systemd-udevd was not frozen and just started up normal. Not to get me wrong, after the last script of distro-sync is executed a reboot follows, so technically it does not matter for a dist upgrade as long as the system boots again, but i makes me worry about "normal" updates of those rpms as well. With the "new" general upgrade path, by doing it on boot in a blackbox, you will never see those mistakes.
This message is a reminder that Fedora Linux 41 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 41 on 2025-12-15. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '41'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 41 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
testresults on this matter ina few days.
PROBLEM PERSISTS. Installed package: systemd-udev-257.10-1.fc42.x86_64 Unit does not start because it's frozen and can't be unfrozen by systemctl thaw , because it's not support freezing :D
*SOLVED* by upgrading the lts kernel.