Fedora Account System
Red Hat Associate
Red Hat Customer
I upgrade Fedora Server yesterday. The system failed to boot because only 1 of the 3 LUKS encrypted partitions where unlocked. There are LUKS partitions for /, /var and a MD raid /shares. System only prompted to unlock /. In the emergency shell lsblk -f shows only / unlocked. After some work was able to boot the system and downgrade systemd. Then the system booted as expected. Working version of systemd: systemd-259.5-1.fc44.x86_64 systemd-libs-259.5-1.fc44.x86_64 systemd-networkd-259.5-1.fc44.x86_64 systemd-oomd-defaults-259.5-1.fc44.noarch systemd-pam-259.5-1.fc44.x86_64 systemd-resolved-259.5-1.fc44.x86_64 systemd-shared-259.5-1.fc44.x86_64 systemd-sysusers-259.5-1.fc44.x86_64 systemd-udev-259.5-1.fc44.x86_64 Version that failed to allow booting (from log of dnf history): $ dnf history info 77 | grep systemd Upgrade systemd-0:259.6-1.fc44.x86_64 Group updates Upgrade systemd-libs-0:259.6-1.fc44.x86_64 Dependency updates Upgrade systemd-shared-0:259.6-1.fc44.x86_64 Dependency updates Upgrade systemd-udev-0:259.6-1.fc44.x86_64 Group updates Upgrade systemd-resolved-0:259.6-1.fc44.x86_64 Group updates Upgrade systemd-networkd-0:259.6-1.fc44.x86_64 User updates Upgrade systemd-oomd-defaults-0:259.6-1.fc44.noarch Group updates Upgrade systemd-pam-0:259.6-1.fc44.x86_64 Dependency updates Upgrade systemd-sysusers-0:259.6-1.fc44.x86_64 Dependency updates Reproducible: Always Steps to Reproduce: 1. Create system with luks encrypted / and /var 2. upgrade from systemd 259.5-1 to 259.6-1 3. reboot Actual Results: System does not complete boot. Drops into emergency shell. Expected Results: System boots.
Here is the disk info from lsblk for the working case. In the failing case the the luks-26... was not present. NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 600M 0 part /boot/efi ├─sda2 8:2 0 4G 0 part /boot └─sda3 8:3 0 230G 0 part ├─fender2-root 252:0 0 30G 0 lvm │ └─luks-59487bf2-7ae4-4699-aef7-96ee7f74f057 │ 252:1 0 30G 0 crypt / └─fender2-var 252:2 0 200G 0 lvm └─luks-2600c0ec-f0aa-4b62-b3cc-d5db29da9f70 252:3 0 200G 0 crypt /var/lib/containers/storage/overlay /var sdb 8:16 0 7.3T 0 disk └─sdb1 8:17 0 7.3T 0 part └─md127 9:127 0 7.3T 0 raid1 └─crypt_shares 252:4 0 7.3T 0 crypt ├─fender_shares-Media 252:5 0 320G 0 lvm /shared/Media ├─fender_shares-Downloads 252:6 0 180G 0 lvm /shared/Downloads ... sdc 8:32 0 7.3T 0 disk └─sdc1 8:33 0 7.3T 0 part └─md127 9:127 0 7.3T 0 raid1 └─crypt_shares 252:4 0 7.3T 0 crypt ├─fender_shares-Media 252:5 0 320G 0 lvm /shared/Media ├─fender_shares-Downloads 252:6 0 180G 0 lvm /shared/Downloads
Can you please add debug to kernel cmdline, reproduce the issue and send us output of journalctl -b
There is a lot of commits between 259.5 and 259.6. But nothing that would seem obviously related. The debug logs would be nice to have.
Created attachment 2146964 [details] good boot with systemd-259.5-1
Created attachment 2146965 [details] bad boot with systemd-259.6-1 The boot dropped into the emergency shell. I waited a while then Ctrl-D and the boot completed.
While trying to reproduce without losing production data I add "nofail" to the /var mount. One time I saw that boot succeed. But rebooting saw it fail. In the emergency shell I saw that the mount of /var was completed. My guess is that there is a timing change between the good and bad systemd code. Are there explicit Requires= After= added by the generators for fstab and crypttab?
It looks like the culprit might be https://github.com/systemd/systemd/commit/0b50107fe2 that creates a dependency loop: cryptsetup-pre.target -> (via Before=) systemd-pcrnvdone.service -> (via After=) systemd-tpm2-setup.service -> (via RequiresMountsFor=/var/lib/systemd) var.mount -> this needs luks-2600c0ec-f0aa-4b62-b3cc-d5db29da9f70 -> systemd-cryptsetup@...service -> (via After=) cryptsetup-pre.target -> ... which delays all LUKS mounts by 60 seconds. This then causes dev-fender_shares-Downloads.device to timeout after 90s, and the system ends up in emergency shell: Time Event 13:31:29 v259.6 starts after switchroot. Generators run. dev-fender_shares-Downloads.device job 217 installed (90s timeout starts) 13:31:31 systemd-cryptsetup held back waiting for cryptsetup-pre.target 13:31:31 systemd-cryptsetup held back waiting for cryptsetup-pre.target 13:31:31 cryptsetup-pre.target blocked by systemd-pcrnvdone.service (Before=) 13:31:31 systemd-pcrnvdone.service held back waiting for systemd-tpm2-setup.service (After=) 13:31:31 systemd-tpm2-setup.service held back waiting for var.mount (RequiresMountsFor=) 13:31:31 var.mount waiting for its device... which needs luks-2600c0ec... which needs cryptsetup-pre.target -- DEADLOCK 13:32:29 After 60 seconds, the /var device times out, breaking the cycle: var.mount fails -> tpm2-setup fails -> pcrnvdone is skipped -> cryptsetup-pre.target finally activates 13:32:29 Both LUKS cryptsetup services finally start (58s wasted!) 13:32:57 crypt_shares LUKS unlocked, LVM activation begins 13:32:59 dev-fender_shares-Downloads.device TIMES OUT (exactly 90s from 13:31:29). 13:32:59 local-fs.target fails -> OnFailure=emergency.target fires A fix is _probably_ https://github.com/systemd/systemd/commit/856ab04a29 that hasn't been backported to the stable branch.
(In reply to Frantisek Sumsal from comment #7) > A fix is _probably_ https://github.com/systemd/systemd/commit/856ab04a29 > that hasn't been backported to the stable branch. Yeah, looks like that.
Let me know when there is a systemd version I test to confirm the regression is fixed.
(In reply to Barry Scott from comment #9) > Let me know when there is a systemd version I test to confirm the regression > is fixed. v261 (available in Rawhide)