Bug 2487442
| Summary: | fail to boot when / and /var are encrypted | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Barry Scott <barry> | ||||||
| Component: | systemd | Assignee: | systemd-maint | ||||||
| Status: | POST --- | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 44 | CC: | daan.j.demeyer, dtardon, fedoraproject, frantisek, lnykryn, msekleta, suraj.ghimire7, systemd-maint, vmihalko, yuwatana, zbyszek | ||||||
| Target Milestone: | --- | Keywords: | Regression, Triaged | ||||||
| Target Release: | --- | ||||||||
| Hardware: | x86_64 | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | --- | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | Type: | --- | |||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
|
Description
Barry Scott
2026-06-10 09:14:29 UTC
Here is the disk info from lsblk for the working case.
In the failing case the the luks-26... was not present.
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 600M 0 part /boot/efi
├─sda2 8:2 0 4G 0 part /boot
└─sda3 8:3 0 230G 0 part
├─fender2-root 252:0 0 30G 0 lvm
│ └─luks-59487bf2-7ae4-4699-aef7-96ee7f74f057
│ 252:1 0 30G 0 crypt /
└─fender2-var 252:2 0 200G 0 lvm
└─luks-2600c0ec-f0aa-4b62-b3cc-d5db29da9f70
252:3 0 200G 0 crypt /var/lib/containers/storage/overlay
/var
sdb 8:16 0 7.3T 0 disk
└─sdb1 8:17 0 7.3T 0 part
└─md127 9:127 0 7.3T 0 raid1
└─crypt_shares 252:4 0 7.3T 0 crypt
├─fender_shares-Media 252:5 0 320G 0 lvm /shared/Media
├─fender_shares-Downloads 252:6 0 180G 0 lvm /shared/Downloads
...
sdc 8:32 0 7.3T 0 disk
└─sdc1 8:33 0 7.3T 0 part
└─md127 9:127 0 7.3T 0 raid1
└─crypt_shares 252:4 0 7.3T 0 crypt
├─fender_shares-Media 252:5 0 320G 0 lvm /shared/Media
├─fender_shares-Downloads 252:6 0 180G 0 lvm /shared/Downloads
Can you please add debug to kernel cmdline, reproduce the issue and send us output of journalctl -b There is a lot of commits between 259.5 and 259.6. But nothing that would seem obviously related. The debug logs would be nice to have. Created attachment 2146964 [details]
good boot with systemd-259.5-1
Created attachment 2146965 [details]
bad boot with systemd-259.6-1
The boot dropped into the emergency shell.
I waited a while then Ctrl-D and the boot completed.
While trying to reproduce without losing production data I add "nofail" to the /var mount. One time I saw that boot succeed. But rebooting saw it fail. In the emergency shell I saw that the mount of /var was completed. My guess is that there is a timing change between the good and bad systemd code. Are there explicit Requires= After= added by the generators for fstab and crypttab? It looks like the culprit might be https://github.com/systemd/systemd/commit/0b50107fe2 that creates a dependency loop: cryptsetup-pre.target -> (via Before=) systemd-pcrnvdone.service -> (via After=) systemd-tpm2-setup.service -> (via RequiresMountsFor=/var/lib/systemd) var.mount -> this needs luks-2600c0ec-f0aa-4b62-b3cc-d5db29da9f70 -> systemd-cryptsetup@...service -> (via After=) cryptsetup-pre.target -> ... which delays all LUKS mounts by 60 seconds. This then causes dev-fender_shares-Downloads.device to timeout after 90s, and the system ends up in emergency shell: Time Event 13:31:29 v259.6 starts after switchroot. Generators run. dev-fender_shares-Downloads.device job 217 installed (90s timeout starts) 13:31:31 systemd-cryptsetup held back waiting for cryptsetup-pre.target 13:31:31 systemd-cryptsetup held back waiting for cryptsetup-pre.target 13:31:31 cryptsetup-pre.target blocked by systemd-pcrnvdone.service (Before=) 13:31:31 systemd-pcrnvdone.service held back waiting for systemd-tpm2-setup.service (After=) 13:31:31 systemd-tpm2-setup.service held back waiting for var.mount (RequiresMountsFor=) 13:31:31 var.mount waiting for its device... which needs luks-2600c0ec... which needs cryptsetup-pre.target -- DEADLOCK 13:32:29 After 60 seconds, the /var device times out, breaking the cycle: var.mount fails -> tpm2-setup fails -> pcrnvdone is skipped -> cryptsetup-pre.target finally activates 13:32:29 Both LUKS cryptsetup services finally start (58s wasted!) 13:32:57 crypt_shares LUKS unlocked, LVM activation begins 13:32:59 dev-fender_shares-Downloads.device TIMES OUT (exactly 90s from 13:31:29). 13:32:59 local-fs.target fails -> OnFailure=emergency.target fires A fix is _probably_ https://github.com/systemd/systemd/commit/856ab04a29 that hasn't been backported to the stable branch. (In reply to Frantisek Sumsal from comment #7) > A fix is _probably_ https://github.com/systemd/systemd/commit/856ab04a29 > that hasn't been backported to the stable branch. Yeah, looks like that. Let me know when there is a systemd version I test to confirm the regression is fixed. (In reply to Barry Scott from comment #9) > Let me know when there is a systemd version I test to confirm the regression > is fixed. v261 (available in Rawhide) |