Bug 1475565

This is causing me issues, as DKMS-built kernel modules aren't getting installed into the initramfs that's used by the system to boot.

Why does /boot/$(cat /etc/machine-id)/$(uname -r) exist on your machine?

Normally this is only used with native the systemd bootloader.

It is being automatically generated when upgrading the kernel on this machine.

If you remove /boot/$MACHINE_ID , is it created again by kernel upgrades on your machine? It should not.

Correct.

# uname -r
4.11.11-300.fc26.x86_64
# rm -rf /boot/$(cat /etc/machine-id)
# ls /boot/$(cat /etc/machine-id)
ls: cannot access '/boot/3a04f6d81a0941dca41e1d300776f3f6': No such file or directory
# dnf upgrade -q -y kernel
# ls /boot/$(cat /etc/machine-id)
4.12.8-300.fc26.x86_64/

/boot/initramfs-4.12.8-300.fc26.x86_64.img also exists after upgrading the kernel.

what is the stderr output of:

# bash -x kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz

Created attachment 1320609 [details]
kernel-install output

Output of `kernel-install add`, after having removed `/boot/<machine-id>`.

seems like systemd ships 20-grubby.install now, which only runs after $BOOT_DIR_ABS is created by kernel-install

Created attachment 1320860 [details]
possible fix

With the patch `/boot/<machine-id>/<kver>` is removed, but `/boot/<machine-id>` still exists. Subsequent runs of dracut on that kernel fail.

# dracut -f -v
dracut: Executing: /usr/bin/dracut -f -v
dracut: Can't write to /boot/3a04f6d81a0941dca41e1d300776f3f6/4.12.9-300.fc26.x86_64: Directory /boot/3a04f6d81a0941dca41e1d300776f3f6/4.12.9-300.fc26.x86_64 does not exist or is not accessible.

I don't think the patch from comment #c9 solved the issue. The location should not change when the directory exists, it's just too brittle and unexpected.

I'll submit a PR against dracut to not autodetect the directory like this instead.

This also causes me that I'm not able to boot new kernel while having encrypted home and swap. The initrd is generated badly and it does not ask for password to unlock luks. After removing the /boot/$(cat /etc/machine-id) and dracut --regenerate-all --force, booting works as expected.

Created attachment 1332572 [details]
Patch to 20-grubby.install

Here's a patch to /lib/kernel/install.d/20-grubby.install to remove $BOOT_DIR_ABS, it's siblings and it's parent if:
 1 - the grubby plugin is returning 77 (meaning skip other scripts)
 2 - the directories are empty

This should fix the "leftover" directories created by previous kernel-install runs and get dracut (and scripts that use it, like plymouth-set-default-theme) to work -- provided someone hasn't already created initrd files there (possibly with good reason).

I tested this with a kernel upgrade and it worked as expected (cleaning up the extra directories).

The patch is easy to test on an installed kernel (on a non-UEFI machine!) with:
# mkdir -p /boot/$(</etc/machine-id)/empty_dir
# kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz

FYI, another approach to fix this was proposed here,

https://github.com/dracutdevs/dracut/pull/281

Discussed at blocker review meeting [1]:

Punt (delay decision) - We need to perform more testing to estimate the user impact better.

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-10-02/

I've tried to reproduce the "failed to boot encrypted system" scenario, and I couldn't. After new kernel installation, the initrd is always generated at the correct place for me. Only when you try to update the initrd, it's generated elsewhere.

What I did:
Installed F27 Beta Workstation, looked like:

# cd /boot

# ll
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:26 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21952137 Oct  9 15:26 initramfs-4.13.3-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64


Initrds are still updated in proper place at this point:

# dracut -f

# ll
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:26 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64


After dnf distrosync and installing new kernel, a new /boot/<hash> directory appears:

# ll
drwxr-xr-x. 3 root root     4096 Oct  9 15:57 3698883553db47b88ebe9db6b9a39aba
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
-rw-r--r--. 1 root root   190419 Sep 28 19:14 config-4.13.4-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:58 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
-rw-------. 1 root root 21456991 Oct  9 15:58 initramfs-4.13.4-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rw-------. 1 root root  3661616 Sep 28 19:14 System.map-4.13.4-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7471192 Sep 28 19:15 vmlinuz-4.13.4-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/ 3698883553db47b88ebe9db6b9a39aba/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64
3698883553db47b88ebe9db6b9a39aba/
└── 4.13.4-300.fc27.x86_64


This time, updating initrd updates it in a wrong location:

# dracut -f

# ll
drwxr-xr-x. 3 root root     4096 Oct  9 15:57 3698883553db47b88ebe9db6b9a39aba
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
-rw-r--r--. 1 root root   190419 Sep 28 19:14 config-4.13.4-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:58 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
-rw-------. 1 root root 21456991 Oct  9 15:58 initramfs-4.13.4-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rw-------. 1 root root  3661616 Sep 28 19:14 System.map-4.13.4-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7471192 Sep 28 19:15 vmlinuz-4.13.4-300.fc27.x86_64
[root@localhost-live boot]# date
Mon Oct  9 16:01:11 CEST 2017

# tree aaaaa202093c472fa61803f6078f5d7b/ 3698883553db47b88ebe9db6b9a39aba/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64
3698883553db47b88ebe9db6b9a39aba/
└── 4.13.4-300.fc27.x86_64
    └── initrd


But installing a new kernel from updates-testing again puts initrd into proper location:

# ll
drwxr-xr-x. 4 root root     4096 Oct  9 16:06 3698883553db47b88ebe9db6b9a39aba
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
-rw-r--r--. 1 root root   190419 Sep 28 19:14 config-4.13.4-300.fc27.x86_64
-rw-r--r--. 1 root root   190441 Oct  5 19:17 config-4.13.5-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 16:06 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
-rw-------. 1 root root 21456991 Oct  9 15:58 initramfs-4.13.4-300.fc27.x86_64.img
-rw-------. 1 root root 21457211 Oct  9 16:06 initramfs-4.13.5-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rw-------. 1 root root  3661616 Sep 28 19:14 System.map-4.13.4-300.fc27.x86_64
-rw-------. 1 root root  3664288 Oct  5 19:17 System.map-4.13.5-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7471192 Sep 28 19:15 vmlinuz-4.13.4-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7475288 Oct  5 19:18 vmlinuz-4.13.5-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/ 3698883553db47b88ebe9db6b9a39aba/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64
3698883553db47b88ebe9db6b9a39aba/
├── 4.13.4-300.fc27.x86_64
│   └── initrd
└── 4.13.5-300.fc27.x86_64

For the record, we had additional discussion about this bug with Justin Forbes last week:

<kparal> jforbes: what do you think about security implications of not updating initramfs when dracut -f is called?
<kparal> or any other implications you can think of?
<kparal> I want to hear your thoughts in case new kernel installs generate initramfs fine, but tools trying to update it don't work
<jforbes> Possible missing drivers and such are the biggest issue
<jforbes> I mean it isn't really a workable long term solution, there are many reasons to need to update the initramfs
<kparal> aren't all drivers already present when it's generated with the new kernel installed? how would failing to update initramfs miss some drivers?
<kparal> also, what are your thoughts on this being a blocker (considering this case as described)?
<jforbes> Third part modules could be an issue, drivers not part of the kernel proper than dkms builds
<kparal> oh, I see. nvidia trying to build its module and such?
<kparal> virtualbox, etc?
<jforbes> I would be +1 to blocker
<jforbes> We also see a lot of failures for whatever reason fixed by rebuilding initramfs
<kparal> ok, thanks for details. we'll revisit this bug on our next meeting, hopefully with more testing done

Discussed at blocker review meeting [1]:

punt (delay decision) - No really important breakage is known. We'll wait another week if we find something.

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-10-09/

https://github.com/dracutdevs/dracut/commit/791d80168994f4784569c8ea47b7bff554102f5e

dracut-046-3.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0fac2c45e5

(In reply to Fedora Update System from comment #20)
> dracut-046-3.fc27 has been submitted as an update to Fedora 27.
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-0fac2c45e5

Seems fixed.

dracut-046-3.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0fac2c45e5

I've tested dracut-046-3, bug is fixed.

dracut-046-3.1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a986f116b0

dracut-046-4.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f7109f3f6c

dracut-046-4.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f7109f3f6c

dracut-046-4.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.