Bug 1475565 - initramfs not generated at default location if /boot/<machine-id> exists
initramfs not generated at default location if /boot/<machine-id> exists
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: dracut (Show other bugs)
26
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: dracut-maint-list
Fedora Extras Quality Assurance
https://fedoraproject.org/wiki/Common...
: CommonBugs
Depends On:
Blocks: F27FinalBlocker
  Show dependency treegraph
 
Reported: 2017-07-26 18:12 EDT by Tim Crawford
Modified: 2017-10-13 19:30 EDT (History)
21 users (show)

See Also:
Fixed In Version: dracut-046-4.fc27
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-13 19:30:13 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
kernel-install output (3.51 KB, text/plain)
2017-08-31 10:20 EDT, Tim Crawford
no flags Details
possible fix (736 bytes, patch)
2017-09-01 04:02 EDT, Harald Hoyer
no flags Details | Diff
Patch to 20-grubby.install (1.10 KB, patch)
2017-09-30 00:40 EDT, Scott Shambarger
no flags Details | Diff

  None (edit)
Description Tim Crawford 2017-07-26 18:12:18 EDT
Description of problem:

> If <image> is omitted or empty, then the default location /boot/initramfs-<kernel version>.img is used.

If `/boot/<machine-id>/` exists, dracut attempts to create the initramfs at `/boot/<machine-id>/$(uname -r)/initrd`.

Version-Release number of selected component (if applicable):
044-183.fc26

How reproducible:
Always

Steps to Reproduce:
1. `sudo mkdir -p /boot/$(cat /etc/machine-id)/$(uname -r)`
2. `sudo dracut -f -v`

Actual results:
dracut creates the initramfs at `/boot/<machine-id>/$(uname -r)/initrd`

Expected results:
dracut creates the initramfs at `/boot/initramfs-$(uname -r).img`
Comment 1 Neal Gompa 2017-08-03 22:06:09 EDT
This is causing me issues, as DKMS-built kernel modules aren't getting installed into the initramfs that's used by the system to boot.
Comment 2 Harald Hoyer 2017-08-07 09:32:27 EDT
Why does /boot/$(cat /etc/machine-id)/$(uname -r) exist on your machine?

Normally this is only used with native the systemd bootloader.
Comment 3 Tim Crawford 2017-08-30 09:57:55 EDT
It is being automatically generated when upgrading the kernel on this machine.
Comment 4 Harald Hoyer 2017-08-30 10:23:37 EDT
If you remove /boot/$MACHINE_ID , is it created again by kernel upgrades on your machine? It should not.
Comment 5 Tim Crawford 2017-08-30 13:06:06 EDT
Correct.

# uname -r
4.11.11-300.fc26.x86_64
# rm -rf /boot/$(cat /etc/machine-id)
# ls /boot/$(cat /etc/machine-id)
ls: cannot access '/boot/3a04f6d81a0941dca41e1d300776f3f6': No such file or directory
# dnf upgrade -q -y kernel
# ls /boot/$(cat /etc/machine-id)
4.12.8-300.fc26.x86_64/

/boot/initramfs-4.12.8-300.fc26.x86_64.img also exists after upgrading the kernel.
Comment 6 Harald Hoyer 2017-08-31 04:00:12 EDT
what is the stderr output of:

# bash -x kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz
Comment 7 Tim Crawford 2017-08-31 10:20 EDT
Created attachment 1320609 [details]
kernel-install output

Output of `kernel-install add`, after having removed `/boot/<machine-id>`.
Comment 8 Harald Hoyer 2017-09-01 04:02:11 EDT
seems like systemd ships 20-grubby.install now, which only runs after $BOOT_DIR_ABS is created by kernel-install
Comment 9 Harald Hoyer 2017-09-01 04:02 EDT
Created attachment 1320860 [details]
possible fix
Comment 10 Tim Crawford 2017-09-05 08:23:33 EDT
With the patch `/boot/<machine-id>/<kver>` is removed, but `/boot/<machine-id>` still exists. Subsequent runs of dracut on that kernel fail.

# dracut -f -v
dracut: Executing: /usr/bin/dracut -f -v
dracut: Can't write to /boot/3a04f6d81a0941dca41e1d300776f3f6/4.12.9-300.fc26.x86_64: Directory /boot/3a04f6d81a0941dca41e1d300776f3f6/4.12.9-300.fc26.x86_64 does not exist or is not accessible.
Comment 11 Zbigniew Jędrzejewski-Szmek 2017-09-06 04:38:14 EDT
I don't think the patch from comment #c9 solved the issue. The location should not change when the directory exists, it's just too brittle and unexpected.

I'll submit a PR against dracut to not autodetect the directory like this instead.
Comment 12 Dalibor Pospíšil 2017-09-17 11:07:17 EDT
This also causes me that I'm not able to boot new kernel while having encrypted home and swap. The initrd is generated badly and it does not ask for password to unlock luks. After removing the /boot/$(cat /etc/machine-id) and dracut --regenerate-all --force, booting works as expected.
Comment 13 Scott Shambarger 2017-09-30 00:40 EDT
Created attachment 1332572 [details]
Patch to 20-grubby.install

Here's a patch to /lib/kernel/install.d/20-grubby.install to remove $BOOT_DIR_ABS, it's siblings and it's parent if:
 1 - the grubby plugin is returning 77 (meaning skip other scripts)
 2 - the directories are empty

This should fix the "leftover" directories created by previous kernel-install runs and get dracut (and scripts that use it, like plymouth-set-default-theme) to work -- provided someone hasn't already created initrd files there (possibly with good reason).

I tested this with a kernel upgrade and it worked as expected (cleaning up the extra directories).

The patch is easy to test on an installed kernel (on a non-UEFI machine!) with:
# mkdir -p /boot/$(</etc/machine-id)/empty_dir
# kernel-install add $(uname -r) /lib/modules/$(uname -r)/vmlinuz
Comment 14 Michal Sekletar 2017-10-02 03:46:37 EDT
FYI, another approach to fix this was proposed here,

https://github.com/dracutdevs/dracut/pull/281
Comment 15 Kamil Páral 2017-10-02 13:07:46 EDT
Discussed at blocker review meeting [1]:

Punt (delay decision) - We need to perform more testing to estimate the user impact better.

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-10-02/
Comment 16 Kamil Páral 2017-10-09 10:15:22 EDT
I've tried to reproduce the "failed to boot encrypted system" scenario, and I couldn't. After new kernel installation, the initrd is always generated at the correct place for me. Only when you try to update the initrd, it's generated elsewhere.

What I did:
Installed F27 Beta Workstation, looked like:

# cd /boot

# ll
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:26 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21952137 Oct  9 15:26 initramfs-4.13.3-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64


Initrds are still updated in proper place at this point:

# dracut -f

# ll
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:26 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64


After dnf distrosync and installing new kernel, a new /boot/<hash> directory appears:

# ll
drwxr-xr-x. 3 root root     4096 Oct  9 15:57 3698883553db47b88ebe9db6b9a39aba
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
-rw-r--r--. 1 root root   190419 Sep 28 19:14 config-4.13.4-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:58 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
-rw-------. 1 root root 21456991 Oct  9 15:58 initramfs-4.13.4-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rw-------. 1 root root  3661616 Sep 28 19:14 System.map-4.13.4-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7471192 Sep 28 19:15 vmlinuz-4.13.4-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/ 3698883553db47b88ebe9db6b9a39aba/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64
3698883553db47b88ebe9db6b9a39aba/
└── 4.13.4-300.fc27.x86_64


This time, updating initrd updates it in a wrong location:

# dracut -f

# ll
drwxr-xr-x. 3 root root     4096 Oct  9 15:57 3698883553db47b88ebe9db6b9a39aba
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
-rw-r--r--. 1 root root   190419 Sep 28 19:14 config-4.13.4-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 15:58 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
-rw-------. 1 root root 21456991 Oct  9 15:58 initramfs-4.13.4-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rw-------. 1 root root  3661616 Sep 28 19:14 System.map-4.13.4-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7471192 Sep 28 19:15 vmlinuz-4.13.4-300.fc27.x86_64
[root@localhost-live boot]# date
Mon Oct  9 16:01:11 CEST 2017

# tree aaaaa202093c472fa61803f6078f5d7b/ 3698883553db47b88ebe9db6b9a39aba/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64
3698883553db47b88ebe9db6b9a39aba/
└── 4.13.4-300.fc27.x86_64
    └── initrd


But installing a new kernel from updates-testing again puts initrd into proper location:

# ll
drwxr-xr-x. 4 root root     4096 Oct  9 16:06 3698883553db47b88ebe9db6b9a39aba
drwxr-xr-x. 3 root root     4096 Sep 28 01:22 aaaaa202093c472fa61803f6078f5d7b
-rw-r--r--. 1 root root   190814 Sep 20 20:28 config-4.13.3-300.fc27.x86_64
-rw-r--r--. 1 root root   190419 Sep 28 19:14 config-4.13.4-300.fc27.x86_64
-rw-r--r--. 1 root root   190441 Oct  5 19:17 config-4.13.5-300.fc27.x86_64
drwx------. 4 root root     4096 Sep 28 01:22 efi
-rw-r--r--. 1 root root   184380 Feb 11  2017 elf-memtest86+-5.01
drwxr-xr-x. 2 root root     4096 Sep 28 01:19 extlinux
drwx------. 6 root root     4096 Oct  9 16:06 grub2
-rw-------. 1 root root 70718979 Oct  9 15:25 initramfs-0-rescue-3698883553db47b88ebe9db6b9a39aba.img
-rw-------. 1 root root 21478164 Oct  9 15:45 initramfs-4.13.3-300.fc27.x86_64.img
-rw-------. 1 root root 21456991 Oct  9 15:58 initramfs-4.13.4-300.fc27.x86_64.img
-rw-------. 1 root root 21457211 Oct  9 16:06 initramfs-4.13.5-300.fc27.x86_64.img
drwx------. 2 root root    16384 Oct  9 15:18 lost+found
-rw-r--r--. 1 root root   182704 Feb 11  2017 memtest86+-5.01
-rw-------. 1 root root  3879460 Sep 20 20:28 System.map-4.13.3-300.fc27.x86_64
-rw-------. 1 root root  3661616 Sep 28 19:14 System.map-4.13.4-300.fc27.x86_64
-rw-------. 1 root root  3664288 Oct  5 19:17 System.map-4.13.5-300.fc27.x86_64
-rwxr-xr-x. 1 root root  8233048 Oct  9 15:25 vmlinuz-0-rescue-3698883553db47b88ebe9db6b9a39aba
-rwxr-xr-x. 1 root root  8233048 Sep 20 20:29 vmlinuz-4.13.3-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7471192 Sep 28 19:15 vmlinuz-4.13.4-300.fc27.x86_64
-rwxr-xr-x. 1 root root  7475288 Oct  5 19:18 vmlinuz-4.13.5-300.fc27.x86_64

# tree aaaaa202093c472fa61803f6078f5d7b/ 3698883553db47b88ebe9db6b9a39aba/
aaaaa202093c472fa61803f6078f5d7b/
└── 4.13.3-300.fc27.x86_64
3698883553db47b88ebe9db6b9a39aba/
├── 4.13.4-300.fc27.x86_64
│   └── initrd
└── 4.13.5-300.fc27.x86_64
Comment 17 Kamil Páral 2017-10-09 10:18:15 EDT
For the record, we had additional discussion about this bug with Justin Forbes last week:

<kparal> jforbes: what do you think about security implications of not updating initramfs when dracut -f is called?
<kparal> or any other implications you can think of?
<kparal> I want to hear your thoughts in case new kernel installs generate initramfs fine, but tools trying to update it don't work
<jforbes> Possible missing drivers and such are the biggest issue
<jforbes> I mean it isn't really a workable long term solution, there are many reasons to need to update the initramfs
<kparal> aren't all drivers already present when it's generated with the new kernel installed? how would failing to update initramfs miss some drivers?
<kparal> also, what are your thoughts on this being a blocker (considering this case as described)?
<jforbes> Third part modules could be an issue, drivers not part of the kernel proper than dkms builds
<kparal> oh, I see. nvidia trying to build its module and such?
<kparal> virtualbox, etc?
<jforbes> I would be +1 to blocker
<jforbes> We also see a lot of failures for whatever reason fixed by rebuilding initramfs
<kparal> ok, thanks for details. we'll revisit this bug on our next meeting, hopefully with more testing done
Comment 18 Kamil Páral 2017-10-09 12:49:49 EDT
Discussed at blocker review meeting [1]:

punt (delay decision) - No really important breakage is known. We'll wait another week if we find something.

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-10-09/
Comment 20 Fedora Update System 2017-10-10 06:31:33 EDT
dracut-046-3.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-0fac2c45e5
Comment 21 Kamil Páral 2017-10-10 08:52:03 EDT
(In reply to Fedora Update System from comment #20)
> dracut-046-3.fc27 has been submitted as an update to Fedora 27.
> https://bodhi.fedoraproject.org/updates/FEDORA-2017-0fac2c45e5

Seems fixed.
Comment 22 Fedora Update System 2017-10-11 02:28:59 EDT
dracut-046-3.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-0fac2c45e5
Comment 23 František Zatloukal 2017-10-11 07:54:41 EDT
I've tested dracut-046-3, bug is fixed.
Comment 24 Fedora Update System 2017-10-12 05:00:40 EDT
dracut-046-3.1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-a986f116b0
Comment 25 Fedora Update System 2017-10-12 08:20:18 EDT
dracut-046-4.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-f7109f3f6c
Comment 26 Fedora Update System 2017-10-13 02:25:05 EDT
dracut-046-4.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-f7109f3f6c
Comment 27 Fedora Update System 2017-10-13 19:30:13 EDT
dracut-046-4.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.