Bug 1765297

Summary: System become unbootable (Failed to start Switch Root) after the last upgrade
Product: [Fedora] Fedora Reporter: Mikhail <mikhail.v.gavrilov>
Component: grub2Assignee: Peter Jones <pjones>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 32CC: bobgus, cpp123, david.southwick, d.j, fadnix, fmartine, goodmirek, jbk, lkundrak, lnykryn, ltuan, luya_tfz, msekleta, pjdisclafani, pjones, ppywlkiqletw, rguerra.marin, s, systemd-maint, zbyszek
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: grub2-2.04-15.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1893974 (view as bug list) Environment:
Last Closed: 2020-05-01 04:06:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
rdsosreport.txt
none
photo of error message on the boot screen
none
dnf.log
none
Failing grub.cfg
none
Failing /boot/loader/entries
none
grub.cfg
none
loader folder
none
grubenv (might not be the original)
none
grubx64.efi none

Description Mikhail 2019-10-24 18:40:15 UTC 
Created attachment 1628928 [details]
rdsosreport.txt

Description of problem:
System become unbootable (Failed to start Switch Root) after the last upgrade

[    6.065641] localhost.localdomain systemd[1]: Reached target Switch Root.
[    6.067125] localhost.localdomain systemd[1]: Starting Switch Root...
[    6.075303] localhost.localdomain systemctl[600]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.
[    6.076535] localhost.localdomain systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
[    6.076767] localhost.localdomain systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
[    6.077252] localhost.localdomain systemd[1]: Failed to start Switch Root.
[    6.082124] localhost.localdomain systemd[1]: Startup finished in 10.530s (firmware) + 4.243s (loader) + 3.656s (kernel) + 0 (initrd) + 2.425s (userspace) = 20.855s.
[    6.082156] localhost.localdomain systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.
[    6.082218] localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[    6.083912] localhost.localdomain systemd[1]: Starting Setup Virtual Console...
[    6.167262] localhost.localdomain systemd[1]: systemd-vconsole-setup.service: Succeeded.
[    6.167862] localhost.localdomain systemd[1]: Started Setup Virtual Console.
[    6.167908] localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.167937] localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.169530] localhost.localdomain systemd[1]: Started Emergency Shell.
[    6.169686] localhost.localdomain audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=emergency comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.169836] localhost.localdomain systemd[1]: Reached target Emergency Mode.
[    6.187663] localhost.localdomain systemd[1]: Received SIGRTMIN+21 from PID 496 (plymouthd).
[    6.197725] localhost.localdomain systemd[1]: Received SIGRTMIN+21 from PID 496 (plymouthd).
[    6.198367] localhost.localdomain systemd[1]: plymouth-start.service: Succeeded.
[    6.198867] localhost.localdomain audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=plymouth-start comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Comment 1 Mikhail 2019-10-24 18:45:18 UTC 
Created attachment 1628929 [details]
photo of error message on the boot screen
Comment 2 Zbigniew Jędrzejewski-Szmek 2019-10-24 20:45:48 UTC 
> Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.

That seems to be the crucial error. If you use 'rd.break' on the kernel command line, and look
in /sysroot, do you see /etc/os-release or /usr/lib/os-release?
Comment 3 Mikhail 2019-10-25 04:00:35 UTC 
(In reply to Zbigniew Jędrzejewski-Szmek from comment #2)
> > Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.
> 
> That seems to be the crucial error. If you use 'rd.break' on the kernel
> command line, and look
> in /sysroot, do you see /etc/os-release or /usr/lib/os-release?

# ls -la /sysroot
total 0
drwxr-xr-x  2 root root  40 Jul 24 22:24 .
drwxr-xr-x 14 root root 460 Oct 25 03:54 ..

# ls -la /etc/os-release
lrwxrwxrwx 1 root root 14 Jul 24 22:24 /etc/os-release -> initrd-release

# ls -la /usr/lib/os-release
lrwxrwxrwx 1 root root 14 Jul 24 22:24 /usr/lib/os-release -> initrd-release
Comment 4 Zbigniew Jędrzejewski-Szmek 2019-10-25 05:46:23 UTC 
Hmm, what packages were installed in the upgrade that caused this issue? Nothing important changed
in systemd, so this is most likely related to some change in the boot loader or the kernel...
Comment 5 Mikhail 2019-10-25 18:54:07 UTC 
Created attachment 1629319 [details]
dnf.log

> Hmm, what packages were installed in the upgrade that caused this issue?
Comment 6 Kelly-Rand 2019-11-09 22:22:53 UTC 
I have this same issue on Fedora 31. I have never been able to boot directly into this install. I can boot via other OS grub menu.

I get the same output as in post #3.

This does not seem to effect systems not booting with the new BLS, at least those of mine.
Comment 7 Kelly-Rand 2019-11-10 13:07:28 UTC 
Fixed my issue with update of kernel options. 

CODE
grubby --update-kernel ALL --args="root=UUID=2ccbcb74-4636-424c-bd81-caf5e522b827 ro vconsole.keymap=us resume=UUID=6043208d-2609-4bf8-847d-8e6a54907826 psmouse.proto=bare  selinux=0 LANG=en_US.UTF-8"
END CODE

In prior update I had left out the "root=UUID=2cc....." portion of the arguments. 

My grubenv block now looks like this:

# GRUB Environment Block
kernelopts=root=UUID=2ccbcb74-4636-424c-bd81-caf5e522b827 ro vconsole.keymap=us resume=UUID=6043208d-2609-4bf8-847d-8e6a54907826 psmouse.proto=bare  selinux=0
boot_success=1
###

hash padded to 1024 bytes.

So I don't think the issue is in the initrd but rather may be in grubby not setting or getting the correct environment information. Since the OP is working with rawhide (f32) version this is the direction I would look.
Comment 8 Zbigniew Jędrzejewski-Szmek 2019-11-13 11:58:48 UTC 
> Created attachment 1629319 [details] dnf.log

2019-10-24T04:29:29Z DEBUG ---> Package grub2-common.noarch 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-common.noarch 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-ia32.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-ia32.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-ia32-cdboot.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-ia32-cdboot.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-x64.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-x64.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-x64-cdboot.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-efi-x64-cdboot.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-pc.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-pc.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-pc-modules.noarch 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-pc-modules.noarch 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools-efi.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools-efi.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools-extra.x86_64 1:2.04-2.fc32 will be upgraded
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools-extra.x86_64 1:2.04-3.fc32 will be an upgrade
2019-10-24T04:29:29Z DEBUG ---> Package grub2-tools-minimal.x86_64 1:2.04-2.fc32 will be upgraded

+ cat /proc/cmdline
+ sed -e 's/\(ftp:\/\/.*\):.*@/\1:*******@/g;s/\(cifs:\/\/.*\):.*@/\1:*******@/g;s/cifspass=[^ ]*/cifspass=*******/g;s/iscsi:.*@/iscsi:******@/g;s/rd.iscsi.password=[^ ]*/rd.iscsi.password=******/g;s/rd.iscsi.in.password=[^ ]*/rd.iscsi.in.password=******/g'
BOOT_IMAGE=(hd1,gpt2)/boot/vmlinuz-5.4.0-0.rc4.git1.1.fc32.x86_64

Indeed, it looks as if the commandline has gone empty.
Comment 9 Javier Martinez Canillas 2020-01-15 14:36:22 UTC 
(In reply to Mikhail from comment #0)
> Created attachment 1628928 [details]
> rdsosreport.txt
> 
> Description of problem:
> System become unbootable (Failed to start Switch Root) after the last upgrade

Could you please share the the following files: /boot/grub2/grubenv, /etc/grub2-efi.cfg (or /etc/grub2.cfg if is a legacy BIOS install) and all the files in the /boot/loader/entries directory.
Comment 10 Ben Cotton 2020-02-11 17:35:26 UTC 
This bug appears to have been reported against 'rawhide' during the Fedora 32 development cycle.
Changing version to 32.
Comment 11 David J. Fiddes 2020-02-28 14:10:09 UTC 
I've hit this problem after upgrading my laptop from f31 to f32. The logs mirror those above and indicate that no parameters were being passed to the kernel on boot. The /sysroot had no filesystem mounted in it to switch to.

The file /boot/grub2/grubenv was zero bytes when it failed. I'll attach the grub and loader entries shortly.

My machine is dual-boot with Windows 10 but doesn't see a lot of use so sees upgrades irregularly. It had been working well with fc31.

I was able to fix the machine by:
 - booting a live CD and mounting the partitions manually and chrooting in
 - recreating the grubenv file with "grub2-editenv create" 
 - rebuilding the config with "grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
Comment 12 David J. Fiddes 2020-02-28 14:11:30 UTC 
Created attachment 1666390 [details]
Failing grub.cfg
Comment 13 David J. Fiddes 2020-02-28 14:12:24 UTC 
Created attachment 1666391 [details]
Failing /boot/loader/entries
Comment 14 Luya Tshimbalanga 2020-03-08 10:49:06 UTC 
I hit a similar issue when upgrading from f31 to f32 and also doing a fresh installation. As the problem could be a potential blocker for the beta release of Fedora 32, changing the severity to high.
Comment 15 Fahad Alduraibi 2020-04-29 17:14:31 UTC 
I am facing this issue with the released version 32!
I just upgraded from 31 to 32 and now the system won't boot.

/sysroot is empty as the others have reported, however, when i try to mount my drive to the same folder i get an error telling me it is already mounted!
and after mounting it to a new folder the files appears also under sysroot.

I will now follow what David has suggested.


From the log file "rdsosreport.txt" i see the following lines, not sure if they are relevant:

[   36.502759] mypc systemd-gpt-auto-generator[510]: EFI loader partition unknown, exiting.
[   36.502811] mypc systemd-gpt-auto-generator[510]: (The boot loader did not set EFI variable LoaderDevicePartUUID.)



then later I get the "Failed to switch root" error:

[   36.627513] mypc systemd[1]: Reached target Switch Root.
[   36.628852] mypc systemd[1]: Starting Switch Root...
[   36.636212] mypc systemctl[548]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.
[   36.636781] mypc systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE
[   36.636856] mypc systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'.
[   36.636950] mypc systemd[1]: Failed to start Switch Root.
[   36.641469] mypc systemd[1]: Startup finished in 1.123s (kernel) + 0 (initrd) + 35.517s (userspace) = 36.641s.
[   36.641537] mypc audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=initrd-switch-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[   36.641565] mypc systemd[1]: initrd-switch-root.service: Triggering OnFailure= dependencies.
[   36.641973] mypc systemd[1]: Started Emergency Shell.
Comment 16 Javier Martinez Canillas 2020-04-29 18:00:56 UTC 
(In reply to Fahad Alduraibi from comment #15)
> I am facing this issue with the released version 32!
> I just upgraded from 31 to 32 and now the system won't boot.
> 
> /sysroot is empty as the others have reported, however, when i try to mount
> my drive to the same folder i get an error telling me it is already mounted!
> and after mounting it to a new folder the files appears also under sysroot.
>

Could you please share the the following files:

/boot/grub2/grubenv
/boot/efi/EFI/fedora/grub.cfg (or /boot/grub2/grub.cfg if is a legacy BIOS install)
the files in the /boot/loader/entries directory
Comment 17 Fahad Alduraibi 2020-04-29 18:32:27 UTC 
(In reply to Fahad Alduraibi from comment #15)
> /sysroot is empty as the others have reported, however, when i try to mount
> my drive to the same folder i get an error telling me it is already mounted!
> and after mounting it to a new folder the files appears also under sysroot.

Correction about it being already mounted, i think I was mistaken at that time since I tried it a few times later and didn't face the same issue.
Not sure what caused the previous behavior
Comment 18 Fahad Alduraibi 2020-04-29 18:34:30 UTC 
Created attachment 1683032 [details]
grub.cfg
Comment 19 Fahad Alduraibi 2020-04-29 18:35:09 UTC 
Created attachment 1683033 [details]
loader folder
Comment 20 Fahad Alduraibi 2020-04-29 18:37:01 UTC 
Created attachment 1683034 [details]
grubenv (might not be the original)

The attached file might not be the original since I have already ran the following command:

"grub2-editenv create"

and the file date and time are from the time when i ran the command. Not sure what it was before running this command.
Comment 21 Javier Martinez Canillas 2020-04-29 18:48:21 UTC 
Created attachment 1683036 [details]
grubx64.efi

Can you please test the attached grubx64.efi binary? This is a test build so is not signed, if you have Secure Boot enabled you need to disable it to test.
Comment 22 Fahad Alduraibi 2020-04-29 19:03:37 UTC 
(In reply to Javier Martinez Canillas from comment #21)
> Created attachment 1683036 [details]
> grubx64.efi
> 
> Can you please test the attached grubx64.efi binary? This is a test build so
> is not signed, if you have Secure Boot enabled you need to disable it to
> test.

Yes it works now.

But when can we expect a signed one? since I switch between Windows and Fedora, and windows won't boot without secure boot enabled. Or can I can sign the file myself the way i sign driver?
Comment 23 Javier Martinez Canillas 2020-04-29 19:10:41 UTC 
(In reply to Fahad Alduraibi from comment #22)
> (In reply to Javier Martinez Canillas from comment #21)
> > Created attachment 1683036 [details]
> > grubx64.efi
> > 
> > Can you please test the attached grubx64.efi binary? This is a test build so
> > is not signed, if you have Secure Boot enabled you need to disable it to
> > test.
> 
> Yes it works now.
> 

Thanks a lot for testing.

> But when can we expect a signed one? since I switch between Windows and
> Fedora, and windows won't boot without secure boot enabled. Or can I can
> sign the file myself the way i sign driver?

The signed build is in process now https://koji.fedoraproject.org/koji/taskinfo?taskID=43910644

I just wanted you to test and confirm that it solves the issue before doing the package update.
Comment 24 Fedora Update System 2020-04-29 20:05:42 UTC 
FEDORA-2020-50ee8dac68 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-50ee8dac68
Comment 25 Fedora Update System 2020-04-30 04:13:48 UTC 
FEDORA-2020-50ee8dac68 has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-50ee8dac68`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-50ee8dac68

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.
Comment 26 PJ D 2020-04-30 15:02:31 UTC 
Now that my system will only boot into emergency mode, the "dnf" command is not available to me. So I'm not going to be able to run 
sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-50ee8dac68

I imagine this bug fix will help people who have not yet upgraded their systems, but what is the solution for people who have already done the upgrade and ran into this bug and can only enter emergency mode? 

I hoped to attach files to this ticket, but as far as I can tell I don't have a /boot directory to include the files requested above (/boot/grub2/grubenv, /boot/efi/EFI/fedora/grub.cfg or /boot/grub2/grub.cfg, and the files in the /boot/loader/entries directory). Also, I haven't figured out how to write to a USB file from emergency mode, so I haven't included the rdosreport.txt, either.
Comment 27 Javier Martinez Canillas 2020-04-30 15:45:25 UTC 
(In reply to PJ D from comment #26)
> Now that my system will only boot into emergency mode, the "dnf" command is
> not available to me. So I'm not going to be able to run 
> sudo dnf upgrade --enablerepo=updates-testing
> --advisory=FEDORA-2020-50ee8dac68
> 
> I imagine this bug fix will help people who have not yet upgraded their
> systems, but what is the solution for people who have already done the
> upgrade and ran into this bug and can only enter emergency mode? 
> 
> I hoped to attach files to this ticket, but as far as I can tell I don't
> have a /boot directory to include the files requested above
> (/boot/grub2/grubenv, /boot/efi/EFI/fedora/grub.cfg or /boot/grub2/grub.cfg,
> and the files in the /boot/loader/entries directory). Also, I haven't
> figured out how to write to a USB file from emergency mode, so I haven't
> included the rdosreport.txt, either.

Are you booting with EFI or legacy BIOS? You may try mounting your boot and EFI System Partitions to get that information.

Also, since the bug is about grub not getting the kernel command line parameters for the entries, you could try editing the menu entries in GRUB (by pressing e when an entry is selected), add your command line parameters and then booting using Ctrl + x
Comment 28 Fedora Update System 2020-05-01 04:06:25 UTC 
FEDORA-2020-50ee8dac68 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.
Comment 29 PJ D 2020-05-01 12:03:58 UTC 
Per my system settings menu I am booting in UEFI mode. I'll try to figure out the settings I need and give the grub menu edit a try. Thank you. 

> 
> Are you booting with EFI or legacy BIOS? You may try mounting your boot and
> EFI System Partitions to get that information.
> 
> Also, since the bug is about grub not getting the kernel command line
> parameters for the entries, you could try editing the menu entries in GRUB
> (by pressing e when an entry is selected), add your command line parameters
> and then booting using Ctrl + x
Comment 30 Ruben Guerra Marin 2020-05-01 17:29:41 UTC 
Hi, I upgraded from fedora 31 to 32 (stable) today, and I encountered this exact issue. Somebody in a Fedora forum (https://ask.fedoraproject.org/t/fedora-32-switch-root-wont-start-after-upgrade/6634/20?u=rugebiker) mentioned that @rug Javier Martinez Canillas fix worked for him. I tried it and it also worked for me. A lot of other people in that fedoraproject thread are having the same issue, so this is still an ongoing thing.
Comment 31 Javier Martinez Canillas 2020-05-01 20:35:05 UTC 
(In reply to Ruben Guerra Marin from comment #30)
> Hi, I upgraded from fedora 31 to 32 (stable) today, and I encountered this
> exact issue. Somebody in a Fedora forum
> (https://ask.fedoraproject.org/t/fedora-32-switch-root-wont-start-after-
> upgrade/6634/20?u=rugebiker) mentioned that @rug Javier Martinez Canillas
> fix worked for him. I tried it and it also worked for me. A lot of other
> people in that fedoraproject thread are having the same issue, so this is
> still an ongoing thing.

The root of the issue is that we are storing the kernel command line parameters in the grubenv file as a $kernelopts variable, but that turned being quite fragile since it seems the grubenv file can get easily corrupted. Because all the boot entries use the same variable for the cmdline, then none of them will have a root parameter set if GRUB isn't able to read the variables from the grubenv.

There's a fallback kernel cmdline variable that's set in the grub.cfg file. But that wasn't the case in F30 and since the grub.cfg is never re-generated, you may not had that if variable if were upgrading since F30 or earlier.

For F33 we will change how the kernel cmdline is stored and have it in the BLS snippets instead of the grubenv file. That will prevent this issue to happen again.
Comment 32 Villy Kruse 2020-05-02 05:20:21 UTC 
(In reply to Javier Martinez Canillas from comment #31)


> 
> For F33 we will change how the kernel cmdline is stored and have it in the
> BLS snippets instead of the grubenv file. That will prevent this issue to
> happen again.

Just curious:  why was the boot command line put in the grubenv file in the first place?
Comment 33 Javier Martinez Canillas 2020-05-02 07:24:26 UTC 
(In reply to Villy Kruse from comment #32)
> (In reply to Javier Martinez Canillas from comment #31)
> 
> 
> > 
> > For F33 we will change how the kernel cmdline is stored and have it in the
> > BLS snippets instead of the grubenv file. That will prevent this issue to
> > happen again.
> 
> Just curious:  why was the boot command line put in the grubenv file in the
> first place?

The rationale was that it would allow the following:

* Add a boot entry by just dropping a file in /boot/loader/entries without needing to do any modification to this file. That's why the BLS file is a static file generated at kernel build time and shipped in the kernel package (/lib/modules/$kernel_version/bls.conf).
* Update the kernel cmdline for all the entries without needing to iterate over all installed BLS files and modify them.

But in the end it wasn't the best trade off, since the grubenv file is quite fragile. The file gets easily corrupted, the GRUB tools are very strict about the file size and format (it must have an exact size of 1024 bytes, must be padded with # chars, etc) and the tools refuse to modify them if it was edited by the users (due the strict file size and format mentioned).
Comment 34 Greg 2020-05-02 19:06:24 UTC 
Hi All,

I'm posting here the steps helped me in my case (HP 450 laptop, dual boot, ssd + hdd)

1. Boot from fedora 32 live USB, open terminal, become root
2. Determine efi, boot and root partitions ( fdisk -l; lsblk -f )
3. Prepare environment, chroot, fix, exit and reboot

su 

mkdir -pv /mnt/root

mount /dev/mapper/fedora_localhost--live-root /mnt/root
mount /dev/nvme0n1p6 /mnt/root/boot/
mount /dev/nvme0n1p2 /mnt/root/boot/efi

mount -o bind /dev /mnt/root/dev
mount -o bind /proc /mnt/root/proc
mount -o bind /sys /mnt/root/sys
mount -o bind /run /mnt/root/run

chroot /mnt/root

grub2-editenv /boot/efi/EFI/fedora/grubenv create
grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
dracut --force --regenerate-all

exit
reboot
Comment 35 Javier Martinez Canillas 2020-05-08 12:09:27 UTC 
*** Bug 1829172 has been marked as a duplicate of this bug. ***
Comment 36 Bob Gustafson 2020-05-20 23:21:52 UTC 
Comment #34 nice summary of commands

Unfortunately, it did not work for me.

Still thrown into emergency shell.

My disks are fine - I can mount them and look at data (copied off a few critical files).

I did the chroot, grub2-editenv, grub2-mkconfig, dracut and then reboot..

--

Originally on my upgrade 31->32, I was doing fine on the dns steps, but then when I did the reboot step to update, almost immediately I was thrown into the emergency shell.

It would be nice to be able to go back to a regular terminal window and re-do the dns steps.

---
I have another system that successfully upgraded to 32 and I have another bare metal that is now running 32. Perhaps I got overconfident..
Comment 37 Bob Gustafson 2020-05-20 23:37:01 UTC 
>>> It would be nice to be able to go back to a regular terminal window and re-do the dns steps.
>>>
>>> ---

Even better, it would be nice to have a 'sanity check' that would always be run at this critical moment to see if everything is a 'go' for the reboot step.

Before the reboot, everything can be undone, the dns steps can be redone, the files can be downloaded again (with perhaps a newer version of a problematic file..) and the 'sanity check' done again.

If the 'sanity check' comes up a no-go, at least one still has a running system. It is possible to go on-line and see if others have had the same problem.
Comment 38 David Southwick 2020-05-22 08:40:07 UTC 
This just happened to me from a normal dnf update on F32 stable, not sure what triggered it however. 
@Greg's steps from liveCD were able to recover the system however. 

Packages changes from DNF history:

Packages Altered:
    Install  kernel-5.6.12-300.fc32.x86_64                             @updates
    Install  kernel-core-5.6.12-300.fc32.x86_64                        @updates
    Install  kernel-devel-5.6.12-300.fc32.x86_64                       @updates
    Install  kernel-modules-5.6.12-300.fc32.x86_64                     @updates
    Install  kernel-modules-extra-5.6.12-300.fc32.x86_64               @updates
    Upgrade  abrt-2.14.2-1.fc32.x86_64                                 @updates
    Upgraded abrt-2.14.0-2.fc32.x86_64                                 @@System
    Upgrade  abrt-addon-ccpp-2.14.2-1.fc32.x86_64                      @updates
    Upgraded abrt-addon-ccpp-2.14.0-2.fc32.x86_64                      @@System
    Upgrade  abrt-addon-kerneloops-2.14.2-1.fc32.x86_64                @updates
    Upgraded abrt-addon-kerneloops-2.14.0-2.fc32.x86_64                @@System
    Upgrade  abrt-addon-pstoreoops-2.14.2-1.fc32.x86_64                @updates
    Upgraded abrt-addon-pstoreoops-2.14.0-2.fc32.x86_64                @@System
    Upgrade  abrt-addon-vmcore-2.14.2-1.fc32.x86_64                    @updates
    Upgraded abrt-addon-vmcore-2.14.0-2.fc32.x86_64                    @@System
    Upgrade  abrt-addon-xorg-2.14.2-1.fc32.x86_64                      @updates
    Upgraded abrt-addon-xorg-2.14.0-2.fc32.x86_64                      @@System
    Upgrade  abrt-cli-2.14.2-1.fc32.x86_64                             @updates
    Upgraded abrt-cli-2.14.0-2.fc32.x86_64                             @@System
    Upgrade  abrt-dbus-2.14.2-1.fc32.x86_64                            @updates
    Upgraded abrt-dbus-2.14.0-2.fc32.x86_64                            @@System
    Upgrade  abrt-desktop-2.14.2-1.fc32.x86_64                         @updates
    Upgraded abrt-desktop-2.14.0-2.fc32.x86_64                         @@System
    Upgrade  abrt-gui-2.14.2-1.fc32.x86_64                             @updates
    Upgraded abrt-gui-2.14.0-2.fc32.x86_64                             @@System
    Upgrade  abrt-gui-libs-2.14.2-1.fc32.x86_64                        @updates
    Upgraded abrt-gui-libs-2.14.0-2.fc32.x86_64                        @@System
    Upgrade  abrt-java-connector-1.1.5-1.fc32.x86_64                   @updates
    Upgraded abrt-java-connector-1.1.4-1.fc32.x86_64                   @@System
    Upgrade  abrt-libs-2.14.2-1.fc32.x86_64                            @updates
    Upgraded abrt-libs-2.14.0-2.fc32.x86_64                            @@System
    Upgrade  abrt-plugin-bodhi-2.14.2-1.fc32.x86_64                    @updates
    Upgraded abrt-plugin-bodhi-2.14.0-2.fc32.x86_64                    @@System
    Upgrade  abrt-retrace-client-2.14.2-1.fc32.x86_64                  @updates
    Upgraded abrt-retrace-client-2.14.0-2.fc32.x86_64                  @@System
    Upgrade  abrt-tui-2.14.2-1.fc32.x86_64                             @updates
    Upgraded abrt-tui-2.14.0-2.fc32.x86_64                             @@System
    Upgrade  adwaita-qt4-1.1.3-1.fc32.x86_64                           @updates
    Upgraded adwaita-qt4-1.1.2-1.fc32.x86_64                           @@System
    Upgrade  adwaita-qt5-1.1.3-1.fc32.x86_64                           @updates
    Upgraded adwaita-qt5-1.1.2-1.fc32.x86_64                           @@System
    Upgrade  autocorr-en-1:6.4.3.2-3.fc32.noarch                       @updates
    Upgraded autocorr-en-1:6.4.3.2-1.fc32.noarch                       @@System
    Upgrade  boost-chrono-1.69.0-18.fc32.x86_64                        @updates
    Upgraded boost-chrono-1.69.0-17.fc32.x86_64                        @@System
    Upgrade  boost-date-time-1.69.0-18.fc32.x86_64                     @updates
    Upgraded boost-date-time-1.69.0-17.fc32.x86_64                     @@System
    Upgrade  boost-filesystem-1.69.0-18.fc32.x86_64                    @updates
    Upgraded boost-filesystem-1.69.0-17.fc32.x86_64                    @@System
    Upgrade  boost-iostreams-1.69.0-18.fc32.x86_64                     @updates
    Upgraded boost-iostreams-1.69.0-17.fc32.x86_64                     @@System
    Upgrade  boost-locale-1.69.0-18.fc32.x86_64                        @updates
    Upgraded boost-locale-1.69.0-17.fc32.x86_64                        @@System
    Upgrade  boost-regex-1.69.0-18.fc32.x86_64                         @updates
    Upgraded boost-regex-1.69.0-17.fc32.x86_64                         @@System
    Upgrade  boost-system-1.69.0-18.fc32.x86_64                        @updates
    Upgraded boost-system-1.69.0-17.fc32.x86_64                        @@System
    Upgrade  boost-thread-1.69.0-18.fc32.x86_64                        @updates
    Upgraded boost-thread-1.69.0-17.fc32.x86_64                        @@System
    Upgrade  dnsmasq-2.81-3.fc32.x86_64                                @updates
    Upgraded dnsmasq-2.81-2.fc32.x86_64                                @@System
    Upgrade  firewalld-0.8.2-3.fc32.noarch                             @updates
    Upgraded firewalld-0.8.2-2.fc32.noarch                             @@System
    Upgrade  firewalld-filesystem-0.8.2-3.fc32.noarch                  @updates
    Upgraded firewalld-filesystem-0.8.2-2.fc32.noarch                  @@System
    Upgrade  gnome-abrt-1.3.2-1.fc32.x86_64                            @updates
    Upgraded gnome-abrt-1.3.1-1.fc32.x86_64                            @@System
    Upgrade  gnutls-3.6.13-2.fc32.x86_64                               @updates
    Upgraded gnutls-3.6.13-1.fc32.x86_64                               @@System
    Upgrade  gnutls-dane-3.6.13-2.fc32.x86_64                          @updates
    Upgraded gnutls-dane-3.6.13-1.fc32.x86_64                          @@System
    Upgrade  gnutls-utils-3.6.13-2.fc32.x86_64                         @updates
    Upgraded gnutls-utils-3.6.13-1.fc32.x86_64                         @@System
    Upgrade  json-c-0.13.1-12.fc32.x86_64                              @updates
    Upgraded json-c-0.13.1-11.fc32.x86_64                              @@System
    Upgrade  mesa-dri-drivers-20.0.7-1.fc32.x86_64                     @updates
    Upgraded mesa-dri-drivers-20.0.6-1.fc32.x86_64                     @@System
    Upgrade  mesa-filesystem-20.0.7-1.fc32.x86_64                      @updates
    Upgraded mesa-filesystem-20.0.6-1.fc32.x86_64                      @@System
    Upgrade  mesa-libEGL-20.0.7-1.fc32.x86_64                          @updates
    Upgraded mesa-libEGL-20.0.6-1.fc32.x86_64                          @@System
    Upgrade  mesa-libGL-20.0.7-1.fc32.x86_64                           @updates
    Upgraded mesa-libGL-20.0.6-1.fc32.x86_64                           @@System
    Upgrade  mesa-libgbm-20.0.7-1.fc32.x86_64                          @updates
    Upgraded mesa-libgbm-20.0.6-1.fc32.x86_64                          @@System
    Upgrade  mesa-libglapi-20.0.7-1.fc32.x86_64                        @updates
    Upgraded mesa-libglapi-20.0.6-1.fc32.x86_64                        @@System
    Upgrade  mesa-libxatracker-20.0.7-1.fc32.x86_64                    @updates
    Upgraded mesa-libxatracker-20.0.6-1.fc32.x86_64                    @@System
    Upgrade  mesa-vulkan-drivers-20.0.7-1.fc32.x86_64                  @updates
    Upgraded mesa-vulkan-drivers-20.0.6-1.fc32.x86_64                  @@System
    Upgrade  nftables-1:0.9.3-3.fc32.x86_64                            @updates
    Upgraded nftables-1:0.9.3-2.fc32.x86_64                            @@System
    Upgrade  openconnect-8.10-1.fc32.x86_64                            @updates
    Upgraded openconnect-8.05-2.fc32.x86_64                            @@System
    Upgrade  osinfo-db-20200515-1.fc32.noarch                          @updates
    Upgraded osinfo-db-20200325-1.fc32.noarch                          @@System
    Upgrade  pcre2-10.35-1.fc32.x86_64                                 @updates
    Upgraded pcre2-10.34-9.fc32.x86_64                                 @@System
    Upgrade  pcre2-syntax-10.35-1.fc32.noarch                          @updates
    Upgraded pcre2-syntax-10.34-9.fc32.noarch                          @@System
    Upgrade  pcre2-utf16-10.35-1.fc32.x86_64                           @updates
    Upgraded pcre2-utf16-10.34-9.fc32.x86_64                           @@System
    Upgrade  pcre2-utf32-10.35-1.fc32.x86_64                           @updates
    Upgraded pcre2-utf32-10.34-9.fc32.x86_64                           @@System
    Upgrade  python3-abrt-2.14.2-1.fc32.x86_64                         @updates
    Upgraded python3-abrt-2.14.0-2.fc32.x86_64                         @@System
    Upgrade  python3-abrt-addon-2.14.2-1.fc32.noarch                   @updates
    Upgraded python3-abrt-addon-2.14.0-2.fc32.noarch                   @@System
    Upgrade  python3-firewall-0.8.2-3.fc32.noarch                      @updates
    Upgraded python3-firewall-0.8.2-2.fc32.noarch                      @@System
    Upgrade  python3-libreport-2.13.1-4.fc32.x86_64                    @updates
    Upgraded python3-libreport-2.12.0-3.fc32.x86_64                    @@System
    Upgrade  python3-nftables-1:0.9.3-3.fc32.x86_64                    @updates
    Upgraded python3-nftables-1:0.9.3-2.fc32.x86_64                    @@System
    Upgrade  tcl-1:8.6.10-2.fc32.x86_64                                @updates
    Upgraded tcl-1:8.6.10-1.fc32.x86_64                                @@System
    Upgrade  tpm2-tss-2.4.1-1.fc32.x86_64                              @updates
    Upgraded tpm2-tss-2.4.0-1.fc32.x86_64                              @@System
    Upgrade  vim-common-2:8.2.752-1.fc32.x86_64                        @updates
    Upgraded vim-common-2:8.2.735-1.fc32.x86_64                        @@System
    Upgrade  vim-enhanced-2:8.2.752-1.fc32.x86_64                      @updates
    Upgraded vim-enhanced-2:8.2.735-1.fc32.x86_64                      @@System
    Upgrade  vim-filesystem-2:8.2.752-1.fc32.noarch                    @updates
    Upgraded vim-filesystem-2:8.2.735-1.fc32.noarch                    @@System
    Upgrade  vim-minimal-2:8.2.752-1.fc32.x86_64                       @updates
    Upgraded vim-minimal-2:8.2.735-1.fc32.x86_64                       @@System
    Upgrade  codium-1.45.1-1589539729.el7.x86_64                       @gitlab.com_paulcarroty_vscodium_repo
    Upgraded codium-1.45.0-1588934872.el7.x86_64                       @@System
    Removed  kernel-5.6.7-300.fc32.x86_64                              @@System
    Removed  kernel-core-5.6.7-300.fc32.x86_64                         @@System
    Removed  kernel-devel-5.6.7-300.fc32.x86_64                        @@System
    Removed  kernel-modules-5.6.7-300.fc32.x86_64                      @@System
    Removed  kernel-modules-extra-5.6.7-300.fc32.x86_64                @@System
    Removed  kmod-nvidia-5.6.7-300.fc32.x86_64-3:440.82-1.fc32.x86_64  @@System
    Removed  kmod-wl-5.6.7-300.fc32.x86_64-6.30.223.271-32.fc32.x86_64 @@System
Scriptlet output:
   1 Warning: The unit file, source configuration file or drop-ins of abrt-journal-core.service changed on disk. Run 'systemctl daemon-reload' to reload units.
   2 Warning: The unit file, source configuration file or drop-ins of abrt-vmcore.service changed on disk. Run 'systemctl daemon-reload' to reload units.
   3 Warning: The unit file, source configuration file or drop-ins of abrt-oops.service changed on disk. Run 'systemctl daemon-reload' to reload units.
   4 Warning: The unit file, source configuration file or drop-ins of abrt-xorg.service changed on disk. Run 'systemctl daemon-reload' to reload units.
   5 Warning: The unit file, source configuration file or drop-ins of firewalld.service changed on disk. Run 'systemctl daemon-reload' to reload units.
   6 Warning: The unit file, source configuration file or drop-ins of abrtd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
   7 dkms: removing: anbox-ashmem 1 (5.6.7-300.fc32.x86_64) (x86_64)
   8 
   9 -------- Uninstall Beginning --------
  10 Module:  anbox-ashmem
  11 Version: 1
  12 Kernel:  5.6.7-300.fc32.x86_64 (x86_64)
  13 -------------------------------------
  14 
  15 Status: Before uninstall, this module version was ACTIVE on this kernel.
  16 Removing any linked weak-modules
  17 
  18 ashmem_linux.ko.xz:
  19  - Uninstallation
  20    - Deleting from: /lib/modules/5.6.7-300.fc32.x86_64/extra/
  21  - Original module
  22    - No original module was found for this module on this kernel.
  23    - Use the dkms install command to reinstall any previous module version.
  24 
  25 depmod....
  26 
  27 DKMS: uninstall completed.
  28 dkms: removing: anbox-binder 1 (5.6.7-300.fc32.x86_64) (x86_64)
  29 
  30 -------- Uninstall Beginning --------
  31 Module:  anbox-binder
  32 Version: 1
  33 Kernel:  5.6.7-300.fc32.x86_64 (x86_64)
  34 -------------------------------------
  35 
  36 Status: Before uninstall, this module version was ACTIVE on this kernel.
  37 Removing any linked weak-modules
  38 
  39 binder_linux.ko.xz:
  40  - Uninstallation
  41    - Deleting from: /lib/modules/5.6.7-300.fc32.x86_64/extra/
  42  - Original module
  43    - No original module was found for this module on this kernel.
  44    - Use the dkms install command to reinstall any previous module version.
  45 
  46 depmod....
  47 
  48 DKMS: uninstall completed.
  49 warning: file /lib/modules/5.6.7-300.fc32.x86_64/updates: remove failed: No such file or directory
  50 grub2-editenv: error: invalid environment block.
  51 dkms: running auto installation service for kernel 5.6.12-300.fc32.x86_64
  52 
  53 Kernel preparation unnecessary for this kernel.  Skipping...
  54 
  55 Building module:
  56 cleaning build area....
  57 make -j12 KERNELRELEASE=5.6.12-300.fc32.x86_64 all KERNEL_SRC=/lib/modules/5.6.12-300.fc32.x86_64/build....
  58 cleaning build area....
  59 
  60 DKMS: build completed.
  61 
  62 ashmem_linux.ko.xz:
  63 Running module version sanity check.
  64  - Original module
  65    - No original module exists within this kernel
  66  - Installation
  67    - Installing to /lib/modules/5.6.12-300.fc32.x86_64/extra/
  68 Adding any weak-modules
  69 
  70 depmod.....
  71 
  72 DKMS: install completed.
  73 
  74 Kernel preparation unnecessary for this kernel.  Skipping...
  75 
  76 Building module:
  77 cleaning build area....
  78 make -j12 KERNELRELEASE=5.6.12-300.fc32.x86_64 all KERNEL_SRC=/lib/modules/5.6.12-300.fc32.x86_64/build....
  79 cleaning build area....
  80 
  81 DKMS: build completed.
  82 
  83 binder_linux.ko.xz:
  84 Running module version sanity check.
  85  - Original module
  86    - No original module exists within this kernel
  87  - Installation
  88    - Installing to /lib/modules/5.6.12-300.fc32.x86_64/extra/
  89 Adding any weak-modules
  90 
  91 depmod.....
  92 
  93 DKMS: install completed.
  94  Done. 
  95 dkms: running auto installation service for kernel 5.6.12-300.fc32.x86_64
  96  Done.
Comment 39 Bob Gustafson 2020-05-23 17:54:51 UTC 
(In reply to Bob Gustafson from comment #37)
> >>> It would be nice to be able to go back to a regular terminal window and re-do the dns steps.
> >>>
> >>> ---
> 
> Even better, it would be nice to have a 'sanity check' that would always be
> run at this critical moment to see if everything is a 'go' for the reboot
> step.
> 
> Before the reboot, everything can be undone, the dns steps can be redone,
> the files can be downloaded again (with perhaps a newer version of a
> problematic file..) and the 'sanity check' done again.
> 
> If the 'sanity check' comes up a no-go, at least one still has a running
> system. It is possible to go on-line and see if others have had the same
> problem.

Most of my above wish list can be done from a USB live boot (except for the actual crash recovery).

My problem was that I had a loop mount active before I started my dnf upgrade process. At some point, the file that was being loop mounted was deleted.

When I came time to reboot and install the 32 system, the loop mount failed. Apparently, a failing mount is a Dependency for the new install and that is what failed it and dumped me into emergency.

The fix was just to comment out that loop mount line in /etc/fstab. All this can be done while in the USB live system. I will write this up in a "Common Problems" note.
Comment 40 Mirek Svoboda 2020-06-03 20:54:48 UTC 
This issue just happened to me yesterday, on Fedora 31, after dnf update, using stable updates repo.
I have collected whole /boot, /etc and output of dnf history info before fixing the issue.
Does it make sense to post the information I have collected here?
Or should I do anything else to help preventing the issue happening to others?
Comment 41 Bob Gustafson 2020-06-03 22:07:55 UTC 
If you get dumped into the emergency shell, type in the root password and when you get a prompt:

# journalctl -b | grep fail

Look at the last fail lines for a clue.

If there is the word 'dependency' in the line, look at previous fail lines for a possible clue.

My problem was a loop mount in /etc/fstab which could not find its file.
I commented out that line in /etc/fstab and the next boot was able to move past that point.

The dependency on finding all mounts seems to be a feature of the upgrade reboot.
Comment 42 Javier Martinez Canillas 2020-06-03 22:35:59 UTC 
(In reply to Mirek Svoboda from comment #40)
> This issue just happened to me yesterday, on Fedora 31, after dnf update,
> using stable updates repo.
> I have collected whole /boot, /etc and output of dnf history info before
> fixing the issue.
> Does it make sense to post the information I have collected here?

Yes, please share your grubenv, grub.cfg and /boot/loader/entries/*.conf files. Was this on an EFI or legacy BIOS install?
Comment 43 Villy Kruse 2020-06-04 05:45:48 UTC 
(In reply to Javier Martinez Canillas from comment #42)
> (In reply to Mirek Svoboda from comment #40)
> > This issue just happened to me yesterday, on Fedora 31, after dnf update,
> > using stable updates repo.
> > I have collected whole /boot, /etc and output of dnf history info before
> > fixing the issue.
> > Does it make sense to post the information I have collected here?
> 
> Yes, please share your grubenv, grub.cfg and /boot/loader/entries/*.conf
> files. Was this on an EFI or legacy BIOS install?

Also show the output of 

  lsblk -f