Bug 1286317

Summary: With 4.5.2 Xen update, the system no longer boot
Product: [Fedora] Fedora Reporter: Romain <romain.pelissier>
Component: grub2Assignee: Peter Jones <pjones>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 28CC: adam.luchjenbroers, c.r, crobinso, dan.dunbar, dmoerner, jforbes, jonas.k, lantw44, lkundrak, mads, mathieu.tarral, m.a.young, mike, ngompa13, pjones, robinlee.sysu, siuskitman, sudikeru, thomas, virt-maint, zakhuff
Target Milestone: ---Keywords: Reopened, Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-28 23:32:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Romain 2015-11-28 00:57:32 UTC
Description of problem:
I have install 3 or 4 weeks ago fedora 23 and then setup xen using this HowTo: http://wiki.xen.org/wiki/Fedora_Host_Installation. All was working fine: the system boots correctly after the modification made on the /etc/boot-efi.cfg file.
Now with the 4.5.2 update for the xen kernel, the boot entries does not work at all, here is what I have right now:

Loading Xen 4.5.2 error: can't find command 'multiboot'. 
Loading Linux 4.2.6-301.fc23.x86_64 ... 
error: cana't find command 'module'. 
Loading Initial ramdisk ... 
error: can't find command 'module'.

Other entries for fedora with latest kernel version without Xen are working

Version-Release number of selected component (if applicable):
Fedora 23
Xen 4.5.2


How reproducible:
Everytime with the latest 4.5.2 xen kernel update


Steps to Reproduce:
1. Install Fedora 23
2. Install Xen according to http://wiki.xen.org/wiki/Fedora_Host_Installatio
3. Update the system

Actual results:
System unable to boot with xen entry


Expected results:
The boot/grub config should work with latest Xen kernel


Additional info:

Comment 1 Romain 2015-12-02 19:51:06 UTC
Additional information that could help you:
My fedora server is installed on a software raid and an efi bios:

[root@pdrvsrv002 ~]# cat /proc/mdstat
Personalities : [raid1]
md125 : active raid1 sdb2[1] sda2[0]
      205760 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 sdb1[1] sda1[0]
      512704 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sdb3[2] sda3[0]
      524271616 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 262144KB chunk

unused devices: <none>

[root@pdrvsrv002 ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/md127 fedora lvm2 a--  499.98g 165.38g


[root@pdrvsrv002 ~]# efibootmgr
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0000,0008,0009,0006,0007
Boot0000* Fedora
Boot0001* Fedora
Boot0006* Hard Drive
Boot0007* Network Card
Boot0008* UEFI OS
Boot0009* UEFI OS


[root@pdrvsrv002 boot]# ls -alrt
total 129142
drwx------.  3 root root    16384 Dec 31  1969 efi
-rw-r--r--.  1 root root      166 Oct  5 11:53 .vmlinuz.hmac-4.2.3-300.fc23.x86_64
-rw-------.  1 root root  3152021 Oct  5 11:58 System.map-4.2.3-300.fc23.x86_64
-rw-r--r--.  1 root root   160270 Oct  5 11:58 config-4.2.3-300.fc23.x86_64
-rwxr-xr-x.  1 root root  5977368 Oct  5 11:59 vmlinuz-4.2.3-300.fc23.x86_64
-rw-r--r--.  1 root root 17030360 Nov 10 17:35 xen-syms-4.5.2
-rw-r--r--.  1 root root   930213 Nov 10 17:35 xen-4.5.2.gz
lrwxrwxrwx.  1 root root       12 Nov 10 17:35 xen.gz -> xen-4.5.2.gz
lrwxrwxrwx.  1 root root       12 Nov 10 17:35 xen-4.5.gz -> xen-4.5.2.gz
-rw-r--r--.  1 root root      166 Nov 20 17:29 .vmlinuz.hmac-4.2.6-301.fc23.x86_64
-rw-------.  1 root root  3153903 Nov 20 17:35 System.map-4.2.6-301.fc23.x86_64
-rw-r--r--.  1 root root   160238 Nov 20 17:35 config-4.2.6-301.fc23.x86_64
-rwxr-xr-x.  1 root root  5980440 Nov 20 17:35 vmlinuz-4.2.6-301.fc23.x86_64
drwx------.  2 root root    12288 Nov 27 00:51 lost+found
dr-xr-xr-x. 17 root root      224 Nov 27 00:51 ..
-rw-r--r--.  1 root root 49442914 Nov 27 00:56 initramfs-0-rescue-3a4348fa23dc43d1805ac93fbde6305d.img
-rwxr-xr-x.  1 root root  5977368 Nov 27 00:56 vmlinuz-0-rescue-3a4348fa23dc43d1805ac93fbde6305d
-rw-r--r--.  1 root root 20145140 Nov 27 00:58 initramfs-4.2.3-300.fc23.x86_64.img
drwxr-xr-x.  3 root root     1024 Nov 27 09:42 grub2
-rw-r--r--.  1 root root   217619 Nov 27 09:43 initrd-plymouth.img
-rw-r--r--.  1 root root 19850862 Nov 27 09:44 initramfs-4.2.6-301.fc23.x86_64.img
drwxr-xr-x.  2 root root     1024 Nov 27 09:49 flask

[root@pdrvsrv002 fedora]# ls -alrt /boot/efi/EFI/fedora/
total 7972
-rwx------. 1 root root 1287032 Feb 17  2015 shim-fedora.efi
-rwx------. 1 root root 1293304 Feb 17  2015 shim.efi
-rwx------. 1 root root 1276224 Feb 17  2015 MokManager.efi
-rwx------. 1 root root     102 Feb 17  2015 BOOT.CSV
-rwx------. 1 root root 2230770 Nov 10 17:35 xen-4.5.2.efi
-rwx------. 1 root root 1005384 Nov 20 18:26 grubx64.efi
-rwx------. 1 root root 1005384 Nov 20 18:26 gcdx64.efi
drwx------. 4 root root    4096 Nov 27 00:55 ..
drwx------. 2 root root    4096 Nov 27 09:42 fonts
-rwx------. 1 root root    1024 Nov 27 11:39 grubenv
-rwx------. 1 root root   15194 Dec  1 20:24 grub.cfg.orig
-rwx------. 1 root root   15184 Dec  1 20:25 grub.cfg
drwx------. 3 root root    4096 Dec  1 21:08 .

If you need additional information, please let me know. I will try to test to changes on the config file of grub to see if I can make it works but it sound to me that something is missing fro grub to understand the "multiboot" and the "module" directive.

Comment 2 Romain 2015-12-03 03:55:58 UTC
Ok I found a fix:
- First in the grub menu, when I tried to to insmod to install some modules to be able to boot, I got some message indicating that the none of the modules I tried to install was found. So I search what rpm package provide the multiboot.mod module and the other ones. Found that the package grub2-efi-modules-2.02-0.24.fc23.x86_64 was not installed. One installed I found the files in /usr/lib/grub/x86_64-efi.
After another reboot, still no luck and tried this time again to install manually some modules and found a message saying that the modules could not be found in /EFI/fedora/x86_64-efi.
I reboot using the working fedora boot entry and there are no modules on folder in /boot/efi/EFI/fedora/.
I tried to create a symlink to /usr/lib/grub/x86_64-efi but for some reason it was not working (operation not permitted) So I simple copy the folder x86_64-efi to /boot/efi/EFI/fedora/. And this time I was able to boot Xen.
So I guess the package was not installed after the update (should it be a requirement?) and the path is either not correct or a symlink should be created and checked to be sure the modules are where grub is expecting them.

Comment 3 Michael Young 2015-12-03 09:01:23 UTC
I haven't found a good way to automate grub efi boot configuration so far. The problem is that multiboot doesn't work under efi (see
http://wiki.xenproject.org/wiki/Xen_EFI ). I currently have a workaround, with the file /boot/efi/EFI/fedora/xen.cfg containing
[global]
default=fedora

[fedora]
options=
kernel=vmlinuz root=/dev/mapper/vg1-root ro rd.lvm.lv=vg1/swap rd.lvm.lv=vg1/root rhgb quiet LANG=en_GB.UTF-8
ramdisk=initramfs.img

A grub configuration file to create an entry in the grub menu, /etc/grub.d/21_linux_xen_efi containing

#!/bin/sh
exec tail -n +3 $0
menuentry 'Fedora, with Xen EFI' --class fedora --class gnu-linux --class gnu --
class os --class xen $menuentry_id_option 'xen-gnulinux-simple-12217b01-b7d6-456
1-a50d-d1894122e90c' {
	insmod part_gpt
	insmod fat
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-ef
i=hd0,gpt2 --hint-baremetal=ahci0,gpt2  2E64-FD01
	else
	  search --no-floppy --fs-uuid --set=root 2E64-FD01
	fi
	chainloader /EFI/fedora/xen-4.5.2.efi
}

and by copying the current kernel and initramfs to the files vmlinuz and initramfs.img in /boot/efi/EFI/fedora/ though clearly this will need option and partition reference changes to work on your system, also space in the efi partition to hold the files.

Comment 4 sudikeru 2015-12-04 15:21:52 UTC
I have the same issue.
Workaround: downgrade packets grub2, grub2-efi, grub2-efi-modules, grub2-tools from version 1:2.02-0.24.fc23.x86_64 to 1:2.02-0.23.fc23.x86_64:
su -c 'dnf --allowerasing downgrade grub2-efi-modules'

Comment 5 Michael Young 2015-12-31 18:48:33 UTC
This looks like a result of the fix for CVE-2015-5281 (see Bug 1264103 ) so is really a grub issue rather than a xen issue.

Comment 6 W. Michael Petullo 2016-03-22 03:09:57 UTC
I had this problem too. I copied the EFI GRUB modules to /boot as described in comment #2, and this allowed EFI to boot Xen/Dom0. I am using the latest Fedora 23 packages.

Comment 7 Colin Rudakiewicz 2016-08-02 06:49:58 UTC
I too have this problem on Fedora 24 (Server). What is the best workaround? I will try http://wiki.xenproject.org/wiki/Xen_EFI (Xen as EFI binary).

Comment 8 Fedora End Of Life 2016-11-24 13:47:52 UTC
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 9 Fedora End Of Life 2016-12-20 16:22:05 UTC
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 10 Daniel Moerner 2017-01-08 20:23:13 UTC
I am requesting that this bug be reopened. I have the same problem on a fresh install of Fedora 25 with Xen 4.7.1-6.fc25. It has also been reproduced by another user on Fedora 24 (https://bugzilla.redhat.com/show_bug.cgi?id=1286317#c7). 

I see the following error:

Loading Xen 4.7.1 ...
error: can't find command 'multiboot'. 
Loading Linux 4.8.15-300.fc25.x86_64 ... 
error: can't find command 'module'. 
Loading initial ramdisk ... 
error: can't find command 'module'.

Press any key to continue...

It was suggested above (https://bugzilla.redhat.com/show_bug.cgi?id=1286317#c2) that you can fix this problem by copying /usr/lib/grub/x86_64-efi into /boot/efi/EFI/fedora. This does not solve the problem for me. It removes the errors, but the system still does not boot, hanging on "Loading initial ramdisk ...". 

I think the working solution is to chainload Xen and write up a simple xen.cfg, as explained in comment #3 above. But I'm not quite able to get it to work yet.

Comment 11 Michael Young 2017-01-08 23:15:21 UTC
(In reply to Daniel Moerner from comment #10)
> I think the working solution is to chainload Xen and write up a simple
> xen.cfg, as explained in comment #3 above. But I'm not quite able to get it
> to work yet.
I have had difficulties with that with the comment #3 solution with recent versions of grub2. The final F23 version of grub2 works but not at least some of the more recent ones. I keep meaning to investigate it enough to submit a meaningful bug report.

Comment 12 Neal Gompa 2017-06-13 22:55:57 UTC
This issue affects me with Fedora 25 and 26. Reopening.

Comment 13 Trowa Barton 2017-08-08 23:49:12 UTC
This bug still exists in Fedora 26 ( on Kernel 4.11.11-300.fc26.x86_64 now) and Xen 4.8.1

To expand on the previous workaround of installing the grub2-efi-modules package and moving the files manually to /EFI/fedora/x86_64-efi - I've found that I also have to change my UEFI to boot with CSM support to theoretically support booting from BIOS (even though I am still booting with "UEFI Only").

Comment 14 mathieu.tarral 2018-03-27 11:35:01 UTC
Hello,

any update here ?
I cannot boot on Xen and i opened a new bug report before realizing there was already this one.
https://bugzilla.redhat.com/show_bug.cgi?id=1554088

Can i work with Xen on Fedora without having to do hacks around GRUB2 and UEFI ?

Thanks.

Comment 15 Fedora End Of Life 2018-05-03 08:49:44 UTC
This message is a reminder that Fedora 26 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 26. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '26'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not
able to fix it before Fedora 26 is end of life. If you would still like
to see this bug fixed and are able to reproduce it against a later version
of Fedora, you are encouraged  change the 'version' to a later Fedora
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

Comment 16 mathieu.tarral 2018-05-03 08:58:55 UTC
Hi,

this bug is still here on Fedora 27

i will try to update to Fedora 28 in a week or two, and give you the feedback.

In the meantime, is there anything we can do ?
This bug has been here since a few years it seems... :/

Thank you.

Comment 17 Michael Young 2018-05-03 17:55:17 UTC
I found F27 usable for EFI boots. You need at least multiboot2.mod and relocator.mod from grub2-efi-x64-modules in /boot/efi/EFI/fedora/x86_64-efi/ after which the default grub built grub.cfg will boot xen dom0 though you have to skip a couple of warnings trying to load the non-existent module2 grub module, so for an unattended boot you need to remove the corresponding insmod lines in grub.cfg.

Comment 18 mathieu.tarral 2018-05-20 12:10:04 UTC
I upgraded to Fedora 28 last week.
Trying again to boot on the Xen GRUB entry gives me the following error:

https://photos.app.goo.gl/Q7ENf5urEYg65sJJ3

error: Unknown TPM error

So it's different than last time.
Any ideas guys ?

Thanks.

Comment 19 Fedora End Of Life 2018-05-29 11:24:07 UTC
Fedora 26 changed to end-of-life (EOL) status on 2018-05-29. Fedora 26
is no longer maintained, which means that it will not receive any
further security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 20 enenou 2018-08-12 13:49:14 UTC
As stated by the post above, this bug should be reopened, since it's met on Fedora 28 as well.

Comment 21 Adam Luchjenbroers 2018-12-28 10:29:54 UTC
Can confirm that this is still broken on Fedora 29.

Comment 22 Ben Cotton 2019-05-02 20:03:53 UTC
This message is a reminder that Fedora 28 is nearing its end of life.
On 2019-May-28 Fedora will stop maintaining and issuing updates for
Fedora 28. It is Fedora's policy to close all bug reports from releases
that are no longer maintained. At that time this bug will be closed as
EOL if it remains open with a Fedora 'version' of '28'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 28 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 23 Ben Cotton 2019-05-28 23:32:36 UTC
Fedora 28 changed to end-of-life (EOL) status on 2019-05-28. Fedora 28 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.