Bug 1286317 - With 4.5.2 Xen update, the system no longer boot
With 4.5.2 Xen update, the system no longer boot
Status: NEW
Product: Fedora
Classification: Fedora
Component: grub2 (Show other bugs)
26
x86_64 Linux
unspecified Severity medium
: ---
: ---
Assigned To: Peter Jones
Fedora Extras Quality Assurance
: Reopened, Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-11-27 19:57 EST by Romain
Modified: 2017-08-08 19:49 EDT (History)
18 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-12-20 11:22:05 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Romain 2015-11-27 19:57:32 EST
Description of problem:
I have install 3 or 4 weeks ago fedora 23 and then setup xen using this HowTo: http://wiki.xen.org/wiki/Fedora_Host_Installation. All was working fine: the system boots correctly after the modification made on the /etc/boot-efi.cfg file.
Now with the 4.5.2 update for the xen kernel, the boot entries does not work at all, here is what I have right now:

Loading Xen 4.5.2 error: can't find command 'multiboot'. 
Loading Linux 4.2.6-301.fc23.x86_64 ... 
error: cana't find command 'module'. 
Loading Initial ramdisk ... 
error: can't find command 'module'.

Other entries for fedora with latest kernel version without Xen are working

Version-Release number of selected component (if applicable):
Fedora 23
Xen 4.5.2


How reproducible:
Everytime with the latest 4.5.2 xen kernel update


Steps to Reproduce:
1. Install Fedora 23
2. Install Xen according to http://wiki.xen.org/wiki/Fedora_Host_Installatio
3. Update the system

Actual results:
System unable to boot with xen entry


Expected results:
The boot/grub config should work with latest Xen kernel


Additional info:
Comment 1 Romain 2015-12-02 14:51:06 EST
Additional information that could help you:
My fedora server is installed on a software raid and an efi bios:

[root@pdrvsrv002 ~]# cat /proc/mdstat
Personalities : [raid1]
md125 : active raid1 sdb2[1] sda2[0]
      205760 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 sdb1[1] sda1[0]
      512704 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sdb3[2] sda3[0]
      524271616 blocks super 1.2 [2/2] [UU]
      bitmap: 1/1 pages [4KB], 262144KB chunk

unused devices: <none>

[root@pdrvsrv002 ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/md127 fedora lvm2 a--  499.98g 165.38g


[root@pdrvsrv002 ~]# efibootmgr
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0000,0008,0009,0006,0007
Boot0000* Fedora
Boot0001* Fedora
Boot0006* Hard Drive
Boot0007* Network Card
Boot0008* UEFI OS
Boot0009* UEFI OS


[root@pdrvsrv002 boot]# ls -alrt
total 129142
drwx------.  3 root root    16384 Dec 31  1969 efi
-rw-r--r--.  1 root root      166 Oct  5 11:53 .vmlinuz.hmac-4.2.3-300.fc23.x86_64
-rw-------.  1 root root  3152021 Oct  5 11:58 System.map-4.2.3-300.fc23.x86_64
-rw-r--r--.  1 root root   160270 Oct  5 11:58 config-4.2.3-300.fc23.x86_64
-rwxr-xr-x.  1 root root  5977368 Oct  5 11:59 vmlinuz-4.2.3-300.fc23.x86_64
-rw-r--r--.  1 root root 17030360 Nov 10 17:35 xen-syms-4.5.2
-rw-r--r--.  1 root root   930213 Nov 10 17:35 xen-4.5.2.gz
lrwxrwxrwx.  1 root root       12 Nov 10 17:35 xen.gz -> xen-4.5.2.gz
lrwxrwxrwx.  1 root root       12 Nov 10 17:35 xen-4.5.gz -> xen-4.5.2.gz
-rw-r--r--.  1 root root      166 Nov 20 17:29 .vmlinuz.hmac-4.2.6-301.fc23.x86_64
-rw-------.  1 root root  3153903 Nov 20 17:35 System.map-4.2.6-301.fc23.x86_64
-rw-r--r--.  1 root root   160238 Nov 20 17:35 config-4.2.6-301.fc23.x86_64
-rwxr-xr-x.  1 root root  5980440 Nov 20 17:35 vmlinuz-4.2.6-301.fc23.x86_64
drwx------.  2 root root    12288 Nov 27 00:51 lost+found
dr-xr-xr-x. 17 root root      224 Nov 27 00:51 ..
-rw-r--r--.  1 root root 49442914 Nov 27 00:56 initramfs-0-rescue-3a4348fa23dc43d1805ac93fbde6305d.img
-rwxr-xr-x.  1 root root  5977368 Nov 27 00:56 vmlinuz-0-rescue-3a4348fa23dc43d1805ac93fbde6305d
-rw-r--r--.  1 root root 20145140 Nov 27 00:58 initramfs-4.2.3-300.fc23.x86_64.img
drwxr-xr-x.  3 root root     1024 Nov 27 09:42 grub2
-rw-r--r--.  1 root root   217619 Nov 27 09:43 initrd-plymouth.img
-rw-r--r--.  1 root root 19850862 Nov 27 09:44 initramfs-4.2.6-301.fc23.x86_64.img
drwxr-xr-x.  2 root root     1024 Nov 27 09:49 flask

[root@pdrvsrv002 fedora]# ls -alrt /boot/efi/EFI/fedora/
total 7972
-rwx------. 1 root root 1287032 Feb 17  2015 shim-fedora.efi
-rwx------. 1 root root 1293304 Feb 17  2015 shim.efi
-rwx------. 1 root root 1276224 Feb 17  2015 MokManager.efi
-rwx------. 1 root root     102 Feb 17  2015 BOOT.CSV
-rwx------. 1 root root 2230770 Nov 10 17:35 xen-4.5.2.efi
-rwx------. 1 root root 1005384 Nov 20 18:26 grubx64.efi
-rwx------. 1 root root 1005384 Nov 20 18:26 gcdx64.efi
drwx------. 4 root root    4096 Nov 27 00:55 ..
drwx------. 2 root root    4096 Nov 27 09:42 fonts
-rwx------. 1 root root    1024 Nov 27 11:39 grubenv
-rwx------. 1 root root   15194 Dec  1 20:24 grub.cfg.orig
-rwx------. 1 root root   15184 Dec  1 20:25 grub.cfg
drwx------. 3 root root    4096 Dec  1 21:08 .

If you need additional information, please let me know. I will try to test to changes on the config file of grub to see if I can make it works but it sound to me that something is missing fro grub to understand the "multiboot" and the "module" directive.
Comment 2 Romain 2015-12-02 22:55:58 EST
Ok I found a fix:
- First in the grub menu, when I tried to to insmod to install some modules to be able to boot, I got some message indicating that the none of the modules I tried to install was found. So I search what rpm package provide the multiboot.mod module and the other ones. Found that the package grub2-efi-modules-2.02-0.24.fc23.x86_64 was not installed. One installed I found the files in /usr/lib/grub/x86_64-efi.
After another reboot, still no luck and tried this time again to install manually some modules and found a message saying that the modules could not be found in /EFI/fedora/x86_64-efi.
I reboot using the working fedora boot entry and there are no modules on folder in /boot/efi/EFI/fedora/.
I tried to create a symlink to /usr/lib/grub/x86_64-efi but for some reason it was not working (operation not permitted) So I simple copy the folder x86_64-efi to /boot/efi/EFI/fedora/. And this time I was able to boot Xen.
So I guess the package was not installed after the update (should it be a requirement?) and the path is either not correct or a symlink should be created and checked to be sure the modules are where grub is expecting them.
Comment 3 Michael Young 2015-12-03 04:01:23 EST
I haven't found a good way to automate grub efi boot configuration so far. The problem is that multiboot doesn't work under efi (see
http://wiki.xenproject.org/wiki/Xen_EFI ). I currently have a workaround, with the file /boot/efi/EFI/fedora/xen.cfg containing
[global]
default=fedora

[fedora]
options=
kernel=vmlinuz root=/dev/mapper/vg1-root ro rd.lvm.lv=vg1/swap rd.lvm.lv=vg1/root rhgb quiet LANG=en_GB.UTF-8
ramdisk=initramfs.img

A grub configuration file to create an entry in the grub menu, /etc/grub.d/21_linux_xen_efi containing

#!/bin/sh
exec tail -n +3 $0
menuentry 'Fedora, with Xen EFI' --class fedora --class gnu-linux --class gnu --
class os --class xen $menuentry_id_option 'xen-gnulinux-simple-12217b01-b7d6-456
1-a50d-d1894122e90c' {
	insmod part_gpt
	insmod fat
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,gpt2 --hint-ef
i=hd0,gpt2 --hint-baremetal=ahci0,gpt2  2E64-FD01
	else
	  search --no-floppy --fs-uuid --set=root 2E64-FD01
	fi
	chainloader /EFI/fedora/xen-4.5.2.efi
}

and by copying the current kernel and initramfs to the files vmlinuz and initramfs.img in /boot/efi/EFI/fedora/ though clearly this will need option and partition reference changes to work on your system, also space in the efi partition to hold the files.
Comment 4 sudikeru 2015-12-04 10:21:52 EST
I have the same issue.
Workaround: downgrade packets grub2, grub2-efi, grub2-efi-modules, grub2-tools from version 1:2.02-0.24.fc23.x86_64 to 1:2.02-0.23.fc23.x86_64:
su -c 'dnf --allowerasing downgrade grub2-efi-modules'
Comment 5 Michael Young 2015-12-31 13:48:33 EST
This looks like a result of the fix for CVE-2015-5281 (see Bug 1264103 ) so is really a grub issue rather than a xen issue.
Comment 6 W. Michael Petullo 2016-03-21 23:09:57 EDT
I had this problem too. I copied the EFI GRUB modules to /boot as described in comment #2, and this allowed EFI to boot Xen/Dom0. I am using the latest Fedora 23 packages.
Comment 7 Colin Rudakiewicz 2016-08-02 02:49:58 EDT
I too have this problem on Fedora 24 (Server). What is the best workaround? I will try http://wiki.xenproject.org/wiki/Xen_EFI (Xen as EFI binary).
Comment 8 Fedora End Of Life 2016-11-24 08:47:52 EST
This message is a reminder that Fedora 23 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 23. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '23'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 23 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.
Comment 9 Fedora End Of Life 2016-12-20 11:22:05 EST
Fedora 23 changed to end-of-life (EOL) status on 2016-12-20. Fedora 23 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.
Comment 10 Daniel Moerner 2017-01-08 15:23:13 EST
I am requesting that this bug be reopened. I have the same problem on a fresh install of Fedora 25 with Xen 4.7.1-6.fc25. It has also been reproduced by another user on Fedora 24 (https://bugzilla.redhat.com/show_bug.cgi?id=1286317#c7). 

I see the following error:

Loading Xen 4.7.1 ...
error: can't find command 'multiboot'. 
Loading Linux 4.8.15-300.fc25.x86_64 ... 
error: can't find command 'module'. 
Loading initial ramdisk ... 
error: can't find command 'module'.

Press any key to continue...

It was suggested above (https://bugzilla.redhat.com/show_bug.cgi?id=1286317#c2) that you can fix this problem by copying /usr/lib/grub/x86_64-efi into /boot/efi/EFI/fedora. This does not solve the problem for me. It removes the errors, but the system still does not boot, hanging on "Loading initial ramdisk ...". 

I think the working solution is to chainload Xen and write up a simple xen.cfg, as explained in comment #3 above. But I'm not quite able to get it to work yet.
Comment 11 Michael Young 2017-01-08 18:15:21 EST
(In reply to Daniel Moerner from comment #10)
> I think the working solution is to chainload Xen and write up a simple
> xen.cfg, as explained in comment #3 above. But I'm not quite able to get it
> to work yet.
I have had difficulties with that with the comment #3 solution with recent versions of grub2. The final F23 version of grub2 works but not at least some of the more recent ones. I keep meaning to investigate it enough to submit a meaningful bug report.
Comment 12 Neal Gompa 2017-06-13 18:55:57 EDT
This issue affects me with Fedora 25 and 26. Reopening.
Comment 13 Trowa Barton 2017-08-08 19:49:12 EDT
This bug still exists in Fedora 26 ( on Kernel 4.11.11-300.fc26.x86_64 now) and Xen 4.8.1

To expand on the previous workaround of installing the grub2-efi-modules package and moving the files manually to /EFI/fedora/x86_64-efi - I've found that I also have to change my UEFI to boot with CSM support to theoretically support booting from BIOS (even though I am still booting with "UEFI Only").

Note You need to log in before you can comment on or make changes to this bug.