Bug 1486002 - grub2-mkconfig does not work if xen.gz is installed.
Summary: grub2-mkconfig does not work if xen.gz is installed.
Keywords:
Status: NEW
Alias: None
Product: Fedora
Classification: Fedora
Component: grub2
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Peter Jones
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
: 1498300 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-08-28 16:50 UTC by Konrad Rzeszutek Wilk
Modified: 2019-05-14 00:26 UTC (History)
22 users (show)

Fixed In Version: grub2-2.02-19.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-11 03:04:13 UTC


Attachments (Terms of Use)
insmod modules xen needs when it needs them. (1.17 KB, patch)
2017-10-19 15:33 UTC, Peter Jones
no flags Details | Diff

Description Konrad Rzeszutek Wilk 2017-08-28 16:50:56 UTC
Description of problem:

If I have xen installed, doing grub2-mkconfig won't produce a new configuration file.


[root@tst063 ~]# rpm -q grub2-efi
grub2-efi-2.02-8.fc27.x86_64
[root@tst063 ~]# ls -al /boot/
total 110400
dr-xr-xr-x.  6 root root     4096 Aug 28 10:42 .
dr-xr-xr-x. 18 root root      238 Aug 28 11:39 ..
-rw-r--r--.  1 root root   190334 Aug 15 15:20 config-4.13.0-0.rc5.git1.1.fc27.x86_64
drwx------.  4 root root    16384 Dec 31  1969 efi
drwxr-xr-x.  2 root root     4096 Aug 28 10:42 flask
drwxr-xr-x.  3 root root     4096 Aug 28 10:40 grub2
-rw-------.  1 root root 69080768 Aug 28 10:40 initramfs-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2.img
-rw-------.  1 root root 24065112 Aug 28 10:41 initramfs-4.13.0-0.rc5.git1.1.fc27.x86_64.img
drwx------.  2 root root    16384 Aug 28 10:36 lost+found
-rw-------.  1 root root  3658588 Aug 15 15:20 System.map-4.13.0-0.rc5.git1.1.fc27.x86_64
-rwxr-xr-x.  1 root root  7467096 Aug 28 10:40 vmlinuz-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2
-rwxr-xr-x.  1 root root  7467096 Aug 15 15:21 vmlinuz-4.13.0-0.rc5.git1.1.fc27.x86_64
-rw-r--r--.  1 root root      176 Aug 15 15:13 .vmlinuz-4.13.0-0.rc5.git1.1.fc27.x86_64.hmac
-rw-r--r--.  1 root root     1193 Aug 23 16:40 xen-4.9.0.config
-rw-r--r--.  1 root root  1047166 Aug 23 16:47 xen-4.9.0.gz
lrwxrwxrwx.  1 root root       12 Aug 23 16:47 xen-4.9.gz -> xen-4.9.0.gz
lrwxrwxrwx.  1 root root       12 Aug 23 16:47 xen.gz -> xen-4.9.0.gz
[root@tst063 ~]# grub2-mkconfig -o /tmp/test
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.13.0-0.rc5.git1.1.fc27.x86_64
Found initrd image: /boot/initramfs-4.13.0-0.rc5.git1.1.fc27.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2
Found initrd image: /boot/initramfs-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2.img
[root@tst063 ~]# cat /tmp/test
cat: /tmp/test: No such file or directory


Just to double-check:

[root@tst063 ~]# grub2-mkconfig | tail
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.13.0-0.rc5.git1.1.fc27.x86_64
Found initrd image: /boot/initramfs-4.13.0-0.rc5.git1.1.fc27.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2
Found initrd image: /boot/initramfs-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2.img
        else
          search --no-floppy --fs-uuid --set=root 166243e1-7050-490c-9b0a-eb40a33a3be0
        fi
        linuxefi /vmlinuz-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2 root=/dev/mapper/fedora_tst063-root ro rd.lvm.lv=fedora_tst063/root rd.lvm.lv=fedora_tst063/swap loglevel=8 console=ttyS0,115200 
        initrdefi /initramfs-0-rescue-ec082ee24aea41b9b16aca52a6d10cc2.img
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

[root@tst063 ~]# 

And it seems to stop there.

Version-Release number of selected component (if applicable):

grub2-efi-2.02-8.fc27.x86_64

How reproducible:

100%
Steps to Reproduce:
1. Install Fedora rawhide (Fedora-Server-dvd-x86_64-27-20170817.n.3.iso)
2. yum install xen
3. grub2-mkconfig -o /etc/grub2-efi.cfg


Actual results:

Same grub2-efi.cfg entries

Expected results:

An Xen entry in the grub2-efi.cfg

Additional info:

Comment 1 Konrad Rzeszutek Wilk 2017-08-28 17:34:59 UTC
Ugh, also happens with upstream..

Author: Pete Batard <pete@akeo.ie>
Date:   Mon Aug 7 16:23:12 2017 +0100

    io: add a GRUB_GZ prefix to gzio specific defines
    
    * This is done to avoid a conflict with a PACKED define in the EDK2

Comment 2 Konrad Rzeszutek Wilk 2017-08-28 17:38:51 UTC
Ah,

Commenting out 
$grub_file --is-arm64-efi $current_xen

makes it work

Hmm. why does it have '$' in front?


Removing the '$' fixes it as well.

Comment 3 Konrad Rzeszutek Wilk 2017-08-28 17:48:14 UTC
err, scratch the '$' - that is with the upstream version (which is built without the 'grub2' prefix). For the rawhide version, the workoarund is to comment out the '$grub_file --is-arm64-efi $current_xen'

Or:

    if ($grub_file --is-arm64-efi $current_xen); then
        xen_loader="xen_hypervisor"
        module_loader="xen_module"
    else
        xen_loader="multiboot"
        module_loader="module"
    fi


Let me post a patch upstream for the above fix.

Comment 4 Konrad Rzeszutek Wilk 2017-08-28 19:16:55 UTC
Posted at http://lists.gnu.org/archive/html/grub-devel/2017-08/msg00073.html

Also it may make sense to modify the grub.macros for GRUB_MODULES to also
have "multiboot multiboot2" in it.

And in the efi_modules for aarch64 to also have "xen_boot". like this:

[root@tst063 ~]# more grub2.spec.diff 
--- grub.macros.orig    2017-08-28 14:54:23.511123738 -0400
+++ grub.macros 2017-08-28 14:54:56.573933975 -0400
@@ -87,7 +87,7 @@
 
 ### fixme
 %ifarch aarch64 %{arm}
-%global efi_modules " http linux "
+%global efi_modules " http linux xen_boot"
 %else
 %global efi_modules " backtrace http linuxefi usb usbserial_common usbserial_pl2303 usbserial_ftdi usbserial_usbdebug "
 %endif
@@ -325,7 +325,7 @@
                password_pbkdf2 png reboot                      \\\
                search search_fs_uuid search_fs_file            \\\
                search_label serial sleep syslinuxcfg test tftp \\\
-               video xfs"                                      \
+               video xfs multiboot multiboot2"                 \
 GRUB_MODULES+=%{efi_modules}                                   \
 %{expand:%%{mkimage %{1} %{2} %{3} %{4}}}                      \
 %{nil}


And naturally expand the grub.patches with the patches.

Comment 5 Konrad Rzeszutek Wilk 2017-08-30 00:28:29 UTC
Update:
http://lists.gnu.org/archive/html/grub-devel/2017-08/msg00095.html

Comment 6 Fedora Blocker Bugs Application 2017-09-04 11:58:30 UTC
Proposed as a Blocker for 27-final by Fedora user konradr using the blocker tracking app because:

 Fails QA:Testcase_Boot_Methods_Xen_Para_Virt

Comment 7 Kamil Páral 2017-09-04 17:16:17 UTC
Discussed during blocker review [1]:

AcceptedBlocker (Beta) - we're happy to accept Konrad's assertion that this violates the Final criterion "The release must boot successfully as Xen DomU with releases providing a functional, supported Xen Dom0 and widely used cloud providers utilizing Xen"

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-09-04/

Comment 8 Konrad Rzeszutek Wilk 2017-09-13 15:44:57 UTC
They are upstream (git://git.sv.gnu.org/grub.git)

b4d709b6e Use grub-file to figure out whether multiboot2 should be used for Xen.gz
a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for aarch64

Peter, if there is way for me to submit the patch to grub RPM repo I can try doing that if that would be easier?

Comment 9 Konrad Rzeszutek Wilk 2017-09-26 00:11:13 UTC
Still an issue with Fedora 27 Beta 1.2

Comment 10 Konrad Rzeszutek Wilk 2017-09-29 14:46:46 UTC
Still an issue with Fedora-Server-dvd-x86_64-27_Beta-1.3.iso

Comment 11 Robin Lee 2017-10-05 09:39:51 UTC
(In reply to Konrad Rzeszutek Wilk from comment #8)
> They are upstream (git://git.sv.gnu.org/grub.git)
> 
> b4d709b6e Use grub-file to figure out whether multiboot2 should be used for
> Xen.gz
> a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for
> aarch64
> 
> Peter, if there is way for me to submit the patch to grub RPM repo I can try
> doing that if that would be easier?

You may try to send a PR on https://src.fedoraproject.org/rpms/grub/.

Comment 12 Robin Lee 2017-10-05 09:41:07 UTC
*** Bug 1498300 has been marked as a duplicate of this bug. ***

Comment 13 Robin Lee 2017-10-05 09:42:57 UTC
(In reply to Robin Lee from comment #11)
> (In reply to Konrad Rzeszutek Wilk from comment #8)
> > They are upstream (git://git.sv.gnu.org/grub.git)
> > 
> > b4d709b6e Use grub-file to figure out whether multiboot2 should be used for
> > Xen.gz
> > a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for
> > aarch64
> > 
> > Peter, if there is way for me to submit the patch to grub RPM repo I can try
> > doing that if that would be easier?
> 
> You may try to send a PR on https://src.fedoraproject.org/rpms/grub/.

That should be https://src.fedoraproject.org/rpms/grub2/

Comment 14 Konrad Rzeszutek Wilk 2017-10-06 21:15:28 UTC
First ever pull request!
https://src.fedoraproject.org/rpms/grub2/pull-request/1

Lets see how that goes

Comment 15 Peter Jones 2017-10-19 15:21:47 UTC
(In reply to Konrad Rzeszutek Wilk from comment #4)
> Posted at http://lists.gnu.org/archive/html/grub-devel/2017-08/msg00073.html
> 
> Also it may make sense to modify the grub.macros for GRUB_MODULES to also
> have "multiboot multiboot2" in it.
> 
> And in the efi_modules for aarch64 to also have "xen_boot". like this:
> 
> [root@tst063 ~]# more grub2.spec.diff 
> --- grub.macros.orig    2017-08-28 14:54:23.511123738 -0400
> +++ grub.macros 2017-08-28 14:54:56.573933975 -0400
> @@ -87,7 +87,7 @@
>  
>  ### fixme
>  %ifarch aarch64 %{arm}
> -%global efi_modules " http linux "
> +%global efi_modules " http linux xen_boot"
>  %else
>  %global efi_modules " backtrace http linuxefi usb usbserial_common
> usbserial_pl2303 usbserial_ftdi usbserial_usbdebug "
>  %endif
> @@ -325,7 +325,7 @@
>                 password_pbkdf2 png reboot                      \\\
>                 search search_fs_uuid search_fs_file            \\\
>                 search_label serial sleep syslinuxcfg test tftp \\\
> -               video xfs"                                      \
> +               video xfs multiboot multiboot2"                 \
>  GRUB_MODULES+=%{efi_modules}                                   \
>  %{expand:%%{mkimage %{1} %{2} %{3} %{4}}}                      \
>  %{nil}
> 
> 
> And naturally expand the grub.patches with the patches.

No, we absolutely cannot do either of these.  This would be a CVE, as none of these modules have signature checking.

We can do the 20_linux_xen.in fix, but the modules need to be loaded dynamically when SB is disabled.

Comment 16 Peter Jones 2017-10-19 15:33:37 UTC
Created attachment 1340817 [details]
insmod modules xen needs when it needs them.

Note that you'll still need to have grub2-efi-aa64-modules installed (and probably do grub2-install bits to put them in /boot/grub2 or whatnot as well.)

Comment 17 Peter Jones 2017-10-19 15:34:40 UTC
> No, we absolutely cannot do either of these.  This would be a CVE, as none
> of these modules have signature checking.

Actually I'm not sure that's strictly true.

> We can do the 20_linux_xen.in fix, but the modules need to be loaded
> dynamically when SB is disabled.

But I still prefer this method.

Comment 18 Konrad Rzeszutek Wilk 2017-10-19 19:06:07 UTC
You got an RPM I can test?

Comment 19 Fedora Update System 2017-10-24 17:43:08 UTC
grub2-2.02-19.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd36ce6cfd

Comment 20 Fedora Update System 2017-10-25 15:25:08 UTC
grub2-2.02-19.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd36ce6cfd

Comment 21 Konrad Rzeszutek Wilk 2017-10-25 20:14:21 UTC
No go:

error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/multiboot2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/multiboot2.mod' not found.
error: can't find command `multiboot2'.
error: can't find command `multiboot2'.
Loading Linux 4.13.8-300.fc27.x86_64 ...Loading Linux 4.13.8-300.fc27.x86_64 ...

error: can't find command `module2'.
error: can't find command `module2'.
Loading initial ramdisk ...Loading initial ramdisk ...

error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: can't find command `module2'.
error: can't find command `module2'.


Let me do a full re-install just in case I messed it up somehow.

Comment 22 Konrad Rzeszutek Wilk 2017-10-25 20:40:07 UTC
Did a brand new install, with an extra repo of the RPMs from https://koji.fedoraproject.org/koji/buildinfo?buildID=988586
and while the grub.cfg looks right, it does not boot (see previous message)

Comment 23 Konrad Rzeszutek Wilk 2017-10-26 13:06:32 UTC
The list of RPMs:
rpm [root@tst063 ~]# rpm -qa | grep grub
grub2-tools-minimal-2.02-19.fc27.x86_64
grub2-efi-x64-modules-2.02-19.fc27.noarch
grubby-8.40-7.fc27.x86_64
grub2-tools-2.02-19.fc27.x86_64
grub2-common-2.02-19.fc27.noarch
grub2-efi-x64-2.02-19.fc27.x86_64
grub2-tools-efi-2.02-19.fc27.x86_64
grub2-tools-extra-2.02-19.fc27.x86_64

I do see:
[root@tst063 ~]# rpm -ql grub2-efi-x64-modules | grep mul
/usr/lib/grub/x86_64-efi/mul_test.mod
/usr/lib/grub/x86_64-efi/multiboot.mod
/usr/lib/grub/x86_64-efi/multiboot2.mod


But that is not helping as it is trying to find the modules from somewhere else..

Comment 24 Adam Williamson 2017-10-30 17:35:14 UTC
Konrad, did UEFI on Xen actually ever work for any previous release?

Comment 25 Peter Jones 2017-10-30 17:40:11 UTC
> But that is not helping as it is trying to find the modules from somewhere else..

Yes, that's why in https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c16 I said you'd probably have to use grub2-install to install the modules.

But also, has this *ever* worked, at least since F18 or later, on an EFI machine?  I think this has not ever worked, and we're basically re-interpreting Testcase_Boot_Methods_Xen_Para_Virt in a way we historically have not, because it was phrased without taking that how we boot on EFI into account.

Honestly I think this is an RFE and we should handle it in F28.

Comment 26 Matthew Miller 2017-10-31 16:49:18 UTC
If this has not ever worked, I propose waiving the blocker this release.

Comment 27 Adam Williamson 2017-10-31 22:12:40 UTC
I'm dropping AcceptedBlocker status for this one (sending it back to 'proposed' status) as we clearly didn't consider the UEFI wrinkle when accepting this as a blocker; we should re-consider its status in that light, hopefully after Konrad answers the question of whether it worked before.

Comment 28 Kamil Páral 2017-11-02 17:51:16 UTC
Discussed during blocker review [1]:

RejectedBlocker RejectedFreezeException (Final) - best information at present is that Xen UEFI installs have never worked. given this, we don't think it makes sense to treat this as a last-minute release blocking bug, or grant a freeze exception. If this turns out to be incorrect, we will re-evaluate.

[1] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2017-11-02/

Comment 29 Konrad Rzeszutek Wilk 2017-11-03 14:08:57 UTC
(In reply to Kamil Páral from comment #28)
> Discussed during blocker review [1]:
> 
> RejectedBlocker RejectedFreezeException (Final) - best information at
> present is that Xen UEFI installs have never worked. given this, we don't
> think it makes sense to treat this as a last-minute release blocking bug, or

I am not sure I understand "last minute" as this bug was created oh, in August? Around three months ago.

> grant a freeze exception. If this turns out to be incorrect, we will
> re-evaluate.


In terms of Xen UEFI never working - the work that was done by Fu Wei - to boot Xen in an UEFI env in ARM64 - and it was ported from upstream in-to grub2.

I am assuming there must have been an RFE for it? If you would like I can contact the Linaro folks to find the exact bug for it.

> 
> [1] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2017-11-02/

Comment 30 Konrad Rzeszutek Wilk 2017-11-03 14:16:48 UTC
(In reply to Peter Jones from comment #25)
> > But that is not helping as it is trying to find the modules from somewhere else..
> 
> Yes, that's why in https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c16 I
> said you'd probably have to use grub2-install to install the modules.

Totally missed that. That is a bit cumbersome, and your previous comments said:

 > No, we absolutely cannot do either of these.  This would be a CVE, as none
 > of these modules have signature checking.

 Actually I'm not sure that's strictly true.

 > We can do the 20_linux_xen.in fix, but the modules need to be loaded
 > dynamically when SB is disabled.

 But I still prefer this method.

What would it take to convince you to do it the other way - the two line change in the rpm.macros script? Is it for Xen to be using the shim GUID to check the vmlinuz? (Meaning it making an EFI call?)

> 
> But also, has this *ever* worked, at least since F18 or later, on an EFI
> machine?  I think this has not ever worked, and we're basically

Yes. By using xen.efi.

And since I am not too clear here, when you say EFI, do you mean EFI+SB+SHIM?
Or really just normal EFI with or without SB?

> re-interpreting Testcase_Boot_Methods_Xen_Para_Virt in a way we historically
> have not, because it was phrased without taking that how we boot on EFI into
> account.
> 
> Honestly I think this is an RFE and we should handle it in F28.

Not sure if RFE = bug. But the Xen in an UEFI env in ARM64 work was ported from upstream in-to grub2.

I am assuming there must have been an RFE for it? If you would like I can contact the Linaro folks to find the exact bug for it.

Comment 31 Adam Williamson 2017-11-03 18:59:58 UTC
"I am not sure I understand "last minute" as this bug was created oh, in August? Around three months ago."

Last minute from the perspective of the release process, not from the perspective of the bug report. From the perspective of the release process, up until #c21 were working on the basis this was under control and would be ready to go.

What pjones means by 'RFE' is "this is an enhancement - making something work that never previously worked - not a bug fix". The point being that it doesn't really make sense to block the release on adding support for something that we've never previously had support for, which wasn't listed as being one of the major new features for the release or anything like that.

Comment 32 Fedora Update System 2017-11-11 03:04:13 UTC
grub2-2.02-19.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Comment 33 M.Cerveny 2017-12-22 20:12:41 UTC
Still buggy:
- no module.mod or module2.mod in /boot/grub2/i386-pc/
  (embedded https://github.com/rhboot/grub2/blob/92bfc33db984eec22966a163eed7b6f2ab0266bf/grub-core/loader/multiboot.c#L451 )
- try to boot xen from "/boot/xen*.map" if exists

My workaround:

--- /etc/grub.d/20_linux_xen.orig	2017-10-24 19:02:02.000000000 +0200
+++ /etc/grub.d/20_linux_xen	2017-12-22 20:54:27.391076447 +0100
@@ -126,7 +126,7 @@
         else
             xen_rm_opts="no-real-mode edd=off"
         fi
-	insmod ${module_loader}
+	#insmod ${module_loader}
 	insmod ${xen_loader}
 	${xen_loader}	${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} \${xen_rm_opts}
 	echo	'$(echo "$lmessage" | grub_quote)'
@@ -137,7 +137,7 @@
     message="$(gettext_printf "Loading initial ramdisk ...")"
     sed "s/^/$submenu_indentation/" << EOF
 	echo	'$(echo "$message" | grub_quote)'
-	insmod ${module_loader}
+	#insmod ${module_loader}
 	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
 EOF
   fi
@@ -168,6 +168,8 @@
 
 file_is_not_sym () {
     case "$1" in
+	*/xen*.map)
+	    return 1;;
 	*/xen-syms-*)
 	    return 1;;
 	*)

Comment 34 Konrad Rzeszutek Wilk 2018-01-03 03:54:28 UTC
I exchanged email with Adam and he said to talk to Peter on the bug on how to resolve this.

So, how can we resolve so that users don't have to go through hoops?

Comment 35 John Reiser 2018-05-14 17:15:54 UTC
In Fedora 27, booting "Fedora 27 with Xen hypervisor" fails with complaints about not finding /EFI/Fedora/X86_64-efi/module2.mod and multiboot2.mod in separate "Inserting ..." commands.

The complete list of installed grub* rpms is:
$ rpm -qa | sort | grep grub
grub2-common-2.02-22.fc27.noarch
grub2-efi-ia32-2.02-22.fc27.x86_64
grub2-efi-ia32-cdboot-2.02-22.fc27.x86_64
grub2-efi-x64-2.02-22.fc27.x86_64
grub2-efi-x64-cdboot-2.02-22.fc27.x86_64
grub2-pc-2.02-22.fc27.x86_64
grub2-pc-modules-2.02-22.fc27.noarch
grub2-tools-2.02-22.fc27.x86_64
grub2-tools-efi-2.02-22.fc27.x86_64
grub2-tools-extra-2.02-22.fc27.x86_64
grub2-tools-minimal-2.02-22.fc27.x86_64
grubby-8.40for i8.fc27.x86_64

Looking for the missing files:
$ rpm -ql $(rpm -qa | sort | grep grub) | egrep 'module|multiboot'
/usr/lib/grub/i386-pc/gmodule.pl
/usr/lib/grub/i386-pc/multiboot.mod
/usr/lib/grub/i386-pc/multiboot2.mod

So module2 is missing, and multiboot2.mod is not in the expected directory /EFI/Fedora/X86_64efi.  That last part hints at UEFI is not supported.

Comment 36 Lloyd Kvam 2018-07-03 01:44:46 UTC
XEN was working fine for me in Fedora 26. It continued working in Fedora 27 until the latest update. The error message is about being unable to find the multiboot module.

That brought me to this bug report.

From your comments, there is a way to edit /etc/grub.d/20_linux_xen so that XEN will load. The instructions were too cryptic for me. How do I change 20_linux_xen so that XEN will boot?  This has nothing to do with UEFI.

Comment 37 John Reiser 2018-07-03 02:31:00 UTC
(In reply to Lloyd Kvam from comment #36)
> XEN was working fine for me in Fedora 26. It continued working in Fedora 27
> until the latest update.

Which specific packages were in the "latest update"?  Check /var/log/dnf.rpm.log-* or other files in /var/log.

> The error message is about being unable to find the
> multiboot module.

Was the complaint about 'multiboot.mod' or 'multiboot2.mod' [note the '2'] or something else?  It matters, and your text did not give an exact quote nor inspire confidence in the name.

> That brought me to this bug report.
> 
> From your comments, there is a way to edit /etc/grub.d/20_linux_xen so that
> XEN will load. The instructions were too cryptic for me.

If you are referring to https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c33 then that is a standard output from "diff -u" for input to the /usr/bin/patch utility; see the manual page.

Comment 38 Lloyd Kvam 2018-07-03 04:16:02 UTC
(In reply to John Reiser from comment #37)
> (In reply to Lloyd Kvam from comment #36)
> > XEN was working fine for me in Fedora 26. It continued working in Fedora 27
> > until the latest update.
> 
> Which specific packages were in the "latest update"?  Check
> /var/log/dnf.rpm.log-* or other files in /var/log.

I am in the middle of upgrading to Fedora 28. I am hoping that will work.

> > 
> > The error message is about being unable to find the
> > multiboot module.
> 
> Was the complaint about 'multiboot.mod' or 'multiboot2.mod' [note the '2']
> or something else?  It matters, and your text did not give an exact quote
> nor inspire confidence in the name.

The screen refreshes too quickly on the the failure. I believe it is multiboot2.mod.

> 
> > That brought me to this bug report.
> > 
> > From your comments, there is a way to edit /etc/grub.d/20_linux_xen so that
> > XEN will load. The instructions were too cryptic for me.
> 
> If you are referring to
> https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c33 then that is a
> standard output from "diff -u" for input to the /usr/bin/patch utility; see
> the manual page.

That was totally different from the COMMENT 3 advice:
    "the workoarund is to comment out the '$grub_file --is-arm64-efi $current_xen'"

That snippet is part of an if/else block and any changes I made there just led to different errors.

I'll see how the Fedora 28 version boots and apply the comment33 patch if I am still having trouble.

Thank you very much for responding.

Comment 39 Lloyd Kvam 2018-07-03 12:16:30 UTC
The original error message is:
/grub2/i386-pc/module2.mod not found

I applied the comment33 changes to 20_linux_xen and ran grub2-mkconfig.

I no longer get the module2.mod not found error message, but the XEN configuration does not boot.

Comment 40 mathieu.tarral 2018-07-03 20:59:46 UTC
Hi everyone,

I found this thread because i have the same problem: Xen doesn't boot on Fedora 28 and 27: module2.mod is missing.

Like someone said earlier it was working fine in Fedora 26.
I'm developing a Vagrant Box to automatically install Xen and virtual machine introspection tools:
https://github.com/Wenzel/vagrant-xen

That's how i found the issue.
Maybe this project can help you find the bug more easily.

Thanks !

Comment 41 Lloyd Kvam 2018-07-03 21:17:04 UTC
I tried installing grub2-efi-modules, but that had no impact. I restored the original 20_linux_xen config file.

Then I tried booting with my oldest kernel (uname -a)
Linux vmhost.home.lan 4.16.16-200.fc27.x86_64 #1 SMP Sun Jun 17 03:06:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I still got the error message about module2.mod not found. But XEN booted. I think it is running properly.

That suggests that the module2.mod not found error has been occurring for a while. The server is headless and I use SSH to connect to it. I do not mormally see the boot messages.

So I think there is a kernel issue of some sort.

Comment 42 mathieu.tarral 2018-07-03 21:25:06 UTC
Ok, I just found something interesting:

The fact that Xen doesn't boot anymore doesn't come from this module2, but from the kernel itself.

I was able to boot a Xen entry using and older kernel (not the first entry in the menu): 4.16.3-301.fc28.x86_64

So this GRUB entry:
menuentry 'Fedora, with Xen 4.10.1 and Linux 4.16.3-301.fc28.x86_64'
is booting, even though we have 2 annoying messages about module2.

Comment 43 JNoake 2018-07-06 00:24:44 UTC
I have a similar problem.
Upgraded from 4.13.16-100.fc25.x86_64 to fc27
Now the XEN boot fails when booting with:
'Fedora, with Xen 4.9.2.config and Linux 4.17.3-100.fc27.x86_64'

it goes black screen and the reboots just after showing :
'setup 0000:00:00.2 for d0 failed (-19)'
'Scrubbing Free RAM on 1nodes using 8 CPUs'

adding a 'noreboot' option to the grub command line just leaves me with a hung system with a black screen.
I have subsequently booted with a live USB and checked /var/log/messages, /var/log/boot.log and /var/log/Xorg.0.log - I cant find anything that looks like  they have even been written to during the failed boot - nothing about a panic or other terminal failure message at the ends.

but it *will* boot with:
'Fedora, with Xen 4.9.2.config and Linux 4.13.16-100.fc25.x86_64'

and appears to work normally.

I too see:
error: file '/grub2/i386-pc/module2.mod' not found
twice, and on BOTH kernels, despite the fc25 then continuing normally after that.

I am interested in a solution, even if that is to continue upgrading to fc28 - if it works.

Comment 44 mathieu.tarral 2018-11-04 14:51:50 UTC
Hi,
I upgraded to Fedora 29 this week, and problems is still here, kernel won't boot under Xen hypervisor.

Can anyone from Fedora team look into this ?
It has been here since at least 2 releases.

Also, i personally need to run Xen for both security and development purposes and i just can't do it on Fedora.

Thank you in advance.

Comment 45 Lloyd Kvam 2018-11-05 01:03:29 UTC
(In reply to mathieu.tarral from comment #44)
> Hi,
> I upgraded to Fedora 29 this week, and problems is still here, kernel won't
> boot under Xen hypervisor.
> 
> Can anyone from Fedora team look into this ?
> It has been here since at least 2 releases.
> 
> Also, i personally need to run Xen for both security and development
> purposes and i just can't do it on Fedora.
> 
> Thank you in advance.

XEN has been working for me. The problem was fixed some time ago. I would recommend carefully reviewing your error messages and configuration.

Comment 46 mathieu.tarral 2018-11-05 16:24:42 UTC
Hi Lloyd, thanks for your reply, and for confirming that you have a working installation of Xen.

This what I know about my boot issue:
before it was due to the 'module2' grub command not found, but this has been fixed, as you said.

And since I upgraded from f27 to f28 I think, the error changed to the following:

error: ../../grub-core/kern/efi/tpm.c:255:Unknown TPM error

image: https://user-images.githubusercontent.com/964610/48011149-60426580-e11f-11e8-8e85-0914fd11d2ea.JPG

I thought i was on the right bug report, but there was another one, where I already replied :)
https://bugzilla.redhat.com/show_bug.cgi?id=1575282

But thanks !

Comment 47 mathieu.tarral 2018-11-05 16:48:46 UTC
Another update, I managed to boot on Xen !! \o/

But I had to give up secure boot along the way...

Surprisingly, I still have this weird error with the 'module2.mod' not being found, but then I press ENTER and Xen manages to boot anyway.

Image: https://user-images.githubusercontent.com/964610/48012600-b06ef700-e122-11e8-9a55-b34809905af8.JPG

Comment 48 Roy A. Gilmore 2019-02-04 22:32:09 UTC
I'm not sure what is going on here, UEFI + grub2 + Xen does not work for me.

Fedora 29
kernel*-4.20.5-200.fc29.x86_64
grub2-(efi-x64}*-2.02-62.fc29.x86_64
shim-x64-15-7.x86_64
xen*-4.11.1-1.fc29.x86_64

I get the same issues everyone else has reported:

`/EFI/fedora/x86_64-efi/module2.mod' not found.
and
`/EFI/fedora/x86_64-efi/multiboot.mod' not found.

/etc/grub.d/20_linux_xen requests both "module2.mod" and "multiboot2.mod"

I don't know enough about UEFI, grub2 or xen, but it seems like either /etc/grub.d/20_linux_xen needs to be modified to not request "module2.mod" and "multiboot2.mod" *or* those modules need to be built and supplied. This has going on for a long time, what could be a higher priority than booting the system? This should be escalated to the highest priority.

Comment 49 Roy A. Gilmore 2019-02-07 06:13:14 UTC
(In reply to Roy A. Gilmore from comment #48)
> I'm not sure what is going on here, UEFI + grub2 + Xen does not work for me.
> 
> Fedora 29
> kernel*-4.20.5-200.fc29.x86_64
> grub2-(efi-x64}*-2.02-62.fc29.x86_64
> shim-x64-15-7.x86_64
> xen*-4.11.1-1.fc29.x86_64
> 
> I get the same issues everyone else has reported:
> 
> `/EFI/fedora/x86_64-efi/module2.mod' not found.
> and
> `/EFI/fedora/x86_64-efi/multiboot.mod' not found.
> 
> /etc/grub.d/20_linux_xen requests both "module2.mod" and "multiboot2.mod"
> 
> I don't know enough about UEFI, grub2 or xen, but it seems like either
> /etc/grub.d/20_linux_xen needs to be modified to not request "module2.mod"
> and "multiboot2.mod" *or* those modules need to be built and supplied. This
> has going on for a long time, what could be a higher priority than booting
> the system? This should be escalated to the highest priority.

OK, I just downloaded the grub2 srpm and poked around a little bit, and apparently multiboot and multiboot2 were disable on purpose. This brings up a few questions: 1) why were multiboot and multiboot2 disabled, and 2) if there was a valid reason why they were disabled, why wasn't /etc/grub.d/20_linux_xen modified at the same time, and finally 3) did QA sign off on this? As shipped, grub2 will not boot xen on efi, doesn't anyone test these packages before they are released.

Comment 50 Adam Williamson 2019-02-07 14:33:48 UTC
Per the changelog, the reason multiboot modules are disabled in the EFI build is that they allow execution of arbitrary code, breaking secure boot:

https://bugzilla.redhat.com/show_bug.cgi?id=1264103

"2) if there was a valid reason why they were disabled, why wasn't /etc/grub.d/20_linux_xen modified at the same time"

pjones probably didn't notice this issue. xen isn't that widely used or tested on Fedora.

"3) did QA sign off on this?"

not explicitly. QA does not get to "sign off" on every decision made in every package (or even in the most important packages), because there are far too many things going on for that to be plausible. We don't test Fedora with Xen every day, in fact we mainly rely on Konrad - who filed this bug - to do that testing (there is a deal in place that Xen remains in the Fedora release criteria so long as Konrad does the testing, or finds someone else to do it). Xen on UEFI is not considered a release-blocking feature for Fedora, per the release blocker bug proposal discussion above. (Though if fixing this is as simple as taking the multiboot bit out of 20_linux_xen, we should probably do that.)

Comment 51 Konrad Rzeszutek Wilk 2019-02-11 22:27:46 UTC
Fedora != RHEL

Or is expected that from a SecureBoot perspective it has everything that RHEL has (like the patcheset that locks down the kernel which Linus has very strongly Nacked?)

Comment 52 Roy A. Gilmore 2019-02-12 01:31:30 UTC
(In reply to Adam Williamson from comment #50)
> Per the changelog, the reason multiboot modules are disabled in the EFI
> build is that they allow execution of arbitrary code, breaking secure boot:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1264103
> 

OK, fair enough, although in my opinion, secure boot is a poorly thought out specification that breaks as much (or more) than it fixes, and I don't care if packages break secure boot, because secure boot is broken by design. Regardless, Xen has been able to be built as EFI binaries since version 4.3, and Xen has only been able to built with multiboot2 support since version 4.9. Fedora currently supplies version 4.11, which can still be built as EFI binaries, completely bypassing grub2, multiboot2, and this secure boot issue. If Fedora requires secure boot, then Fedora should build Xen as EFI binaries.

> "2) if there was a valid reason why they were disabled, why wasn't
> /etc/grub.d/20_linux_xen modified at the same time"
> 
> pjones probably didn't notice this issue. xen isn't that widely used or
> tested on Fedora.
> 

How could anyone *not* notice this issue? Install the stock Xen hypervisor from the standard Fedora repos on a stock Fedora UEFI system, and the Xen hypervisor will not boot. It's impossible to miss, *if* someone actually tries it. Does Fedora build packages and not bother to test them?

Where did this statistic come from? Who says that Xen isn't "widely used" on Fedora. Nobody has ever asked me (or anyone I know) what packages I (we) use, or what packages I (we) want added or dropped. Just because the steering committee decided that kvm should be the only fully supported hypervisor on Fedora, doesn't mean end-users haven't compiled Xen from source to bypass Fedora's broken implementation. Anyone actually using a hypervisor probably knows how to compile one.

> "3) did QA sign off on this?"
> 
> not explicitly. QA does not get to "sign off" on every decision made in
> every package (or even in the most important packages), because there are
> far too many things going on for that to be plausible. We don't test Fedora
> with Xen every day, in fact we mainly rely on Konrad - who filed this bug -
> to do that testing (there is a deal in place that Xen remains in the Fedora
> release criteria so long as Konrad does the testing, or finds someone else
> to do it). Xen on UEFI is not considered a release-blocking feature for
> Fedora, per the release blocker bug proposal discussion above. (Though if
> fixing this is as simple as taking the multiboot bit out of 20_linux_xen, we
> should probably do that.)

QA doesn't need to sign off on every decision made in every package, that's management's/steering committee's or engineering's job. QA's job is to test and verify that every package works as advertised. If QA isn't testing and verifying that every package works as advertised, what are they doing that's interfering with their primary function?. Packages should not be *released* to the "wild" until QA has signed off on them. I'm former career military, one of my duties was QA, and I certified aircraft as "safe for flight". I understand QA's job very well because if I made a mistake, people could $#@!ing die, and *nobody* dies on my watch. Military aircraft are for more complex than Fedora, and have far more things "going on". Not only is testing and verifying every package plausible, it's possible, and if it's not done, it's because people *choose* not to.

Comment 53 Adam Williamson 2019-02-12 07:43:41 UTC
"Does Fedora build packages and not bother to test them?"

Quite a lot of packagers seem to, yes. But that's hardly necessary in this case: whoever is packaging xen could have tested it diligently...on a BIOS system.

"Where did this statistic come from?"

It's not a statistic, per se...

"Who says that Xen isn't "widely used" on Fedora."

...I do. Based on my experience that I get asked about issues using Fedora on VMWare and VirtualBox *all the time*, hyper-v and KVM somewhat less often, and Xen almost never.

"QA doesn't need to sign off on every decision made in every package, that's management's/steering committee's or engineering's job"

Practically speaking, in Fedora, it's no-one's job, because it's impossible. There is far too much stuff in Fedora and we don't have a zillion people in "management" or "steering committee" or even in "engineering". Packagers are, broadly, responsible for ensuring their packages work. QA does its best to make sure the overall distribution works as well as possible, within the confines of the distribution being gigantic and QA being very tiny.

"If QA isn't testing and verifying that every package works as advertised, what are they doing that's interfering with their primary function?"

That's not our primary function. There are tens of thousands of packages in the distro. There are tens of people in QA. You can probably figure out the math on that one.

"Packages should not be *released* to the "wild" until QA has signed off on them. I'm former career military, one of my duties was QA, and I certified aircraft as "safe for flight". I understand QA's job very well because if I made a mistake, people could $#@!ing die, and *nobody* dies on my watch. Military aircraft are for more complex than Fedora, and have far more things "going on"."

Funnily enough I actually doubt that's true; I also doubt they change anywhere near as often, or have anywhere near as many supported (kinda) configurations. You wouldn't, for instance, permit a plane to have three entirely different combinations of engines it could use depending on the whim of the pilot, yet here we are with about a half dozen virtualization systems to be worried about, never mind container runtimes. I also suspect there are a damn sight more people involving in doing QA on any given aircraft than there are involved in doing QA on Fedora. :P

Basically: no, there is no guarantee that every change in Fedora is "signed off" or tested by someone (other than the person making it, and frankly, not really even them). If that is what you expect from software, Fedora is not the software for you.

"Not only is testing and verifying every package plausible, it's possible, and if it's not done, it's because people *choose* not to."

The "choice" in this case is that no-one is paying for several hundred or thousand QA people to do this. Since we don't have those resources, we target the resources we have as best we can.

Comment 54 Lars Kurth 2019-05-14 00:26:34 UTC
> "Who says that Xen isn't "widely used" on Fedora."
>
> ...I do. Based on my experience that I get asked about issues using Fedora on VMWare and VirtualBox 
> *all the time*, hyper-v and KVM somewhat less often, and Xen almost never.

This doesn't necessarily mean that Fedora guests on Xen aren't widely used. A lot of the Fedora questions
(as well as questions for other distros) tend to get raised on Xen lists. Because we tend to answer these, 
people don't often go to distro lists. Then there are downstreams such as QubesOS that use Fedora under the
hood, cloud services and commercial products such as XenServer, where Fedora is a key part of the offering, 
but questions don't necessarily come up through Fedora user channels.


Note You need to log in before you can comment on or make changes to this bug.