Bug 1486002

Ugh, also happens with upstream..

Author: Pete Batard <pete>
Date:   Mon Aug 7 16:23:12 2017 +0100

    io: add a GRUB_GZ prefix to gzio specific defines
    
    * This is done to avoid a conflict with a PACKED define in the EDK2

Ah,

Commenting out 
$grub_file --is-arm64-efi $current_xen

makes it work

Hmm. why does it have '$' in front?


Removing the '$' fixes it as well.

err, scratch the '$' - that is with the upstream version (which is built without the 'grub2' prefix). For the rawhide version, the workoarund is to comment out the '$grub_file --is-arm64-efi $current_xen'

Or:

    if ($grub_file --is-arm64-efi $current_xen); then
        xen_loader="xen_hypervisor"
        module_loader="xen_module"
    else
        xen_loader="multiboot"
        module_loader="module"
    fi


Let me post a patch upstream for the above fix.

Posted at http://lists.gnu.org/archive/html/grub-devel/2017-08/msg00073.html

Also it may make sense to modify the grub.macros for GRUB_MODULES to also
have "multiboot multiboot2" in it.

And in the efi_modules for aarch64 to also have "xen_boot". like this:

[root@tst063 ~]# more grub2.spec.diff 
--- grub.macros.orig    2017-08-28 14:54:23.511123738 -0400
+++ grub.macros 2017-08-28 14:54:56.573933975 -0400
@@ -87,7 +87,7 @@
 
 ### fixme
 %ifarch aarch64 %{arm}
-%global efi_modules " http linux "
+%global efi_modules " http linux xen_boot"
 %else
 %global efi_modules " backtrace http linuxefi usb usbserial_common usbserial_pl2303 usbserial_ftdi usbserial_usbdebug "
 %endif
@@ -325,7 +325,7 @@
                password_pbkdf2 png reboot                      \\\
                search search_fs_uuid search_fs_file            \\\
                search_label serial sleep syslinuxcfg test tftp \\\
-               video xfs"                                      \
+               video xfs multiboot multiboot2"                 \
 GRUB_MODULES+=%{efi_modules}                                   \
 %{expand:%%{mkimage %{1} %{2} %{3} %{4}}}                      \
 %{nil}


And naturally expand the grub.patches with the patches.

Update:
http://lists.gnu.org/archive/html/grub-devel/2017-08/msg00095.html

Proposed as a Blocker for 27-final by Fedora user konradr using the blocker tracking app because:

 Fails QA:Testcase_Boot_Methods_Xen_Para_Virt

Discussed during blocker review [1]:

AcceptedBlocker (Beta) - we're happy to accept Konrad's assertion that this violates the Final criterion "The release must boot successfully as Xen DomU with releases providing a functional, supported Xen Dom0 and widely used cloud providers utilizing Xen"

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-09-04/

They are upstream (git://git.sv.gnu.org/grub.git)

b4d709b6e Use grub-file to figure out whether multiboot2 should be used for Xen.gz
a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for aarch64

Peter, if there is way for me to submit the patch to grub RPM repo I can try doing that if that would be easier?

Still an issue with Fedora 27 Beta 1.2

Still an issue with Fedora-Server-dvd-x86_64-27_Beta-1.3.iso

(In reply to Konrad Rzeszutek Wilk from comment #8)
> They are upstream (git://git.sv.gnu.org/grub.git)
> 
> b4d709b6e Use grub-file to figure out whether multiboot2 should be used for
> Xen.gz
> a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for
> aarch64
> 
> Peter, if there is way for me to submit the patch to grub RPM repo I can try
> doing that if that would be easier?

You may try to send a PR on https://src.fedoraproject.org/rpms/grub/.

*** Bug 1498300 has been marked as a duplicate of this bug. ***

(In reply to Robin Lee from comment #11)
> (In reply to Konrad Rzeszutek Wilk from comment #8)
> > They are upstream (git://git.sv.gnu.org/grub.git)
> > 
> > b4d709b6e Use grub-file to figure out whether multiboot2 should be used for
> > Xen.gz
> > a8e0f1adf Fix util/grub.d/20_linux_xen.in: Add xen_boot command support for
> > aarch64
> > 
> > Peter, if there is way for me to submit the patch to grub RPM repo I can try
> > doing that if that would be easier?
> 
> You may try to send a PR on https://src.fedoraproject.org/rpms/grub/.

That should be https://src.fedoraproject.org/rpms/grub2/

First ever pull request!
https://src.fedoraproject.org/rpms/grub2/pull-request/1

Lets see how that goes

(In reply to Konrad Rzeszutek Wilk from comment #4)
> Posted at http://lists.gnu.org/archive/html/grub-devel/2017-08/msg00073.html
> 
> Also it may make sense to modify the grub.macros for GRUB_MODULES to also
> have "multiboot multiboot2" in it.
> 
> And in the efi_modules for aarch64 to also have "xen_boot". like this:
> 
> [root@tst063 ~]# more grub2.spec.diff 
> --- grub.macros.orig    2017-08-28 14:54:23.511123738 -0400
> +++ grub.macros 2017-08-28 14:54:56.573933975 -0400
> @@ -87,7 +87,7 @@
>  
>  ### fixme
>  %ifarch aarch64 %{arm}
> -%global efi_modules " http linux "
> +%global efi_modules " http linux xen_boot"
>  %else
>  %global efi_modules " backtrace http linuxefi usb usbserial_common
> usbserial_pl2303 usbserial_ftdi usbserial_usbdebug "
>  %endif
> @@ -325,7 +325,7 @@
>                 password_pbkdf2 png reboot                      \\\
>                 search search_fs_uuid search_fs_file            \\\
>                 search_label serial sleep syslinuxcfg test tftp \\\
> -               video xfs"                                      \
> +               video xfs multiboot multiboot2"                 \
>  GRUB_MODULES+=%{efi_modules}                                   \
>  %{expand:%%{mkimage %{1} %{2} %{3} %{4}}}                      \
>  %{nil}
> 
> 
> And naturally expand the grub.patches with the patches.

No, we absolutely cannot do either of these.  This would be a CVE, as none of these modules have signature checking.

We can do the 20_linux_xen.in fix, but the modules need to be loaded dynamically when SB is disabled.

Created attachment 1340817 [details]
insmod modules xen needs when it needs them.

Note that you'll still need to have grub2-efi-aa64-modules installed (and probably do grub2-install bits to put them in /boot/grub2 or whatnot as well.)

> No, we absolutely cannot do either of these.  This would be a CVE, as none
> of these modules have signature checking.

Actually I'm not sure that's strictly true.

> We can do the 20_linux_xen.in fix, but the modules need to be loaded
> dynamically when SB is disabled.

But I still prefer this method.

You got an RPM I can test?

grub2-2.02-19.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd36ce6cfd

grub2-2.02-19.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-bd36ce6cfd

No go:

error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/multiboot2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/multiboot2.mod' not found.
error: can't find command `multiboot2'.
error: can't find command `multiboot2'.
Loading Linux 4.13.8-300.fc27.x86_64 ...Loading Linux 4.13.8-300.fc27.x86_64 ...

error: can't find command `module2'.
error: can't find command `module2'.
Loading initial ramdisk ...Loading initial ramdisk ...

error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: file `/EFI/fedora/x86_64-efi/module2.mod' not found.
error: can't find command `module2'.
error: can't find command `module2'.


Let me do a full re-install just in case I messed it up somehow.

Did a brand new install, with an extra repo of the RPMs from https://koji.fedoraproject.org/koji/buildinfo?buildID=988586
and while the grub.cfg looks right, it does not boot (see previous message)

The list of RPMs:
rpm [root@tst063 ~]# rpm -qa | grep grub
grub2-tools-minimal-2.02-19.fc27.x86_64
grub2-efi-x64-modules-2.02-19.fc27.noarch
grubby-8.40-7.fc27.x86_64
grub2-tools-2.02-19.fc27.x86_64
grub2-common-2.02-19.fc27.noarch
grub2-efi-x64-2.02-19.fc27.x86_64
grub2-tools-efi-2.02-19.fc27.x86_64
grub2-tools-extra-2.02-19.fc27.x86_64

I do see:
[root@tst063 ~]# rpm -ql grub2-efi-x64-modules | grep mul
/usr/lib/grub/x86_64-efi/mul_test.mod
/usr/lib/grub/x86_64-efi/multiboot.mod
/usr/lib/grub/x86_64-efi/multiboot2.mod


But that is not helping as it is trying to find the modules from somewhere else..

Konrad, did UEFI on Xen actually ever work for any previous release?

> But that is not helping as it is trying to find the modules from somewhere else..

Yes, that's why in https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c16 I said you'd probably have to use grub2-install to install the modules.

But also, has this *ever* worked, at least since F18 or later, on an EFI machine?  I think this has not ever worked, and we're basically re-interpreting Testcase_Boot_Methods_Xen_Para_Virt in a way we historically have not, because it was phrased without taking that how we boot on EFI into account.

Honestly I think this is an RFE and we should handle it in F28.

If this has not ever worked, I propose waiving the blocker this release.

I'm dropping AcceptedBlocker status for this one (sending it back to 'proposed' status) as we clearly didn't consider the UEFI wrinkle when accepting this as a blocker; we should re-consider its status in that light, hopefully after Konrad answers the question of whether it worked before.

Discussed during blocker review [1]:

RejectedBlocker RejectedFreezeException (Final) - best information at present is that Xen UEFI installs have never worked. given this, we don't think it makes sense to treat this as a last-minute release blocking bug, or grant a freeze exception. If this turns out to be incorrect, we will re-evaluate.

[1] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2017-11-02/

(In reply to Kamil Páral from comment #28)
> Discussed during blocker review [1]:
> 
> RejectedBlocker RejectedFreezeException (Final) - best information at
> present is that Xen UEFI installs have never worked. given this, we don't
> think it makes sense to treat this as a last-minute release blocking bug, or

I am not sure I understand "last minute" as this bug was created oh, in August? Around three months ago.

> grant a freeze exception. If this turns out to be incorrect, we will
> re-evaluate.


In terms of Xen UEFI never working - the work that was done by Fu Wei - to boot Xen in an UEFI env in ARM64 - and it was ported from upstream in-to grub2.

I am assuming there must have been an RFE for it? If you would like I can contact the Linaro folks to find the exact bug for it.

> 
> [1] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2017-11-02/

(In reply to Peter Jones from comment #25)
> > But that is not helping as it is trying to find the modules from somewhere else..
> 
> Yes, that's why in https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c16 I
> said you'd probably have to use grub2-install to install the modules.

Totally missed that. That is a bit cumbersome, and your previous comments said:

 > No, we absolutely cannot do either of these.  This would be a CVE, as none
 > of these modules have signature checking.

 Actually I'm not sure that's strictly true.

 > We can do the 20_linux_xen.in fix, but the modules need to be loaded
 > dynamically when SB is disabled.

 But I still prefer this method.

What would it take to convince you to do it the other way - the two line change in the rpm.macros script? Is it for Xen to be using the shim GUID to check the vmlinuz? (Meaning it making an EFI call?)

> 
> But also, has this *ever* worked, at least since F18 or later, on an EFI
> machine?  I think this has not ever worked, and we're basically

Yes. By using xen.efi.

And since I am not too clear here, when you say EFI, do you mean EFI+SB+SHIM?
Or really just normal EFI with or without SB?

> re-interpreting Testcase_Boot_Methods_Xen_Para_Virt in a way we historically
> have not, because it was phrased without taking that how we boot on EFI into
> account.
> 
> Honestly I think this is an RFE and we should handle it in F28.

Not sure if RFE = bug. But the Xen in an UEFI env in ARM64 work was ported from upstream in-to grub2.

I am assuming there must have been an RFE for it? If you would like I can contact the Linaro folks to find the exact bug for it.

"I am not sure I understand "last minute" as this bug was created oh, in August? Around three months ago."

Last minute from the perspective of the release process, not from the perspective of the bug report. From the perspective of the release process, up until #c21 were working on the basis this was under control and would be ready to go.

What pjones means by 'RFE' is "this is an enhancement - making something work that never previously worked - not a bug fix". The point being that it doesn't really make sense to block the release on adding support for something that we've never previously had support for, which wasn't listed as being one of the major new features for the release or anything like that.

grub2-2.02-19.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.

Still buggy:
- no module.mod or module2.mod in /boot/grub2/i386-pc/
  (embedded https://github.com/rhboot/grub2/blob/92bfc33db984eec22966a163eed7b6f2ab0266bf/grub-core/loader/multiboot.c#L451 )
- try to boot xen from "/boot/xen*.map" if exists

My workaround:

--- /etc/grub.d/20_linux_xen.orig	2017-10-24 19:02:02.000000000 +0200
+++ /etc/grub.d/20_linux_xen	2017-12-22 20:54:27.391076447 +0100
@@ -126,7 +126,7 @@
         else
             xen_rm_opts="no-real-mode edd=off"
         fi
-	insmod ${module_loader}
+	#insmod ${module_loader}
 	insmod ${xen_loader}
 	${xen_loader}	${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} \${xen_rm_opts}
 	echo	'$(echo "$lmessage" | grub_quote)'
@@ -137,7 +137,7 @@
     message="$(gettext_printf "Loading initial ramdisk ...")"
     sed "s/^/$submenu_indentation/" << EOF
 	echo	'$(echo "$message" | grub_quote)'
-	insmod ${module_loader}
+	#insmod ${module_loader}
 	${module_loader}	--nounzip   ${rel_dirname}/${initrd}
 EOF
   fi
@@ -168,6 +168,8 @@
 
 file_is_not_sym () {
     case "$1" in
+	*/xen*.map)
+	    return 1;;
 	*/xen-syms-*)
 	    return 1;;
 	*)

I exchanged email with Adam and he said to talk to Peter on the bug on how to resolve this.

So, how can we resolve so that users don't have to go through hoops?

In Fedora 27, booting "Fedora 27 with Xen hypervisor" fails with complaints about not finding /EFI/Fedora/X86_64-efi/module2.mod and multiboot2.mod in separate "Inserting ..." commands.

The complete list of installed grub* rpms is:
$ rpm -qa | sort | grep grub
grub2-common-2.02-22.fc27.noarch
grub2-efi-ia32-2.02-22.fc27.x86_64
grub2-efi-ia32-cdboot-2.02-22.fc27.x86_64
grub2-efi-x64-2.02-22.fc27.x86_64
grub2-efi-x64-cdboot-2.02-22.fc27.x86_64
grub2-pc-2.02-22.fc27.x86_64
grub2-pc-modules-2.02-22.fc27.noarch
grub2-tools-2.02-22.fc27.x86_64
grub2-tools-efi-2.02-22.fc27.x86_64
grub2-tools-extra-2.02-22.fc27.x86_64
grub2-tools-minimal-2.02-22.fc27.x86_64
grubby-8.40for i8.fc27.x86_64

Looking for the missing files:
$ rpm -ql $(rpm -qa | sort | grep grub) | egrep 'module|multiboot'
/usr/lib/grub/i386-pc/gmodule.pl
/usr/lib/grub/i386-pc/multiboot.mod
/usr/lib/grub/i386-pc/multiboot2.mod

So module2 is missing, and multiboot2.mod is not in the expected directory /EFI/Fedora/X86_64efi.  That last part hints at UEFI is not supported.

XEN was working fine for me in Fedora 26. It continued working in Fedora 27 until the latest update. The error message is about being unable to find the multiboot module.

That brought me to this bug report.

From your comments, there is a way to edit /etc/grub.d/20_linux_xen so that XEN will load. The instructions were too cryptic for me. How do I change 20_linux_xen so that XEN will boot?  This has nothing to do with UEFI.

(In reply to Lloyd Kvam from comment #36)
> XEN was working fine for me in Fedora 26. It continued working in Fedora 27
> until the latest update.

Which specific packages were in the "latest update"?  Check /var/log/dnf.rpm.log-* or other files in /var/log.

> The error message is about being unable to find the
> multiboot module.

Was the complaint about 'multiboot.mod' or 'multiboot2.mod' [note the '2'] or something else?  It matters, and your text did not give an exact quote nor inspire confidence in the name.

> That brought me to this bug report.
> 
> From your comments, there is a way to edit /etc/grub.d/20_linux_xen so that
> XEN will load. The instructions were too cryptic for me.

If you are referring to https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c33 then that is a standard output from "diff -u" for input to the /usr/bin/patch utility; see the manual page.

(In reply to John Reiser from comment #37)
> (In reply to Lloyd Kvam from comment #36)
> > XEN was working fine for me in Fedora 26. It continued working in Fedora 27
> > until the latest update.
> 
> Which specific packages were in the "latest update"?  Check
> /var/log/dnf.rpm.log-* or other files in /var/log.

I am in the middle of upgrading to Fedora 28. I am hoping that will work.

> > 
> > The error message is about being unable to find the
> > multiboot module.
> 
> Was the complaint about 'multiboot.mod' or 'multiboot2.mod' [note the '2']
> or something else?  It matters, and your text did not give an exact quote
> nor inspire confidence in the name.

The screen refreshes too quickly on the the failure. I believe it is multiboot2.mod.

> 
> > That brought me to this bug report.
> > 
> > From your comments, there is a way to edit /etc/grub.d/20_linux_xen so that
> > XEN will load. The instructions were too cryptic for me.
> 
> If you are referring to
> https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c33 then that is a
> standard output from "diff -u" for input to the /usr/bin/patch utility; see
> the manual page.

That was totally different from the COMMENT 3 advice:
    "the workoarund is to comment out the '$grub_file --is-arm64-efi $current_xen'"

That snippet is part of an if/else block and any changes I made there just led to different errors.

I'll see how the Fedora 28 version boots and apply the comment33 patch if I am still having trouble.

Thank you very much for responding.

The original error message is:
/grub2/i386-pc/module2.mod not found

I applied the comment33 changes to 20_linux_xen and ran grub2-mkconfig.

I no longer get the module2.mod not found error message, but the XEN configuration does not boot.

Hi everyone,

I found this thread because i have the same problem: Xen doesn't boot on Fedora 28 and 27: module2.mod is missing.

Like someone said earlier it was working fine in Fedora 26.
I'm developing a Vagrant Box to automatically install Xen and virtual machine introspection tools:
https://github.com/Wenzel/vagrant-xen

That's how i found the issue.
Maybe this project can help you find the bug more easily.

Thanks !

I tried installing grub2-efi-modules, but that had no impact. I restored the original 20_linux_xen config file.

Then I tried booting with my oldest kernel (uname -a)
Linux vmhost.home.lan 4.16.16-200.fc27.x86_64 #1 SMP Sun Jun 17 03:06:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

I still got the error message about module2.mod not found. But XEN booted. I think it is running properly.

That suggests that the module2.mod not found error has been occurring for a while. The server is headless and I use SSH to connect to it. I do not mormally see the boot messages.

So I think there is a kernel issue of some sort.

Ok, I just found something interesting:

The fact that Xen doesn't boot anymore doesn't come from this module2, but from the kernel itself.

I was able to boot a Xen entry using and older kernel (not the first entry in the menu): 4.16.3-301.fc28.x86_64

So this GRUB entry:
menuentry 'Fedora, with Xen 4.10.1 and Linux 4.16.3-301.fc28.x86_64'
is booting, even though we have 2 annoying messages about module2.

I have a similar problem.
Upgraded from 4.13.16-100.fc25.x86_64 to fc27
Now the XEN boot fails when booting with:
'Fedora, with Xen 4.9.2.config and Linux 4.17.3-100.fc27.x86_64'

it goes black screen and the reboots just after showing :
'setup 0000:00:00.2 for d0 failed (-19)'
'Scrubbing Free RAM on 1nodes using 8 CPUs'

adding a 'noreboot' option to the grub command line just leaves me with a hung system with a black screen.
I have subsequently booted with a live USB and checked /var/log/messages, /var/log/boot.log and /var/log/Xorg.0.log - I cant find anything that looks like  they have even been written to during the failed boot - nothing about a panic or other terminal failure message at the ends.

but it *will* boot with:
'Fedora, with Xen 4.9.2.config and Linux 4.13.16-100.fc25.x86_64'

and appears to work normally.

I too see:
error: file '/grub2/i386-pc/module2.mod' not found
twice, and on BOTH kernels, despite the fc25 then continuing normally after that.

I am interested in a solution, even if that is to continue upgrading to fc28 - if it works.

Hi,
I upgraded to Fedora 29 this week, and problems is still here, kernel won't boot under Xen hypervisor.

Can anyone from Fedora team look into this ?
It has been here since at least 2 releases.

Also, i personally need to run Xen for both security and development purposes and i just can't do it on Fedora.

Thank you in advance.

(In reply to mathieu.tarral from comment #44)
> Hi,
> I upgraded to Fedora 29 this week, and problems is still here, kernel won't
> boot under Xen hypervisor.
> 
> Can anyone from Fedora team look into this ?
> It has been here since at least 2 releases.
> 
> Also, i personally need to run Xen for both security and development
> purposes and i just can't do it on Fedora.
> 
> Thank you in advance.

XEN has been working for me. The problem was fixed some time ago. I would recommend carefully reviewing your error messages and configuration.

Hi Lloyd, thanks for your reply, and for confirming that you have a working installation of Xen.

This what I know about my boot issue:
before it was due to the 'module2' grub command not found, but this has been fixed, as you said.

And since I upgraded from f27 to f28 I think, the error changed to the following:

error: ../../grub-core/kern/efi/tpm.c:255:Unknown TPM error

image: https://user-images.githubusercontent.com/964610/48011149-60426580-e11f-11e8-8e85-0914fd11d2ea.JPG

I thought i was on the right bug report, but there was another one, where I already replied :)
https://bugzilla.redhat.com/show_bug.cgi?id=1575282

But thanks !

Another update, I managed to boot on Xen !! \o/

But I had to give up secure boot along the way...

Surprisingly, I still have this weird error with the 'module2.mod' not being found, but then I press ENTER and Xen manages to boot anyway.

Image: https://user-images.githubusercontent.com/964610/48012600-b06ef700-e122-11e8-9a55-b34809905af8.JPG

I'm not sure what is going on here, UEFI + grub2 + Xen does not work for me.

Fedora 29
kernel*-4.20.5-200.fc29.x86_64
grub2-(efi-x64}*-2.02-62.fc29.x86_64
shim-x64-15-7.x86_64
xen*-4.11.1-1.fc29.x86_64

I get the same issues everyone else has reported:

`/EFI/fedora/x86_64-efi/module2.mod' not found.
and
`/EFI/fedora/x86_64-efi/multiboot.mod' not found.

/etc/grub.d/20_linux_xen requests both "module2.mod" and "multiboot2.mod"

I don't know enough about UEFI, grub2 or xen, but it seems like either /etc/grub.d/20_linux_xen needs to be modified to not request "module2.mod" and "multiboot2.mod" *or* those modules need to be built and supplied. This has going on for a long time, what could be a higher priority than booting the system? This should be escalated to the highest priority.

(In reply to Roy A. Gilmore from comment #48)
> I'm not sure what is going on here, UEFI + grub2 + Xen does not work for me.
> 
> Fedora 29
> kernel*-4.20.5-200.fc29.x86_64
> grub2-(efi-x64}*-2.02-62.fc29.x86_64
> shim-x64-15-7.x86_64
> xen*-4.11.1-1.fc29.x86_64
> 
> I get the same issues everyone else has reported:
> 
> `/EFI/fedora/x86_64-efi/module2.mod' not found.
> and
> `/EFI/fedora/x86_64-efi/multiboot.mod' not found.
> 
> /etc/grub.d/20_linux_xen requests both "module2.mod" and "multiboot2.mod"
> 
> I don't know enough about UEFI, grub2 or xen, but it seems like either
> /etc/grub.d/20_linux_xen needs to be modified to not request "module2.mod"
> and "multiboot2.mod" *or* those modules need to be built and supplied. This
> has going on for a long time, what could be a higher priority than booting
> the system? This should be escalated to the highest priority.

OK, I just downloaded the grub2 srpm and poked around a little bit, and apparently multiboot and multiboot2 were disable on purpose. This brings up a few questions: 1) why were multiboot and multiboot2 disabled, and 2) if there was a valid reason why they were disabled, why wasn't /etc/grub.d/20_linux_xen modified at the same time, and finally 3) did QA sign off on this? As shipped, grub2 will not boot xen on efi, doesn't anyone test these packages before they are released.

Per the changelog, the reason multiboot modules are disabled in the EFI build is that they allow execution of arbitrary code, breaking secure boot:

https://bugzilla.redhat.com/show_bug.cgi?id=1264103

"2) if there was a valid reason why they were disabled, why wasn't /etc/grub.d/20_linux_xen modified at the same time"

pjones probably didn't notice this issue. xen isn't that widely used or tested on Fedora.

"3) did QA sign off on this?"

not explicitly. QA does not get to "sign off" on every decision made in every package (or even in the most important packages), because there are far too many things going on for that to be plausible. We don't test Fedora with Xen every day, in fact we mainly rely on Konrad - who filed this bug - to do that testing (there is a deal in place that Xen remains in the Fedora release criteria so long as Konrad does the testing, or finds someone else to do it). Xen on UEFI is not considered a release-blocking feature for Fedora, per the release blocker bug proposal discussion above. (Though if fixing this is as simple as taking the multiboot bit out of 20_linux_xen, we should probably do that.)

Fedora != RHEL

Or is expected that from a SecureBoot perspective it has everything that RHEL has (like the patcheset that locks down the kernel which Linus has very strongly Nacked?)

(In reply to Adam Williamson from comment #50)
> Per the changelog, the reason multiboot modules are disabled in the EFI
> build is that they allow execution of arbitrary code, breaking secure boot:
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1264103
> 

OK, fair enough, although in my opinion, secure boot is a poorly thought out specification that breaks as much (or more) than it fixes, and I don't care if packages break secure boot, because secure boot is broken by design. Regardless, Xen has been able to be built as EFI binaries since version 4.3, and Xen has only been able to built with multiboot2 support since version 4.9. Fedora currently supplies version 4.11, which can still be built as EFI binaries, completely bypassing grub2, multiboot2, and this secure boot issue. If Fedora requires secure boot, then Fedora should build Xen as EFI binaries.

> "2) if there was a valid reason why they were disabled, why wasn't
> /etc/grub.d/20_linux_xen modified at the same time"
> 
> pjones probably didn't notice this issue. xen isn't that widely used or
> tested on Fedora.
> 

How could anyone *not* notice this issue? Install the stock Xen hypervisor from the standard Fedora repos on a stock Fedora UEFI system, and the Xen hypervisor will not boot. It's impossible to miss, *if* someone actually tries it. Does Fedora build packages and not bother to test them?

Where did this statistic come from? Who says that Xen isn't "widely used" on Fedora. Nobody has ever asked me (or anyone I know) what packages I (we) use, or what packages I (we) want added or dropped. Just because the steering committee decided that kvm should be the only fully supported hypervisor on Fedora, doesn't mean end-users haven't compiled Xen from source to bypass Fedora's broken implementation. Anyone actually using a hypervisor probably knows how to compile one.

> "3) did QA sign off on this?"
> 
> not explicitly. QA does not get to "sign off" on every decision made in
> every package (or even in the most important packages), because there are
> far too many things going on for that to be plausible. We don't test Fedora
> with Xen every day, in fact we mainly rely on Konrad - who filed this bug -
> to do that testing (there is a deal in place that Xen remains in the Fedora
> release criteria so long as Konrad does the testing, or finds someone else
> to do it). Xen on UEFI is not considered a release-blocking feature for
> Fedora, per the release blocker bug proposal discussion above. (Though if
> fixing this is as simple as taking the multiboot bit out of 20_linux_xen, we
> should probably do that.)

QA doesn't need to sign off on every decision made in every package, that's management's/steering committee's or engineering's job. QA's job is to test and verify that every package works as advertised. If QA isn't testing and verifying that every package works as advertised, what are they doing that's interfering with their primary function?. Packages should not be *released* to the "wild" until QA has signed off on them. I'm former career military, one of my duties was QA, and I certified aircraft as "safe for flight". I understand QA's job very well because if I made a mistake, people could $#@!ing die, and *nobody* dies on my watch. Military aircraft are for more complex than Fedora, and have far more things "going on". Not only is testing and verifying every package plausible, it's possible, and if it's not done, it's because people *choose* not to.

"Does Fedora build packages and not bother to test them?"

Quite a lot of packagers seem to, yes. But that's hardly necessary in this case: whoever is packaging xen could have tested it diligently...on a BIOS system.

"Where did this statistic come from?"

It's not a statistic, per se...

"Who says that Xen isn't "widely used" on Fedora."

...I do. Based on my experience that I get asked about issues using Fedora on VMWare and VirtualBox *all the time*, hyper-v and KVM somewhat less often, and Xen almost never.

"QA doesn't need to sign off on every decision made in every package, that's management's/steering committee's or engineering's job"

Practically speaking, in Fedora, it's no-one's job, because it's impossible. There is far too much stuff in Fedora and we don't have a zillion people in "management" or "steering committee" or even in "engineering". Packagers are, broadly, responsible for ensuring their packages work. QA does its best to make sure the overall distribution works as well as possible, within the confines of the distribution being gigantic and QA being very tiny.

"If QA isn't testing and verifying that every package works as advertised, what are they doing that's interfering with their primary function?"

That's not our primary function. There are tens of thousands of packages in the distro. There are tens of people in QA. You can probably figure out the math on that one.

"Packages should not be *released* to the "wild" until QA has signed off on them. I'm former career military, one of my duties was QA, and I certified aircraft as "safe for flight". I understand QA's job very well because if I made a mistake, people could $#@!ing die, and *nobody* dies on my watch. Military aircraft are for more complex than Fedora, and have far more things "going on"."

Funnily enough I actually doubt that's true; I also doubt they change anywhere near as often, or have anywhere near as many supported (kinda) configurations. You wouldn't, for instance, permit a plane to have three entirely different combinations of engines it could use depending on the whim of the pilot, yet here we are with about a half dozen virtualization systems to be worried about, never mind container runtimes. I also suspect there are a damn sight more people involving in doing QA on any given aircraft than there are involved in doing QA on Fedora. :P

Basically: no, there is no guarantee that every change in Fedora is "signed off" or tested by someone (other than the person making it, and frankly, not really even them). If that is what you expect from software, Fedora is not the software for you.

"Not only is testing and verifying every package plausible, it's possible, and if it's not done, it's because people *choose* not to."

The "choice" in this case is that no-one is paying for several hundred or thousand QA people to do this. Since we don't have those resources, we target the resources we have as best we can.

> "Who says that Xen isn't "widely used" on Fedora."
>
> ...I do. Based on my experience that I get asked about issues using Fedora on VMWare and VirtualBox 
> *all the time*, hyper-v and KVM somewhat less often, and Xen almost never.

This doesn't necessarily mean that Fedora guests on Xen aren't widely used. A lot of the Fedora questions
(as well as questions for other distros) tend to get raised on Xen lists. Because we tend to answer these, 
people don't often go to distro lists. Then there are downstreams such as QubesOS that use Fedora under the
hood, cloud services and commercial products such as XenServer, where Fedora is a key part of the offering, 
but questions don't necessarily come up through Fedora user channels.

I think that both Xen Dom0 hosts and DomU guests should be working correctly now on after Bugs #1703700, #1703872 and #1761799 were fixed.

If that's not the case please re-open this bugzilla or file a new bug with the existing issues.

just revisiting this on

  F32
  grub2-efi-x64-2.04-23
  xen-4.13.2-2.fc32.x86_64
  kernel-core-5.9.10-100.fc32.x86_64


booot to Xen fails,

Loading Xen 4.13.2 ...
error: ../../grub-core/fs/fshelp.c:257:file
`/EFI/fedora/x86_64-efi/module2.mod' not
found.
Loading Linux 5.9.10-100.fc32.x86_64 ...
Loading initial ramdisk ...
error: ../../grub-core/fs/fshelp.c:257:file
`/EFI/fedora/x86_64-efi/module2.mod' not
found.

Press any key to continue...
0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa14d0018
0x0000:0x04:0x00.0x0: ROM: 0x8000 bytes at 0xa14c7018
0x0000:0x10:0x00.0x0: ROM: 0x10800 bytes at 0xa14a5018
 Xen 4.13.2
...

boot to non-Xen, kernel is OK.

b4 opening yet another issue,
*has* this been resolved perhaps elsewhere?  is there a known-to-work Xen on UEFI case on F32?

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days