Bug 1858364

Summary: grub2 boot error: "/EFI/fedora/x86_64-efi/module2.mod' not found." on EFI + Xen
Product: [Fedora] Fedora Reporter: pgnet.dev
Component: grub2Assignee: Javier Martinez Canillas <fmartine>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: Cameronsplaze222, fmartine, jforbes, lkundrak, m.a.young, ngompa13, pasik, pjones, robinlee.sysu, waffshappen
Target Milestone: ---Keywords: Reopened
Target Release: ---Flags: pgnet.dev: needinfo-
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: grub2-2.04-35.fc34 grub2-2.04-32.fc33 grub2-2.04-24.fc32 grub2-2.06~rc1-4.fc35 grub2-2.06~rc1-2.fc33 grub2-2.06~rc1-4.fc34 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-06-11 00:37:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
20_linux_xen none

Description pgnet.dev 2020-07-17 16:46:25 UTC
I've a new install of

	egrep "^NAME|^VERSION" /etc/os-release
	NAME=Fedora
	VERSION="32 (Thirty Two)"
	VERSION_ID=32
	VERSION_CODENAME=""

I boot with grub2 on UEFI

	ls -al /boot/efi/EFI/fedora/grub.cfg
		-rwx------ 1 root root 33K Jul 17 08:45 /boot/efi/EFI/fedora/grub.cfg*

I've installed Xen,

	rpm -qa | grep xen | sort
		xen-4.13.1-4.fc32.x86_64
		xen-hypervisor-4.13.1-4.fc32.x86_64
		xen-libs-4.13.1-4.fc32.x86_64
		xen-licenses-4.13.1-4.fc32.x86_64
		xen-runtime-4.13.1-4.fc32.x86_64


Xen boot fails,

	Loading Xen 4.13.1 ...
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/module2.mod' not
	found.
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/multiboot2.mod' not
	found.
	error: ../../grub-core/script/function.c:109:can't find command `multiboot2'.
	Loading Linux 5.7.8-200.fc32.x86_64 ...
	error: ../../grub-core/script/function.c:109:can't find command `module2'.
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/module2.mod' not
	found.
	error: ../../grub-core/script/function.c:109:can't find command `module2'.

	Press any key to continue...

and does not continue.

per,

 Bug 1486002 - grub2-mkconfig does not work if xen.gz is installed
  https://bugzilla.redhat.com/show_bug.cgi?id=1486002

after installing

	dnf install grub2-efi-x64-modules

	rpm -qa | grep grub2
		grub2-common-2.04-21.fc32.noarch
		grub2-efi-x64-2.04-21.fc32.x86_64
!		grub2-efi-x64-modules-2.04-21.fc32.noarch
		grub2-tools-2.04-21.fc32.x86_64
		grub2-tools-extra-2.04-21.fc32.x86_64
		grub2-tools-minimal-2.04-21.fc32.x86_64


& manually copying

	cp -af \
	 /usr/lib/grub/x86_64-efi \
	 /boot/efi/EFI/fedora/

on reboot, the missing "multiboot2.mod" is resolved, but still get

	Loading Xen 4.13.1 ...                                                                                                                                                                                                                                      
	error: ../../grub-core/fs/fshelp.c:257:file                                                                                                                                                                                                                 
	`/EFI/fedora/x86_64-efi/module2.mod' not                                                                                                                                                                                                                    
	found.                                                                                                                                                                                                                                                      
	Loading Linux 5.7.8-200.fc32.x86_64 ...                                                                                                                                                                                                                     
	Loading initial ramdisk ...                                                                                                                                                                                                                                 
	error: ../../grub-core/fs/fshelp.c:257:file                                                                                                                                                                                                                 
	`/EFI/fedora/x86_64-efi/module2.mod' not                                                                                                                                                                                                                    
	found.                                                                                                                                                                                                                                                      
	                                                                                                                                                                                                                                                            
	Press any key to continue...

on launch, but after a delay, Xen boots successfully

	xl list
		Name                                        ID   Mem VCPUs      State   Time(s)
		Domain-0                                     0  4016     4     r-----      34.3
		Xenstore                                     1    31     1     -b----       0.0

checking,

	ls -al /boot/efi/EFI/fedora/x86_64-efi/{multiboot,module}*
		ls: cannot access '/boot/efi/EFI/fedora/x86_64-efi/module*': No such file or directory
		-rwx------ 1 root root 38K Jun 19 09:24  /boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod*
		-rwx------ 1 root root 34K Jun 19 09:24  /boot/efi/EFI/fedora/x86_64-efi/multiboot.mod*

(1) xen pkg install should require grub2-efi-x64-modules dependency
    at least, conditionally if UEFI

(2) the grub2-efi-x64-modules pkg should install/upgrade the mods to req'd location,

	/boot/efi/EFI/fedora/x86_64-efi

manual copy won't correctly track future upgrades

(3) module2.mod is still reported as missing.

The issue was seen/raised in 2017

 Bug 1515694 - Xen boot entries ask for non-exisiting grub2 module2.mod
  https://bugzilla.redhat.com/show_bug.cgi?id=1515694

unresolved and auto-closed, as frequent,

	Status: 	CLOSED EOL  

then seen/reported again in 2018 & never resolved

Comment 1 pgnet.dev 2020-08-19 02:39:50 UTC
no response from the grub2 folks, so switching to xen

this^ is still an issue with latest kernel 5.7.15,

serial console output @ boot

	Loading Xen 4.13.1 ...
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/module2.mod' not
	found.
	Loading Linux 5.7.15-200.fc32.x86_64 ...
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/module2.mod' not
	found.

	Press any key to continue...
	0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa14d0018
	0x0000:0x04:0x00.0x0: ROM: 0x8000 bytes at 0xa14c7018
	0x0000:0x10:0x00.0x0: ROM: 0x10800 bytes at 0xa14a5018
	 Xen 4.13.1
	(XEN) [000000189c9b79e6] Xen version 4.13.1 (mockbuild@[unknown]) (gcc (GCC) 10.1.1 20200507 (Red Hat 10.1.1-1)) debug=n  Tue Jul  7 18:36:24 UTC 2020
	(XEN) [000000189f26a2be] Latest ChangeSet: 
	(XEN) [000000189fe4d27e] build-id: 131db801fd9d8ed1fbc017e12b9a13570043e404
	(XEN) [00000018a1272746] Bootloader: GRUB 2.04
	...
	         Starting User Manager for UID 0...
	[  OK  ] Started User Manager for UID 0.

	Fedora 32 (Server Edition)
	Kernel 5.7.15-200.fc32.x86_64 on an x86_64 (hvc0)

	svr03 login:


xl list
	Name                                        ID   Mem VCPUs      State   Time(s)
	Domain-0                                     0  4016     4     r-----      28.9
	Xenstore                                     1    31     1     -b----       0.0

Comment 2 pgnet.dev 2021-01-26 00:38:25 UTC
These warnings + stall/delay on boot

  `/EFI/fedora/x86_64-efi/module2.mod' not found.

continue with current F33 + xen 4.14.1


there's been no response to prior inquiries here, or

  https://bugzilla.redhat.com/show_bug.cgi?id=1486002

or,

  https://www.mail-archive.com/devel@lists.fedoraproject.org/msg157252.html

it doesn't _appear_ that xen pkgs have been abandoned,

  https://src.fedoraproject.org/rpms/xen


can someone in dev/packaging kindly comment ?

Comment 3 Michael Young 2021-01-26 20:37:59 UTC
You need relocator.mod as well as multiboot2.mod . The xen-hypervisor package should be doing this for you on install or update provided you have the right grub2 packages installed.

Comment 4 pgnet.dev 2021-01-26 20:57:22 UTC
> You need relocator.mod as well as multiboot2.mod . The xen-hypervisor package should be doing this for you on install or update provided you have the right grub2 packages installed.

they're all here ...


i've got

	rpm -qa | grep xen
		xen-4.14.1-2.fc33.x86_64
		xen-hypervisor-4.14.1-2.fc33.x86_64
		xen-libs-4.14.1-2.fc33.x86_64
		xen-licenses-4.14.1-2.fc33.x86_64
		xen-runtime-4.14.1-2.fc33.x86_64

	rpm -qa | grep -i grub2
		grub2-common-2.04-31.fc33.noarch
		grub2-efi-x64-2.04-31.fc33.x86_64
		grub2-efi-x64-modules-2.04-31.fc33.noarch
		grub2-tools-2.04-31.fc33.x86_64
		grub2-tools-efi-2.04-31.fc33.x86_64
		grub2-tools-extra-2.04-31.fc33.x86_64
		grub2-tools-minimal-2.04-31.fc33.x86_64

installed

with

	ls -al /usr/lib/grub/x86_64-efi/ | egrep "relocator|multiboot"
		-rw-r--r--  1 root root  38K Aug 31 08:12 multiboot2.mod
		-rw-r--r--  1 root root  34K Aug 31 08:12 multiboot.mod
		-rw-r--r--  1 root root  40K Aug 31 08:12 relocator.mod

where

	rpm -q --whatprovides \
	/usr/lib/grub/x86_64-efi/multiboot2.mod \
	/usr/lib/grub/x86_64-efi/multiboot.mod \
	/usr/lib/grub/x86_64-efi/relocator.mod

		grub2-efi-x64-modules-2.04-31.fc33.noarch
		grub2-efi-x64-modules-2.04-31.fc33.noarch
		grub2-efi-x64-modules-2.04-31.fc33.noarch


with those in place, I still see

	Loading Xen 4.14.1 ...
	error: ../../grub-core/fs/fshelp.c:257:file  `/EFI/fedora/x86_64-efi/module2.mod' not found.
	Loading Linux 5.10.9-201.fc33.x86_64 ...
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file  `/EFI/fedora/x86_64-efi/module2.mod' not found.

	Press any key to continue ...

After ~ 10-20 seconds hang/delay, it simply autocontinues.

No such issues/error/delay if booting to NON-xen.

Comment 5 Michael Young 2021-01-26 22:04:50 UTC
I am thinking of the wrong problem. The module2.mod lines are incorrectly added to grub.cfg by the grub2 20_linux_xen script (in some versions of grub2 at least). It is again something the xen-hypervisor should be fixing for you on install or update. You should also do it manually by running sed -i -e '/insmod module2/d' /boot/efi/EFI/fedora/grub.cfg (though you might want to take a copy of /boot/efi/EFI/fedora/grub.cfg first for safety).

Comment 6 pgnet.dev 2021-01-26 22:12:05 UTC
> the xen-hypervisor should be fixing for you on install or update

Sure.  That's the point.  Something -- either in grub &/or xen, @ upstream or Fed pkgs needs to get fixed.

> You should also do it manually by running sed ...

Not something that scales to production.

Comment 7 Neal Gompa 2021-02-05 13:47:45 UTC
This is a grub2 bug and needs to be fixed there.

Comment 8 Neal Gompa 2021-02-05 13:48:36 UTC
Clearing useless needinfo

Comment 9 Javier Martinez Canillas 2021-02-05 15:06:46 UTC
Yes, this is definitely a bug in the grub2 package. I'm assigning the BZ to me.

Comment 10 Javier Martinez Canillas 2021-02-05 16:33:38 UTC
I looked at this and the bug is in the 20_linux_xen script as Michael Young mentioned in Comment 5.

The problem is that the script wrongly try to load the commands used to load the Xen kernel and
modules (multiboot2, module2 for x86_64 and xen_hypervisor, xen_module for aarch64) instead of
the GRUB modules that register these commands (module2 for x86_64 and xen_boot for aarch64).

I think that have a fix but don't have a Xen setup to verify it. I'll share a build for you to test.

Comment 11 Javier Martinez Canillas 2021-02-05 16:34:53 UTC
(In reply to Javier Martinez Canillas from comment #10)

[snip]

> the GRUB modules that register these commands (module2 for x86_64 and
> xen_boot for aarch64).
> 

I meant here multiboot2 for x86_64 and xen_boot for aarch64.

Comment 12 Javier Martinez Canillas 2021-02-05 16:48:59 UTC
Created attachment 1755264 [details]
20_linux_xen

Actually, you just need the 20_linux_xen script.

Could you please test with the attached file ? Just copy it to the /etc/grub.d/ directory.

Comment 13 pgnet.dev 2021-02-05 17:05:35 UTC
with your attached 20_linux_xen in place,

instead of the prior

	...
	Loading Xen 4.13.1 ...
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/module2.mod' not
	found.
	Loading Linux 5.7.15-200.fc32.x86_64 ...
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file
	`/EFI/fedora/x86_64-efi/module2.mod' not
	found.

	Press any key to continue...
	0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa14d0018
	0x0000:0x04:0x00.0x0: ROM: 0x8000 bytes at 0xa14c7018
	0x0000:0x10:0x00.0x0: ROM: 0x10800 bytes at 0xa14a5018
	 Xen 4.13.1
	(XEN) [000000189c9b79e6] Xen version 4.13.1 (mockbuild@[unknown]) (gcc (GCC) 10.1.1 20200507 (Red Hat 10.1.1-1)) debug=n  Tue Jul  7 18:36:24 UTC 2020
	(XEN) [000000189f26a2be] Latest ChangeSet: 
	(XEN) [000000189fe4d27e] build-id: 131db801fd9d8ed1fbc017e12b9a13570043e404
	(XEN) [00000018a1272746] Bootloader: GRUB 2.04
	...

i now see,

	...
	Loading Xen 4.14.1 ...
	Loading Linux 5.10.11-200.fc33.x86_64 ...
	Loading initial ramdisk ...
	0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa148f018
	0x0000:0x03:0x00.0x0: ROM: 0x8000 bytes at 0xa147f018
	0x0000:0x0f:0x00.0x0: ROM: 0x10800 bytes at 0xa145e018
	Xen 4.14.1
	(XEN) [00000032c47cb937] Xen version 4.14.1 (mockbuild@[unknown]) (gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)) debug=n  Th1
	(XEN) [00000032cbcd3d2f] Latest ChangeSet: 
	(XEN) [00000032cdf55ebf] build-id: d97d91a3bacda1305963b1acf3948929e27ca63d
	(XEN) [00000032d1ac4803] Bootloader: GRUB 2.04
	...

no error, no delay

Comment 14 Javier Martinez Canillas 2021-02-09 01:01:35 UTC
Thanks a lot for testing. The patch was included in the grub2-2.04-35.fc34 build.

Comment 15 pgnet.dev 2021-02-09 01:32:15 UTC
This was reported against <=F33.

Please setup builds for F33.

Comment 16 Fedora Update System 2021-02-09 09:21:13 UTC
FEDORA-2021-cc918afb9b has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-cc918afb9b

Comment 17 Fedora Update System 2021-02-09 10:28:57 UTC
FEDORA-2021-b0e8030da9 has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2021-b0e8030da9

Comment 18 Javier Martinez Canillas 2021-02-09 10:30:38 UTC
(In reply to pgnet.dev from comment #15)
> This was reported against <=F33.
> 
> Please setup builds for F33.

Done

Comment 19 Ben Cotton 2021-02-09 16:11:13 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.
Changing version to 34.

Comment 20 pgnet.dev 2021-02-09 17:26:11 UTC
> This bug appears to have been reported against 'rawhide' during the Fedora 34 development cycle.

no.

not that it matters to me as long as a CURRENT RELEASE fix finally gets built, but it was reported originally against F27/grub2

	https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c21

i commented there,

	https://bugzilla.redhat.com/show_bug.cgi?id=1486002#c56

with no further reply

@ #irc told me it was a xen issue.
i then opened THIS bug against F32, on 2020-07-17,  @ component xen -- after

if was then subsequently changed here  -- NOT by me -- 

	https://bugzilla.redhat.com/show_bug.cgi?id=1858364#c7

		CC: ngompa13
		Component: xen → grub2
		Version: 32 → rawhide

to rawhide/grub2

Comment 21 Fedora Update System 2021-02-10 00:44:21 UTC
FEDORA-2021-cc918afb9b has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-cc918afb9b`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-cc918afb9b

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 22 Fedora Update System 2021-02-10 01:47:13 UTC
FEDORA-2021-b0e8030da9 has been pushed to the Fedora 32 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-b0e8030da9`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-b0e8030da9

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 23 Fedora Update System 2021-02-11 01:43:10 UTC
FEDORA-2021-cc918afb9b has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 24 Fedora Update System 2021-02-17 05:09:06 UTC
FEDORA-2021-b0e8030da9 has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 25 Michael Young 2021-02-17 23:05:35 UTC
*** Bug 1921747 has been marked as a duplicate of this bug. ***

Comment 26 pgnet.dev 2021-04-11 11:14:36 UTC
on

	grep PRETTY /etc/os-release
		PRETTY_NAME="Fedora 33 (Thirty Three)"

	uname -rm
		5.11.11-200.fc33.x86_64 x86_64

with

	rpm -qa | grep xen | sort
		xen-4.14.1-7.fc33.x86_64
		xen-hypervisor-4.14.1-7.fc33.x86_64
		xen-libs-4.14.1-7.fc33.x86_64
		xen-licenses-4.14.1-7.fc33.x86_64
		xen-runtime-4.14.1-7.fc33.x86_64

after upgrading grub* 2.04x to 2.06x,

	rpm -qa | grep -i grub2 | sort
		grub2-common-2.06~rc1-1.fc33.noarch
		grub2-efi-x64-2.06~rc1-1.fc33.x86_64
		grub2-efi-x64-modules-2.06~rc1-1.fc33.noarch
		grub2-tools-2.06~rc1-1.fc33.x86_64
		grub2-tools-efi-2.06~rc1-1.fc33.x86_64
		grub2-tools-extra-2.06~rc1-1.fc33.x86_64
		grub2-tools-minimal-2.06~rc1-1.fc33.x86_64

with 

	ls -al /usr/lib/grub/x86_64-efi/ | egrep "relocator|multiboot"
		-rw-r--r--  1 root root  38K Apr  6 12:17 multiboot2.mod
		-rw-r--r--  1 root root  34K Apr  6 12:17 multiboot.mod
		-rw-r--r--  1 root root  40K Apr  6 12:17 relocator.mod

	rpm -q --whatprovides \
	/usr/lib/grub/x86_64-efi/multiboot2.mod \
	/usr/lib/grub/x86_64-efi/multiboot.mod \
	/usr/lib/grub/x86_64-efi/relocator.mod

		grub2-efi-x64-modules-2.06~rc1-1.fc33.noarch
		grub2-efi-x64-modules-2.06~rc1-1.fc33.noarch
		grub2-efi-x64-modules-2.06~rc1-1.fc33.noarch

on boot to xen

	Loading Xen 4.14.1 ...
	error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
	error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.
	error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.
	Loading Linux 5.11.11-200.fc33.x86_64 ...
	error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
	error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.

	Press any key to continue...

currently 

	rpm -q --whatprovides /etc/grub.d/20_linux_xen
		grub2-tools-2.06~rc1-1.fc33.x86_64

diffing with your mod'd version from Comment #12 (https://bugzilla.redhat.com/show_bug.cgi?id=1858364#c12)


diff -ur /tmp/20_linux_xen /etc/grub.d/20_linux_xen
	--- /tmp/20_linux_xen   2021-04-11 07:04:17.900084046 -0400
	+++ /etc/grub.d/20_linux_xen    2021-04-06 12:17:44.000000000 -0400
	@@ -93,12 +93,29 @@
	
	linux_entry ()
	{
	+  linux_entry_xsm "$@" false
	+  linux_entry_xsm "$@" true
	+}
	+linux_entry_xsm ()
	+{
	os="$1"
	version="$2"
	xen_version="$3"
	type="$4"
	args="$5"
	xen_args="$6"
	+  xsm="$7"
	+  # If user wants to enable XSM support, make sure there's
	+  # corresponding policy file.
	+  if ${xsm} ; then
	+      xenpolicy="xenpolicy-$xen_version"
	+      if test ! -e "${xen_dirname}/${xenpolicy}" ; then
	+         return
	+      fi
	+      xen_args="$xen_args flask=enforcing"
	+      xen_version="$(gettext_printf "%s (XSM enabled)" "$xen_version")"
	+      # xen_version is used for messages only; actual file is xen_basename
	+  fi
	if [ -z "$boot_device_id" ]; then
		boot_device_id="$(grub_get_device_id "${GRUB_DEVICE}")"
	fi
	@@ -136,7 +153,8 @@
			else
				xen_rm_opts="no-real-mode edd=off"
			fi
	-       insmod ${xen_module}
	+       insmod ${module_loader}
	+       insmod ${xen_loader}
			${xen_loader}   ${rel_xen_dirname}/${xen_basename} placeholder ${xen_args} \${xen_rm_opts}
			echo    '$(echo "$lmessage" | grub_quote)'
			${module_loader}        ${rel_dirname}/${basename} placeholder root=${linux_root_device_thisversion} ro ${args}
	@@ -150,10 +168,17 @@
		done
		sed "s/^/$submenu_indentation/" << EOF
			echo    '$(echo "$message" | grub_quote)'
	-       insmod ${xen_module}
	+       insmod ${module_loader}
			${module_loader}        --nounzip   $(echo $initrd_path)
	EOF
	fi
	+  if test -n "${xenpolicy}" ; then
	+    message="$(gettext_printf "Loading XSM policy ...")"
	+    sed "s/^/$submenu_indentation/" << EOF
	+       echo    '$(echo "$message" | grub_quote)'
	+       ${module_loader}     ${rel_dirname}/${xenpolicy}
	+EOF
	+  fi
	sed "s/^/$submenu_indentation/" << EOF
	}
	EOF
	@@ -179,10 +204,14 @@
		exit 0
	fi
	
	-file_is_not_sym () {
	+file_is_not_xen_garbage () {
		case "$1" in
			*/xen-syms-*)
				return 1;;
	+       */xenpolicy-*)
	+           return 1;;
	+       */*.config)
	+           return 1;;
			*)
				return 0;;
		esac
	@@ -190,7 +219,7 @@
	
	xen_list=
	for i in /boot/xen*; do
	-    if grub_file_is_not_garbage "$i" && file_is_not_sym "$i" ; then xen_list="$xen_list $i" ; fi
	+    if grub_file_is_not_garbage "$i" && file_is_not_xen_garbage "$i" ; then xen_list="$xen_list $i" ; fi
	done
	prepare_boot_cache=
	boot_device_id=
	@@ -227,16 +256,13 @@
			echo "  submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {"
		fi
		if ($grub_file --is-arm64-efi $current_xen); then
	-       xen_module="xen_boot"
			xen_loader="xen_hypervisor"
			module_loader="xen_module"
		else
			if ($grub_file --is-x86-multiboot2 $current_xen); then
	-           xen_module="multiboot2"
				xen_loader="multiboot2"
				module_loader="module2"
			else
	-           xen_module="multiboot"
				xen_loader="multiboot"
				module_loader="module"
			fi
	@@ -297,7 +323,15 @@
				fi
			fi

	-       if [ "x$is_top_level" = xtrue ] && [ "x${GRUB_DISABLE_SUBMENU}" != xy ]; then
	+       # The GRUB_DISABLE_SUBMENU option used to be different than others since it was
	+       # mentioned in the documentation that has to be set to 'y' instead of 'true' to
	+       # enable it. This caused a lot of confusion to users that set the option to 'y',
	+       # 'yes' or 'true'. This was fixed but all of these values must be supported now.
	+       if [ "x${GRUB_DISABLE_SUBMENU}" = xyes ] || [ "x${GRUB_DISABLE_SUBMENU}" = xy ]; then
	+           GRUB_DISABLE_SUBMENU="true"
	+       fi
	+
	+       if [ "x$is_top_level" = xtrue ] && [ "x${GRUB_DISABLE_SUBMENU}" != xtrue ]; then
				linux_entry "${OS}" "${version}" "${xen_version}" simple \
					"${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT}" "${GRUB_CMDLINE_XEN} ${GRUB_CMDLINE_XEN_DEFAULT}"


Does this need to be re-fixed/re-patched? Or has the approach changed?

Comment 27 pgnet.dev 2021-04-11 12:31:09 UTC
looks like the grub 2.04 -> 2.06rc updates are 'critically' needed, and mandated for f33 release.

so this^ is now breakage/regression. bumping severity.

Comment 28 Fedora Update System 2021-04-11 22:59:54 UTC
FEDORA-2021-32e598cfa6 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2021-32e598cfa6

Comment 29 Javier Martinez Canillas 2021-04-11 23:02:22 UTC
(In reply to pgnet.dev from comment #27)
> looks like the grub 2.04 -> 2.06rc updates are 'critically' needed, and
> mandated for f33 release.
> 
> so this^ is now breakage/regression. bumping severity.

This was dropped by mistake when rebasing to 2.06~rc1. I've pushed a
grub2-2.06~rc1-2.fc33 that re-adds the fix. Sorry about that.

Comment 30 Fedora Update System 2021-04-11 23:23:13 UTC
FEDORA-2021-ea80a01b49 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea80a01b49

Comment 31 Fedora Update System 2021-04-11 23:37:20 UTC
FEDORA-2021-591b1819b8 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 32 Fedora Update System 2021-04-12 15:09:38 UTC
FEDORA-2021-ea80a01b49 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-ea80a01b49`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-ea80a01b49

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 33 Fedora Update System 2021-04-12 16:42:10 UTC
FEDORA-2021-32e598cfa6 has been pushed to the Fedora 33 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-32e598cfa6`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-32e598cfa6

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 34 pgnet.dev 2021-04-12 19:06:30 UTC
> FEDORA-2021-32e598cfa6 has been pushed to the Fedora 33 testing repository.

After updating to

	dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-32e598cfa6

	rpm -qa | grep -i grub |sort
		grub2-common-2.06~rc1-2.fc33.noarch
		grub2-efi-x64-2.06~rc1-2.fc33.x86_64
		grub2-efi-x64-modules-2.06~rc1-2.fc33.noarch
		grub2-tools-2.06~rc1-2.fc33.x86_64
		grub2-tools-efi-2.06~rc1-2.fc33.x86_64
		grub2-tools-extra-2.06~rc1-2.fc33.x86_64
		grub2-tools-minimal-2.06~rc1-2.fc33.x86_64
		grubby-8.40-47.fc33.x86_64

then, re-generating initrd, on boot to Xen, still


	Loading Xen 4.14.1 ...
		error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
		error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.
		error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.
		Loading Linux 5.11.11-200.fc33.x86_64 ...
		error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.
		Loading initial ramdisk ...
		error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
		error: ../../grub-core/kern/dl.c:429:symbol `grub_efi_secure_boot' not found.

		Press any key to continue...

Comment 35 Javier Martinez Canillas 2021-04-12 19:10:42 UTC
Did yu re-generate your grub.cfg file ?

Comment 36 Javier Martinez Canillas 2021-04-12 19:12:39 UTC
also, the EFI modules are not copied to /boot/efi/EFI/fedora/ by RPM. You need to manually copy them
from /usr/lib/grub/x86_64-efi/ to /boot/efi/EFI/fedora/x86_64-efi/

Comment 37 pgnet.dev 2021-04-12 19:22:16 UTC
> Did yu re-generate your grub.cfg file ?

yep

> the EFI modules are not copied to /boot/efi/EFI/fedora/ by RPM. You need to manually copy them
from /usr/lib/grub/x86_64-efi/ to /boot/efi/EFI/fedora/x86_64-efi/

That's new?

Each time a grub update is installed?

Comment 38 pgnet.dev 2021-04-12 19:28:02 UTC
after 

	/bin/cp -af /usr/lib/grub/x86_64-efi/* /boot/efi/EFI/fedora/x86_64-efi/

and re-gen initrd/grub cfg, reboot to Xen, still

	Loading Xen 4.14.1 ...
	error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
	Loading Linux 5.11.11-200.fc33.x86_64 ...
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.

but now proceeds

	Press any key to continue...
	0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa148f018
	0x0000:0x03:0x00.0x0: ROM: 0x8000 bytes at 0xa147f018
	0x0000:0x0f:0x00.0x0: ROM: 0x10800 bytes at 0xa145e018
	Xen 4.14.1
	(XEN) [0000003671416259] Xen version 4.14.1 (mockbuild@[unknown]) (gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)) debug=n  Th1
	(XEN) [0000003678957f29] Latest ChangeSet: 
	(XEN) [000000367abd9795] build-id: 50fc0f4876a53584be3e6ba3015a8e668490a422
	...


	xl list
		Name                                        ID   Mem VCPUs      State   Time(s)
		Domain-0                                     0  4015     4     r-----      50.5
		Xenstore                                     1    31     1     -b----       0.0

Comment 39 Michael Young 2021-04-12 19:43:16 UTC
(In reply to pgnet.dev from comment #37)
> > Did yu re-generate your grub.cfg file ?
> 
> yep
> 
> > the EFI modules are not copied to /boot/efi/EFI/fedora/ by RPM. You need to manually copy them
> from /usr/lib/grub/x86_64-efi/ to /boot/efi/EFI/fedora/x86_64-efi/
> 
> That's new?
> 
> Each time a grub update is installed?

I expect that copying the EFI modules is only likely to be needed after more significant grub updates, such as going from 2.04 to 2.06, as smaller updates are less likely to change the modules xen uses.

Comment 40 Javier Martinez Canillas 2021-04-12 21:59:53 UTC
(In reply to pgnet.dev from comment #38)
> after 
> 
> 	/bin/cp -af /usr/lib/grub/x86_64-efi/* /boot/efi/EFI/fedora/x86_64-efi/
> 
> and re-gen initrd/grub cfg, reboot to Xen, still
> 
> 	Loading Xen 4.14.1 ...
> 	error: ../../grub-core/fs/fshelp.c:257:file
> `/EFI/fedora/x86_64-efi/module2.mod' not found.
> 	Loading Linux 5.11.11-200.fc33.x86_64 ...
> 	Loading initial ramdisk ...
> 	error: ../../grub-core/fs/fshelp.c:257:file
> `/EFI/fedora/x86_64-efi/module2.mod' not found.
>

I don't understand why it continues to attempt to insmod a 'module2' GRUB module
instead of a 'multiboot2' if that should already be fixed. Can you please share
the Xen snippet in your GRUB config? For example I've the following in my Fedora
Rawhide install:

        insmod multiboot2
        multiboot2      /xen-4.14.1.gz placeholder   ${xen_rm_opts}
        echo    'Loading Linux 5.12.0-0.rc3.20210319git8b12a62a4e3e.172.fc35.x86_64 ...'
        module2 /vmlinuz-5.12.0-0.rc3.20210319git8b12a62a4e3e.172.fc35.x86_64 placeholder root=/dev/mapper/fedora-root ro resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap 
        echo    'Loading initial ramdisk ...'
        insmod multiboot2
        module2 --nounzip   /initramfs-5.12.0-0.rc3.20210319git8b12a62a4e3e.172.fc35.x86_64.img

Comment 41 Javier Martinez Canillas 2021-04-12 22:03:11 UTC
(In reply to pgnet.dev from comment #37)

[snip]

> That's new?
>

That's always has been the case. Loading GRUB modules on EFI is
really not a common case I believe.
 
> Each time a grub update is installed?

Yes... one option is for the grub2-efi-modules package to install
the modules, but we want to move in the opposite direction and not
make the grub2-efi-x64 package install the grubx64.efi binary in
the ESP either.

Probably the correct solution is to use bootupd [0] to install the
GRUB binaries and modules in /boot.

[0]: https://github.com/coreos/bootupd

Comment 42 Javier Martinez Canillas 2021-04-12 22:04:53 UTC
(In reply to Michael Young from comment #39)
> (In reply to pgnet.dev from comment #37)
> > > Did yu re-generate your grub.cfg file ?
> > 
> > yep
> > 
> > > the EFI modules are not copied to /boot/efi/EFI/fedora/ by RPM. You need to manually copy them
> > from /usr/lib/grub/x86_64-efi/ to /boot/efi/EFI/fedora/x86_64-efi/
> > 
> > That's new?
> > 
> > Each time a grub update is installed?
> 
> I expect that copying the EFI modules is only likely to be needed after more
> significant grub updates, such as going from 2.04 to 2.06, as smaller
> updates are less likely to change the modules xen uses.

That's correct. But you can never be sure if it will be compatible.

Comment 43 pgnet.dev 2021-04-12 22:46:50 UTC
> share the Xen snippet in your GRUB config

---------------
...
### BEGIN /etc/grub.d/20_linux_xen ###
menuentry 'Fedora, with Xen 4.14.1 and Linux 5.11.11-200.fc33.x86_64' --class fedora --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-5.11.11-200.fc33.x86_64-advanced-a4a5a182-d39e-4b09-8f76-25d56abc2875' {
        insmod part_gpt
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod ext2
        set root='mduuid/970f1cae5b028e5cb14657fa959d7ab9'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/970f1cae5b028e5cb14657fa959d7ab9'  8666cf64-2b07-4a27-957d-7cc2ece878a5
        else
          search --no-floppy --fs-uuid --set=root 8666cf64-2b07-4a27-957d-7cc2ece878a5
        fi
        echo    'Loading Xen 4.14.1 ...'
        if [ "$grub_platform" = "pc" -o "$grub_platform" = "" ]; then
            xen_rm_opts=
        else
            xen_rm_opts="no-real-mode edd=off"
        fi
        insmod multiboot2
        multiboot2      /xen-4.14.1.gz placeholder  dom0=pvh dom0-iommu=map-reserved dom0_max_vcpus=4 dom0_mem=4016M,max:4096M sched=credit2 reboot=acpi ucode=scan flask=disabled conring_size=16384k console_timestamps loglvl=all guest_loglvl=all iommu=verbose apic_verbosity=verbose console=com1,vga com1=38400,8n1,0x03f8,4 vga=gfx-1920x1080x32  ${xen_rm_opts}
        echo    'Loading Linux 5.11.11-200.fc33.x86_64 ...'
        module2 /vmlinuz-5.11.11-200.fc33.x86_64 placeholder root=/dev/mapper/VG0-ROOT ro   video=HDMI-0:2560x1440@60  nouveau.modeset=1 vconsole.keymap=us vconsole.font=eurlatgr vconsole.font_map=trivial   domdadm dolvm lvmwait=/dev/mapper/VG0-ROOT noresume  mitigations=auto transparent_hugepage=never max_loop=128 kdbus=0 apparmor=0 enforcing=0 selinux=0 rd.plymouth=0 plymouth.enable=0 audit=0 fsck.mode=auto fsck.repair=preen net.ifnames=1  scsi_mod.use_blk_mq=1  acpi_osi=Linux  mce=bootlog reboot=acpi libahci.ignore_sss=1   debug showopts noquiet print_fatal_signals=1 loglevel=5 rd.systemd.show_status=1  root=/dev/mapper/VG0-ROOT rootfstype=ext4 rootflags=journal_checksum rd.shell=1 rd.auto=1 rd.udev.log_priority=info systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on systemd.log_level=info loglevel=8 initcall_debug   spec-ctrl=mds=yes pv-l1tf=dom0=true,domu=true spec-ctrl=l1d-flush=true smt=true   console=hvc0 earlyprintk=xen  
        echo    'Loading initial ramdisk ...'
        insmod multiboot2
        module2 --nounzip   /initramfs-5.11.11-200.fc33.x86_64.img
}...
---------------


> That's always has been the case.

This is the 1st I've ever heard of it, tbh.

> Loading GRUB modules on EFI is really not a common case I believe.

??

booting grub on EFI hardware is pretty common.

all of my boxes boot grub / all of my boxes are EFI.

including my Xen servers, among others.

> but we want to move in the opposite direction and not

whatever that direction is, it really shouldn't involve manual file copying where distro pkgs are involved.
the assumption should be -- if distro pkg, then complete.
otherwise it's a guessing game, and safer to build from src.

Comment 44 Javier Martinez Canillas 2021-04-12 23:07:03 UTC
(In reply to pgnet.dev from comment #43)
> > share the Xen snippet in your GRUB config
> 

[snip]

>         insmod multiboot2
>         multiboot2      /xen-4.14.1.gz placeholder  dom0=pvh

It's correct now, I don't understand why you get errors about the
module2 not being found then if the grub.cfg isn't trying to load
that anymore...

> 
> 
> > That's always has been the case.
> 
> This is the 1st I've ever heard of it, tbh.
>

Yes, probably because as Michael said there weren't a lot of changes
and the old modules worked with the new grubx64.efi binaries updated.
 
> > Loading GRUB modules on EFI is really not a common case I believe.
> 
> ??
> 
> booting grub on EFI hardware is pretty common.
>

I did not say that. I said that loading EFI modules from a file system
is not a common case. To support Secure Boot, most of the needed GRUB
modules are built-in the EFI binary. For this reason most users don't
need to load any modules and can just use the grubx64.efi binary.
 
> all of my boxes boot grub / all of my boxes are EFI.
> 
> including my Xen servers, among others.
>

Yes, but you need to load modules because the multiboot2 isn't built-in
the grubx64.efi binary.
 
> > but we want to move in the opposite direction and not
> 
> whatever that direction is, it really shouldn't involve manual file copying
> where distro pkgs are involved.
> the assumption should be -- if distro pkg, then complete.
> otherwise it's a guessing game, and safer to build from src.

Agreed. I think the correct solution for this should be to teach bootupd
how to support updating a GRUB installation on EFI. But for the short term
may need to make the grub2-efi-modules to install these in /boot as the
grub2-efi packages do for the EFI binary.

In any case, that's a separate issue than the one discussed in this bug.

Comment 45 Fedora Update System 2021-04-13 14:29:59 UTC
FEDORA-2021-32e598cfa6 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 46 Fedora Update System 2021-04-24 20:09:21 UTC
FEDORA-2021-ea80a01b49 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 47 pgnet.dev 2021-04-28 00:21:31 UTC
dnf install https://rpms.remirepo.net/fedora/remi-release-34.rpm



> It's correct now, I don't understand why you get errors about the
> module2 not being found then if the grub.cfg isn't trying to load
> that anymore...

fwiw,

after a server upgrade from f33/Xen f34/Xen, on boot, pause, and an eventual, successful automatic continue to a complete boot:

...
[ 1224.340150] serial 00:05: shutdown
Loading Xen 4.14.1 ...
error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
Loading Linux 5.11.16-200.fc33.x86_64 ...
Loading initial ramdisk ...
error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.

Press any key to continue...
0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa148f018
0x0000:0x03:0x00.0x0: ROM: 0x8000 bytes at 0xa147f018
0x0000:0x0f:0x00.0x0: ROM: 0x10800 bytes at 0xa145e018
 Xen 4.14.1
(XEN) [00000036f5d4a496] Xen version 4.14.1 (mockbuild@[unknown]) (gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)) debug=n  Th1
(XEN) [00000036fd23db3a] Latest ChangeSet: 
(XEN) [00000036ff4c055a] build-id: 50fc0f4876a53584be3e6ba3015a8e668490a422
(XEN) [000000370302de96] Bootloader: GRUB 2.06~rc1
...


> In any case, that's a separate issue than the one discussed in this bug.

_Has_ an issue re: that^^ already been opened elsewhere? if so, got a link?

Comment 48 pgnet.dev 2021-06-10 20:22:40 UTC
still here, on F34:

	Loading Xen 4.14.2 ...
	error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.
	Loading Linux 5.12.9-300.fc34.x86_64 ...
	Loading initial ramdisk ...
	error: ../../grub-core/fs/fshelp.c:257:file `/EFI/fedora/x86_64-efi/module2.mod' not found.

	Press any key to continue...

	Xen 4.14.2
	(XEN) [00000039f16a432c] Xen version 4.14.2 (mockbuild@[unknown]) (gcc (GCC) 11.1.1 20210428 (Red Hat 11.1.1-1)) debug=n  T1
	(XEN) [00000039f8bd398c] Latest ChangeSet: 
	(XEN) [00000039fae56c74] build-id: 2712ab0b14a8d290c557dc982a09b19b1da8deea
	(XEN) [00000039fe9d8e28] Bootloader: GRUB 2.06~rc1
	(XEN) [0000003a011ba3f4] Command line: placeholder dom0=pvh dom0-iommu=map-reserved dom0_max_vcpus=4 dom0_mem=4016M,max:409f...
	(XEN) [0000003a12b673d0] Xen image load base address: 0x9fc00000
	(XEN) [0000003a15e539b4] Video information:
	(XEN) [0000003a180d550c]  VGA is graphics mode 800x600, 32 bpp
	(XEN) [0000003a1b24cc34] Disc information:
	(XEN) [0000003a1d409c5c]  Found 0 MBR signatures
	(XEN) [0000003a1fad8c34]  Found 6 EDD information structures
	(XEN) [0000003a22a64f54] CPU Vendor: Intel, Family 6 (0x6), Model 60 (0x3c), Stepping 3 (raw 000306c3)
	(XEN) [0000003a27a872a0] EFI RAM map:
	(XEN) [0000003a2986fe44]  [0000000000000000, 0000000000057fff] (usable)
	...


with

	rpm -qa | grep grub
		grub2-common-2.06~rc1-4.fc34.noarch
		grub2-pc-modules-2.06~rc1-4.fc34.noarch
		grub2-efi-x64-modules-2.06~rc1-4.fc34.noarch
		grubby-8.40-51.fc34.x86_64
		grub2-tools-minimal-2.06~rc1-4.fc34.x86_64
		grub2-tools-2.06~rc1-4.fc34.x86_64
		grub2-tools-extra-2.06~rc1-4.fc34.x86_64
		grub2-efi-x64-2.06~rc1-4.fc34.x86_64
		grub2-pc-2.06~rc1-4.fc34.x86_64
		grub2-tools-efi-2.06~rc1-4.fc34.x86_64
		grub2-efi-aa64-modules-2.06~rc1-4.fc34.noarch

Comment 49 Javier Martinez Canillas 2021-06-10 21:49:56 UTC
(In reply to pgnet.dev from comment #48)
> still here, on F34:
> 
> 	Loading Xen 4.14.2 ...
> 	error: ../../grub-core/fs/fshelp.c:257:file
> `/EFI/fedora/x86_64-efi/module2.mod' not found.
> 	Loading Linux 5.12.9-300.fc34.x86_64 ...
> 	Loading initial ramdisk ...
> 	error: ../../grub-core/fs/fshelp.c:257:file
> `/EFI/fedora/x86_64-efi/module2.mod' not found.

Did you copy the modules to the /boot/efi/EFI/fedora/x86_64-efi/ directory ?

Comment 50 pgnet.dev 2021-06-10 22:40:14 UTC
for 'multiboot2.mod', yes

	sha1sum \
	/usr/lib/grub/x86_64-efi/multiboot2.mod \
	/boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod

		800657d610c7ceef915a7fcaa64ecf886a786aed  /usr/lib/grub/x86_64-efi/multiboot2.mod
		800657d610c7ceef915a7fcaa64ecf886a786aed  /boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod


for 'module2.mod', there _is_ no such module to copy

	updatedb
	locate module2.mod
		(empty)

	find /usr/lib/grub /boot/efi -iname '*module2*'
		(empty)

Comment 51 Javier Martinez Canillas 2021-06-10 22:44:44 UTC
(In reply to pgnet.dev from comment #50)
> for 'multiboot2.mod', yes
> 
> 	sha1sum \
> 	/usr/lib/grub/x86_64-efi/multiboot2.mod \
> 	/boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod
> 
> 		800657d610c7ceef915a7fcaa64ecf886a786aed 
> /usr/lib/grub/x86_64-efi/multiboot2.mod
> 		800657d610c7ceef915a7fcaa64ecf886a786aed 
> /boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod
> 
> 
> for 'module2.mod', there _is_ no such module to copy
> 
> 	updatedb
> 	locate module2.mod
> 		(empty)
> 
> 	find /usr/lib/grub /boot/efi -iname '*module2*'
> 		(empty)

I'm confused because I remember we already fixed that...

Did you re-generate the GRUB config file just to be sure?

Also, since F34 the GRUB config file is in /boot/grub2/grub.cfg and the
one in /boot/efi/EFI/fedora/grub.cfg is just a stub that loads the former.

Hmm, I think we now need to have the modules in /boot/grub2/x86_64-efi/ instead.

Comment 52 pgnet.dev 2021-06-10 23:08:33 UTC
Did you re-generate the GRUB config file just to be sure?

yes.

per prior comments here, or elsewhere I can't remember,

	Fedora's BLSCFG does not! yet fully implement BLSCFG spec, which no longer requires grub.cfg
	grub.cfg is still required, and provides a fallback to (missing) snippets

exec

	rm -f /boot/efi/EFI/fedora/grub.cfg
	grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
	ls -al \
	 /boot/grub2/grub.cfg \
	 /boot/efi/EFI/fedora/grub.cfg
		-rwx------ 1 root root 63K Jun 10 18:49 /boot/efi/EFI/fedora/grub.cfg*
		-rwx------ 1 root root 63K Jun  8 11:24 /boot/grub2/grub.cfg*

> Also, since F34 the GRUB config file is in /boot/grub2/grub.cfg and the
> one in /boot/efi/EFI/fedora/grub.cfg is just a stub that loads the former.

	sha1sum \
	/boot/grub2/grub.cfg \
	/boot/efi/EFI/fedora/grub.cfg

		ca03dc72d817b523582c2a819fb6aa92d320be1f  /boot/grub2/grub.cfg
		ca03dc72d817b523582c2a819fb6aa92d320be1f  /boot/efi/EFI/fedora/grub.cfg

It'd identical, not just a stub.

Note,

	rpm -q --whatprovides /boot/grub2/grub.cfg
		grub2-efi-x64-2.06~rc1-4.fc34.x86_64
		grub2-pc-2.06~rc1-4.fc34.x86_64

Which appears a problem.  But,

	dnf remove grub2-pc-2.06~rc1-4.fc34.x86_64
		Error:
		Problem: The operation would result in removing the following protected packages: grub2-pc
		(try to add '--skip-broken' to skip uninstallable packages)

for completeness,

	ls -al /boot/loader/entries/
		total 20K
		drwx------. 2 root root 4.0K Jun  8 11:24 ./
		drwxr-xr-x. 4 root root 4.0K Jun 10 18:49 ../
		-rw-r--r--  1 root root 1.2K May 28 08:04 22c80b24af68489987a1ad325a4e0fb3-5.12.7-300.fc34.x86_64.conf
		-rw-r--r--  1 root root 1.2K Jun  1 18:53 22c80b24af68489987a1ad325a4e0fb3-5.12.8-300.fc34.x86_64.conf
		-rw-r--r--  1 root root 1.2K Jun  8 11:24 22c80b24af68489987a1ad325a4e0fb3-5.12.9-300.fc34.x86_64.conf

	cat /boot/loader/entries/*5.12.9*conf
		title Fedora (5.12.9-300.fc34.x86_64) 34 (Thirty Four)
		version 5.12.9-300.fc34.x86_64
		linux /vmlinuz-5.12.9-300.fc34.x86_64
		initrd /initramfs-5.12.9-300.fc34.x86_64.img
		options placeholder root=/dev/mapper/VG0-ROOT ro ... earlyprintk=xen
		grub_users $grub_users
		grub_arg --unrestricted
		grub_class kernel

and,

	grep -rlni module2 /boot/efi/EFI/fedora
		/boot/efi/EFI/fedora/grub.cfg
		/boot/efi/EFI/fedora/x86_64-efi/command.lst
		/boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod
		/boot/efi/EFI/fedora/x86_64-efi/nativedisk.mod

	strings /boot/efi/EFI/fedora/x86_64-efi/multiboot2.mod | grep module2 -B5 -A5
		--nounzip
		you need to load the kernel first
		Load a multiboot 2 kernel.
		multiboot2
		Load a multiboot 2 module.
???		module2
		premature end of file %s
		../../grub-core/loader/multiboot_mbi2.c
		no multiboot header found
		unsupported information tag: 0x%x
		unsupported tag: 0x%x

Comment 53 Javier Martinez Canillas 2021-06-10 23:17:56 UTC
Can you share your full /boot/efi/EFI/fedora/grub.cfg ? By looking at the latest version
it shouldn't attempt to load a "module2" but just "multiboot2", which should be correct.

Comment 54 pgnet.dev 2021-06-10 23:34:14 UTC
since BZ atm *refuses* to allow me to upload an attachment,

-->		https://pastebin.com/D6z68AZM


also, note

	grep module2 /etc/grub.d/20_linux_xen -A10 -B10
	echo "        submenu '$(gettext_printf "Xen hypervisor, version %s" "${xen_version}" | grub_quote)' \$menuentry_id_option 'xen-hypervisor-$xen_version-$boot_device_id' {"
		fi
		if ($grub_file --is-arm64-efi $current_xen); then
	xen_module="xen_boot"
	xen_loader="xen_hypervisor"
	module_loader="xen_module"
		else
	if ($grub_file --is-x86-multiboot2 $current_xen); then
		xen_module="multiboot2"
		xen_loader="multiboot2"
???		module_loader="module2"
	else
		xen_module="multiboot"
		xen_loader="multiboot"
		module_loader="module"
			fi
		fi

		initrd_early=
		for i in ${GRUB_EARLY_INITRD_LINUX_STOCK} \
				${GRUB_EARLY_INITRD_LINUX_CUSTOM}; do

Comment 55 Javier Martinez Canillas 2021-06-10 23:58:24 UTC
You have a /etc/grub.d/20_linux_xen.ORIG that contains the old code before the fix.

Please remove that file and re-generate your grub config file.

Comment 56 pgnet.dev 2021-06-11 00:28:21 UTC
BOTH 20_linux_xen & 20_linux_xen.ORIG contain refs to 'module2'

cleaning up

	cd /etc/grub.d
	mv 20_linux_xen* ../
	dnf -y reinstall grub2-tools
	grep module2 * -A5 -B5
		20_linux_xen-   module_loader="xen_module"
		20_linux_xen-    else
		20_linux_xen-   if ($grub_file --is-x86-multiboot2 $current_xen); then
		20_linux_xen-       xen_module="multiboot2"
		20_linux_xen-       xen_loader="multiboot2"
		20_linux_xen:       module_loader="module2"
		20_linux_xen-   else
		20_linux_xen-       xen_module="multiboot"
		20_linux_xen-       xen_loader="multiboot"
		20_linux_xen-       module_loader="module"
		20_linux_xen-        fi

re-creating grub.cfg,

now,

	ls -al \
	 /boot/grub2/grub.cfg \
	 /boot/efi/EFI/fedora/grub.cfg
		-rwx------ 1 root root 31K Jun 10 20:05 /boot/efi/EFI/fedora/grub.cfg*
		-rwx------ 1 root root 63K Jun  8 11:24 /boot/grub2/grub.cfg*

	sha1sum \
	 /boot/grub2/grub.cfg \
	 /boot/efi/EFI/fedora/grub.cfg
		ca03dc72d817b523582c2a819fb6aa92d320be1f  /boot/grub2/grub.cfg
		9fb64deddee92061c03d4347f1f74f0fb0a35066  /boot/efi/EFI/fedora/grub.cfg


and

	cat /boot/efi/EFI/fedora/grub.cfg

--> https://pastebin.com/9e8FcsBP

still with plenty of 'module2' references.

now, on reboot,

I see a different grub menu

      Fedora (5.12.9-300.fc34.x86_64) 34 (Thirty Four)  <<<< PRE-SELECTED
      Fedora (5.12.8-300.fc34.x86_64) 34 (Thirty Four)
      Fedora (5.12.7-300.fc34.x86_64) 34 (Thirty Four)
      Fedora, with Xen 4.14.2 and Linux 5.12.9-300.fc34.x86_64
      Fedora, with Xen 4.14.2 and Linux 5.12.8-300.fc34.x86_64
      Fedora, with Xen 4.14.2 and Linux 5.12.7-300.fc34.x86_64
      Fedora, with Xen 4.14 and Linux 5.12.9-300.fc34.x86_64
      Fedora, with Xen 4.14 and Linux 5.12.8-300.fc34.x86_64
      Fedora, with Xen 4.14 and Linux 5.12.7-300.fc34.x86_64
      Fedora, with Xen xen and Linux 5.12.9-300.fc34.x86_64
      Fedora, with Xen xen and Linux 5.12.8-300.fc34.x86_64
      Fedora, with Xen xen and Linux 5.12.7-300.fc34.x86_64
      UEFI Firmware Settings

I re-select

	  ...
      Fedora, with Xen 4.14 and Linux 5.12.7-300.fc34.x86_64
      Fedora, with Xen xen and Linux 5.12.9-300.fc34.x86_64   <<<<
      Fedora, with Xen xen and Linux 5.12.8-300.fc34.x86_64
      Fedora, with Xen xen and Linux 5.12.7-300.fc34.x86_64
	  ...

and now,

	Loading Xen xen ...
	Loading Linux 5.12.9-300.fc34.x86_64 ...
	Loading initial ramdisk ...
	Loading XSM policy ...
	error: ../../grub-core/fs/fshelp.c:257:file `/xenpolicy-4.14' not found.

	Press any key to continue...

	(~ 45 second delay) 

	0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0xa148f018
	0x0000:0x03:0x00.0x0: ROM: 0x8000 bytes at 0xa147f018
	0x0000:0x0f:0x00.0x0: ROM: 0x10800 bytes at 0xa145e018
	Xen 4.14.2
	(XEN) [0000007e354306be] Xen version 4.14.2 (mockbuild@[unknown]) (gcc (GCC) 11.1.1 20210428 (Red Hat 11.1.1-1)) debug=n  T1 ...

	Fedora 34 (Thirty Four)
	Kernel 5.12.9-300.fc34.x86_64 on an x86_64 (hvc0)

	xendev login:

checking

	locate xenpolicy-4.14
		/boot/flask/xenpolicy-4.14.2

1st I recall dealing with _that_ file.

rebooting AGAIN,

      Fedora (5.12.9-300.fc34.x86_64) 34 (Thirty Four)  <<<< PRE-SELECTED

so grub boot selection is no longer sticking as it has ...

Comment 57 Javier Martinez Canillas 2021-06-11 00:37:03 UTC
(In reply to pgnet.dev from comment #56)

[snip]

> 
> --> https://pastebin.com/9e8FcsBP
> 
> still with plenty of 'module2' references.
>

those are correct, how else would you load multiboot2 modules without that command?

but there are no more "insmod module2" commands, which was the bug.
 
I think you are referring to different bugs, please file separate reports for those.

Comment 58 Cameron 2021-11-16 21:24:00 UTC
Just fyi, I'm also hitting this on the latest Fedora Workstation 35. In my case, I was also missing the files from `/boot/grub2/x86_64-efi/` (Not `/boot/efi/EFI/fedora/x86_64-efi/`).

With a fresh install, I updated, then installed with:
    sudo yum groupinstall 'Virtualization'
    sudo yum install xen

I fixed this with copying the entire x86_64-efi dir over. It didn't exist before:
   sudo cp -af /usr/lib/grub/x86_64-efi /boot/grub2

Now I'm hitting a "/xenpolicy-4.15 not found" error, but that's out of scope for this ticket, and I haven't looked into it yet. I'm not sure if that first half is worth opening this ticket, or if this is expected.

Comment 59 Cameron 2021-11-17 07:02:40 UTC
Fixed it by disabling SELinux. Is there a way to get pass this without disabling that?