Bug 1615969

Summary: AArch64 - Error when booting "error: out of memory."
Product: [Fedora] Fedora Reporter: Paul Whalen <pwhalen>
Component: grub2Assignee: Peter Jones <pjones>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 29CC: awilliam, fzatlouk, gmarr, kevin, lkundrak, marc.c.dionne, pbrobinson, pjones, robatino
Target Milestone: ---   
Target Release: ---   
Hardware: aarch64   
OS: Linux   
Whiteboard: AcceptedBlocker
Fixed In Version: grub2-2.02-52.fc29 grub2-2.02-57.fc29 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 02:54:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 245418, 1517011    
Attachments:
Description Flags
boot msgs none

Description Paul Whalen 2018-08-14 15:52:02 UTC
Description of problem:
When attempting to boot with grub2-2.02-47.fc29 on aarch64, the boot displays an error "Error: Out of memory.", then a kernel panic. 

Version-Release number of selected component (if applicable):
grub2-2.02-47.fc29

How reproducible:
Every time.

Additional info:

error: out of memory.

Press any key to continue...
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd072]
[    0.000000] Linux version 4.18.0-1.fc29.aarch64 (mockbuild.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Mon Aug 13 15:54:13 UTC 2018
[    0.000000] Machine model: AMD Seattle (Rev.B0) Development Board (Overdrive)
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.40 by American Megatrends
[    0.000000] efi:  ACPI 2.0=0x83ff1c3000  SMBIOS 3.0=0x83ff347798 
[    0.000000] cma: Reserved 64 MiB at 0x00000080fc000000

..

[    2.234658] md: autorun ...
[    2.237448] md: ... autorun DONE.
[    2.240804] VFS: Cannot open root device "mapper/fedora_seattle-root" or unknown-block(0,0): error -6
[    2.250016] Please append a correct "root=" boot option; here are the available partitions:
[    2.258360] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.266612] CPU: 1 PID: 1 Comm: swapper/0 Tainted: G        W         4.18.0-1.fc29.aarch64 #1
[    2.275210] Hardware name: AMD Overdrive/Supercharger/Default string, BIOS ROD1002C 04/08/2016
[    2.283807] Call trace:
[    2.286245]  dump_backtrace+0x0/0x150
[    2.289895]  show_stack+0x24/0x30
[    2.293200]  dump_stack+0x90/0xb4
[    2.296503]  panic+0x114/0x278
[    2.299548]  mount_block_root+0x228/0x2c8
[    2.303545]  mount_root+0x84/0x90
[    2.306847]  prepare_namespace+0x134/0x178
[    2.310930]  kernel_init_freeable+0x318/0x33c
[    2.315275]  kernel_init+0x18/0x114
[    2.318752]  ret_from_fork+0x10/0x18
[    2.322318] SMP: stopping secondary CPUs
[    2.326233] Kernel Offset: disabled
[    2.329709] CPU features: 0x21806082
[    2.333271] Memory Limit: none
[    2.336316] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---

Comment 1 Paul Whalen 2018-08-14 15:55:24 UTC
Created attachment 1475897 [details]
boot msgs

Comment 2 Paul Whalen 2018-08-20 20:07:02 UTC
Testing with grub2-2.02-50.fc29:

loader/arm64/linux.c:62: UEFI stub kernel:
loader/arm64/linux.c:63: PE/COFF header @ 00000040
loader/arm64/linux.c:351: kernel file size: 24844800
loader/arm64/linux.c:354: kernel numpages: 6066
loader/arm64/linux.c:370: kernel @ 0x83fb020000
loader/efi/linux.c:44: shim_lock: 0x83fce316c8
loader/efi/linux.c:51: Asking shim to verify kernel signature
loader/efi/linux.c:53: shim_lock->verify(): 0
loader/efi/linux.c:56: Kernel signature verification passed
loader/arm64/linux.c:293: Loading initrd
error: out of memory.

Press any key to continue...
kern/efi/fdt.c:38: found registered FDT @ 0x801fffc000
loader/efi/fdt.c:64: allocating 13575 bytes for fdt
loader/arm64/linux.c:144: Installed/updated FDT configuration table @
0x83fb01c000
loader/arm64/linux.c:200: linux command line:
'BOOT_IMAGE=(tftp)/fedora/Fedora-29-20180817.n.0/Everything/vmlinuz
inst.repo=https://kojipkgs.fedoraproject.org/compose/branched//Fedora-29-201808
17.n.0/compose/Everything/aarch64/os/'
loader/efi/linux.c:83: kernel_addr: 0x83fb020000 handover_offset: 0x1036004
params: 0x83fb020000
loader/efi/linux.c:85: handover_func() = 0x83fc056004
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, no randomness supplied
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd072]
[    0.000000] Linux version 4.18.1-300.fc29.aarch64 (mockbuild.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Thu Aug 16 16:53:57 UTC 2018

Comment 3 Marc Dionne 2018-08-28 18:09:11 UTC
Any more insights on this problem?

I'm seeing roughly the same thing when trying to boot an OverDrive 1000:

error: out of memory.

Press any key to continue...
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd072]
[    0.000000] Linux version 4.18.5-300.fc29.aarch64 (mockbuild.fedoraproject.org) (gcc version 8.2.1 20180801 (Red Hat 8.2.1-2) (GCC)) #1 SMP Fri Aug 24 17:18:34 UTC 2018
[    0.000000] Machine model: SoftIron Overdrive 1000 (AMD Seattle (Rev.B1))
[    0.000000] efi: Getting EFI parameters from FDT:
[    0.000000] efi: EFI v2.60 by SoftIron Overdrive 1000

.....

[    1.924078] VFS: Cannot open root device "UUID=de35480d-3e54-4296-b171-77876ce9c976" or unknown-block(0,0): error -6
[    1.934599] Please append a correct "root=" boot option; here are the available partitions:
[    1.942949] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Comment 4 Paul Whalen 2018-08-28 20:16:22 UTC
Proposing as a blocker for F29 Beta, this breaks aarch64 installs.

Comment 5 František Zatloukal 2018-08-29 14:26:51 UTC
(In reply to Paul Whalen from comment #4)
> Proposing as a blocker for F29 Beta, this breaks aarch64 installs.

Paul, if I didn't miss anything, aarch64 is not a primary architecture and I guess this won't be accepted as a blocker because of that. 

https://fedoraproject.org/wiki/Architectures

Comment 6 Peter Robinson 2018-08-29 14:29:17 UTC
> Paul, if I didn't miss anything, aarch64 is not a primary architecture and I
> guess this won't be accepted as a blocker because of that. 

ARM64 for Server is primary architecture and hence blocking as of F-28

Comment 7 Kevin Fenzi 2018-08-29 23:09:12 UTC
+1 blocker.

Comment 8 Geoffrey Marr 2018-08-30 14:44:18 UTC
+1 blocker

Comment 9 Geoffrey Marr 2018-08-30 14:45:10 UTC
Voted as a blocker in-bug:

The decision to classify this bug as an "AcceptedBlocker" (Beta) was made as it violates the following criteria:

"The installer must run when launched normally from the release-blocking images."

Comment 10 Peter Robinson 2018-08-30 16:18:56 UTC
For reference grub2-2.02-52.fc29 fixes this for me. Will be on it's way to f29 updates-testing shortly

Comment 11 Paul Whalen 2018-08-30 17:05:19 UTC
Verified fixed in grub2-2.02-52.fc29

Comment 12 Fedora Update System 2018-08-30 17:06:38 UTC
grub2-2.02-52.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 13 Marc Dionne 2018-08-30 19:09:17 UTC
FYI grub2-2.02-52.fc29 also fixes the boot problem for my aarch64 box.

Comment 14 Fedora Update System 2018-08-31 16:22:27 UTC
grub2-2.02-52.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 15 Fedora Update System 2018-09-01 01:49:40 UTC
grub2-2.02-52.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 16 Fedora Update System 2018-09-02 02:56:45 UTC
grub2-2.02-52.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 17 František Zatloukal 2018-09-06 09:47:21 UTC
Switching to VERIFIED as per Comment 11 .

Comment 18 Peter Robinson 2018-09-06 09:56:56 UTC
(In reply to František Zatloukal from comment #17)
> Switching to VERIFIED as per Comment 11 .

The update has been dropped due to issues with x86

Comment 19 Fedora Update System 2018-09-08 04:07:28 UTC
grub2-2.02-54.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 20 František Zatloukal 2018-09-08 11:16:35 UTC
grub2-2.02-54.fc29 worked fine on x86_64 UEFI (VM).

Can't test aarch64 though...

Comment 21 Fedora Update System 2018-09-08 16:11:21 UTC
grub2-2.02-54.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 22 Adam Williamson 2018-09-10 19:37:08 UTC
I did a couple of scratch builds of different possibilities to fix this while not regressing x86_64 UEFI. If folks with affected aarch64 systems could try them, it'd be great. Here they are:

* https://koji.fedoraproject.org/koji/taskinfo?taskID=29603292
* https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437

The first contains all the changes from -52, plus amartinez's fix for the x86_64 UEFI boot issue instead of Hans' that we shipped in -54.

The second contains only the bits from -51, plus the change that was intended to fix *this* bug (and one other that is a precursor for it). Other changes were dropped.

It'd be useful to know how both of these work compared to -51 and to -54. Thanks!

Comment 23 Marc Dionne 2018-09-10 20:14:08 UTC
(In reply to Adam Williamson from comment #22)
> I did a couple of scratch builds of different possibilities to fix this
> while not regressing x86_64 UEFI. If folks with affected aarch64 systems
> could try them, it'd be great. Here they are:
> 
> * https://koji.fedoraproject.org/koji/taskinfo?taskID=29603292
> * https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437
> 
> The first contains all the changes from -52, plus amartinez's fix for the
> x86_64 UEFI boot issue instead of Hans' that we shipped in -54.
> 
> The second contains only the bits from -51, plus the change that was
> intended to fix *this* bug (and one other that is a precursor for it). Other
> changes were dropped.
> 
> It'd be useful to know how both of these work compared to -51 and to -54.
> Thanks!

Both of those boot fine for me on my aarch64 box.  For reference, -51 did not boot, -54 was ok.

Comment 24 Adam Williamson 2018-09-11 01:45:49 UTC
Thanks a lot. Can you also test pjones' -55 build?

https://koji.fedoraproject.org/koji/buildinfo?buildID=1143536

Thanks again!

Comment 25 Paul Whalen 2018-09-11 02:58:37 UTC
grub2-2.02-55.fc29 working on mustang, seattle

Comment 26 Marc Dionne 2018-09-11 11:35:06 UTC
2.02-55.fc29.aarch64 looks fine here

Comment 27 Adam Williamson 2018-09-11 15:04:17 UTC
pwhalen: does the 'minimal' build - https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437 - work OK on your aarch64 systems? Thanks!

Comment 28 Marc Dionne 2018-09-11 16:18:27 UTC
(In reply to Adam Williamson from comment #27)
> pwhalen: does the 'minimal' build -
> https://koji.fedoraproject.org/koji/taskinfo?taskID=29603437 - work OK on
> your aarch64 systems? Thanks!

It does here, as I reported in comment #23

Comment 29 Adam Williamson 2018-09-11 17:46:28 UTC
Marc: I know *you* reported success, I was looking for results from pwhalen as well :) Multiple tests is always good for safety.

Comment 30 Marc Dionne 2018-09-11 17:53:38 UTC
Adam, sorry, I interpreted the label incorrectly, thought you meant pwhalen was asking the question...  :)

Comment 31 Adam Williamson 2018-09-11 17:57:18 UTC
No problem :)

Sorry to be a pain, can you both also please test -56:

https://koji.fedoraproject.org/koji/taskinfo?taskID=29618610

and check that still works? Thanks!

Comment 32 Marc Dionne 2018-09-11 18:12:32 UTC
-56 works fine for me

Comment 33 Paul Whalen 2018-09-11 19:59:44 UTC
adamw : 2.02-54.2.minimal working on the seattle 

2.02-56.fc29 working on seattle and rpi3

Comment 34 Adam Williamson 2018-09-11 20:09:33 UTC
Thanks a lot guys!

Comment 35 Fedora Update System 2018-09-11 22:57:18 UTC
grub2-2.02-54.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2018-2756e3a374

Comment 36 Fedora Update System 2018-09-12 02:54:03 UTC
grub2-2.02-57.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report.