Fedora-Workstation-Live-x86_64-Rawhide-20230830.n.0.iso is failing to boot on some hardware. Reproducible: Always Steps to Reproduce: 1. Download ISO, write to USB key 2. Boot USB 3. Select entry in GRUB menu and attempt boot Actual Results: Systems hangs Expected Results: System boots Currently tested: * Qemu 7.x UEFI/BIOS works * MBP 12,1 UEFI works * MBP 14,1 UEFI doesn't
Created attachment 1986200 [details] Hardware profile for failed UEFI boot
Created attachment 1986201 [details] dmidecode for failed UEFI boot
Do you get anything more useful if you remove 'rhgb' and 'quiet' from the boot options?
uname -a Linux 2603-8001-a700-1c59-0000-0000-0000-17d5.res6.spectrum.com 6.5.0-0.rc7.20230821gitf7757129e3de.50.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 21 16:22:22 UTC 2023 x86_64 GNU/Linux lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 36 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: GenuineIntel Model name: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz CPU family: 6 Model: 42 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 Stepping: 7 CPU(s) scaling MHz: 58% CPU max MHz: 3400.0000 CPU min MHz: 1600.0000 BogoMIPS: 6185.92 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmper f pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_dead line_timer aes xsave avx lahf_lm epb pti xsaveopt dtherm ida arat pln pts Caches (sum of all): L1d: 128 KiB (4 instances) L1i: 128 KiB (4 instances) L2: 1 MiB (4 instances) L3: 6 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-3 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: KVM: Mitigation: VMX unsupported L1tf: Mitigation; PTE Inversion Mds: Vulnerable: Clear CPU buffers attempted, no microcode; S MT disabled Meltdown: Mitigation; PTI Mmio stale data: Unknown: No mitigations Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Vulnerable Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines, STIBP disabled, RSB filling, PBR SB-eIBRS Not affected Srbds: Not affected Tsx async abort: Not affected dmidecode # dmidecode 3.4 Getting SMBIOS data from sysfs. SMBIOS 2.7 present. 62 structures occupying 2110 bytes. Table at 0x000EB630. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: AMI Version: 8.12 Release Date: 09/04/2013 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 3 MB Characteristics: PCI is supported PNP is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported EDD is supported Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Function key-initiated network boot is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 4.6 Handle 0x0001, DMI type 1, 27 bytes System Information Manufacturer: Hewlett-Packard Product Name: HP Compaq Pro 4300 SFF PC Version: Serial Number: JPA4070V98 UUID: 8f45bb01-c85d-b363-b70b-cc9e311d7b21 Wake-up Type: Power Switch SKU Number: QZ219AV Family: 103C_53307F G=D Handle 0x0002, DMI type 2, 15 bytes Base Board Information Manufacturer: Hewlett-Packard Product Name: 2ADE Version: 1.0 Serial Number: JPA4070V98 Asset Tag: JPA4070V98 Features: Board is a hosting board Board is removable Board is replaceable Location In Chassis: Chassis Handle: 0x0003 Type: Motherboard Contained Object Handles: 0 Handle 0x0003, DMI type 3, 22 bytes Chassis Information Manufacturer: Hewlett-Packard Type: Desktop Lock: Not Present Version: Serial Number: JPA4070V98 Asset Tag: JPA4070V98 Boot-up State: Safe Power Supply State: Safe Thermal State: Safe Security Status: None OEM Information: 0x00000000 Height: Unspecified Number Of Power Cords: 1 Contained Elements: 0 SKU Number: Handle 0x0004, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1A1 Internal Connector Type: None External Reference Designator: PS2Mouse External Connector Type: PS/2 Port Type: Mouse Port Handle 0x0005, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J1A1 Internal Connector Type: None External Reference Designator: Keyboard External Connector Type: PS/2 Port Type: Keyboard Port Handle 0x0006, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: LAN External Connector Type: RJ-45 Port Type: Network Port Handle 0x0007, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J2A2A Internal Connector Type: None External Reference Designator: COM A External Connector Type: DB-9 male Port Type: Serial Port 16550A Compatible Handle 0x0008, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J2A2A Internal Connector Type: None External Reference Designator: LPT External Connector Type: DB-9 male Port Type: Serial Port 16550A Compatible Handle 0x0009, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J2A2B Internal Connector Type: None External Reference Designator: Video External Connector Type: DB-15 female Port Type: Video Port Handle 0x000A, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J3A1 Internal Connector Type: None External Reference Designator: USB1 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000B, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J3A1 Internal Connector Type: None External Reference Designator: USB2 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000C, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J3A1 Internal Connector Type: None External Reference Designator: USB3 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000D, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB4 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000E, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB5 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x000F, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB6 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x0010, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB7 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x0011, DMI type 8, 9 bytes Port Connector Information Internal Reference Designator: J5A1 Internal Connector Type: None External Reference Designator: USB8 External Connector Type: Access Bus (USB) Port Type: USB Handle 0x0012, DMI type 9, 17 bytes System Slot Information Designation: PCIE x16 Type: x16 PCI Express Current Usage: In Use Length: Long ID: 0 Characteristics: 3.3 V is provided Opening is shared PME signal is supported Bus Address: 0000:01:00.0 Handle 0x0013, DMI type 9, 17 bytes System Slot Information Designation: PCIE x1 Type: x1 PCI Express Current Usage: Available Length: Short ID: 1 Characteristics: 3.3 V is provided Opening is shared PME signal is supported Bus Address: 0000:02:00.0 Handle 0x0014, DMI type 9, 17 bytes System Slot Information Designation: PCI1 Type: 32-bit PCI Current Usage: Available Length: Short ID: 2 Characteristics: 3.3 V is provided Opening is shared PME signal is supported Bus Address: 0003:04:00.0 Handle 0x0015, DMI type 9, 17 bytes System Slot Information Designation: PCI2 Type: 32-bit PCI Current Usage: Available Length: Short ID: 3 Characteristics: 3.3 V is provided Opening is shared PME signal is supported Bus Address: 0003:04:00.4 Handle 0x0016, DMI type 11, 5 bytes OEM Strings String 1: FBYTE#2U3E3P3V3X47676S6a727B7H7M7Q7T7Wagapaq.Ff;BUILDID#12WWPIIW String 2: 304#SABJ#DABJ; String 3: String 4: String 5: String 6: String 7: String 8: String 9: String 10: String 11: String 12: String 13: String 14: String 15: String 16: String 17: String 18: String 19: String 20: String 21: String 22: String 23: String 24: String 25: String 26: String 27: String 28: String 29: String 30: String 31: String 32: Handle 0x0017, DMI type 12, 5 bytes System Configuration Options Option 1: Handle 0x0018, DMI type 32, 20 bytes System Boot Information Status: No errors detected Handle 0x0019, DMI type 34, 11 bytes Management Device Description: LM78-1 Type: LM78 Address: 0x00000000 Address Type: I/O Port Handle 0x001A, DMI type 26, 22 bytes Voltage Probe Description: LM78A Location: <OUT OF SPEC> Status: <OUT OF SPEC> Maximum Value: Unknown Minimum Value: Unknown Resolution: Unknown Tolerance: Unknown Accuracy: Unknown OEM-specific Information: 0x00000000 Nominal Value: Unknown Handle 0x001B, DMI type 36, 16 bytes Management Device Threshold Data Lower Non-critical Threshold: 1 Upper Non-critical Threshold: 2 Lower Critical Threshold: 3 Upper Critical Threshold: 4 Lower Non-recoverable Threshold: 5 Upper Non-recoverable Threshold: 6 Handle 0x001C, DMI type 35, 11 bytes Management Device Component Description: Management Device Handle: 0x0019 Component Handle: 0x0019 Threshold Handle: 0x001A Handle 0x001D, DMI type 28, 22 bytes Temperature Probe Description: LM78A Location: <OUT OF SPEC> Status: <OUT OF SPEC> Maximum Value: Unknown Minimum Value: Unknown Resolution: Unknown Tolerance: Unknown Accuracy: Unknown OEM-specific Information: 0x00000000 Nominal Value: Unknown Handle 0x001E, DMI type 36, 16 bytes Management Device Threshold Data Lower Non-critical Threshold: 1 Upper Non-critical Threshold: 2 Lower Critical Threshold: 3 Upper Critical Threshold: 4 Lower Non-recoverable Threshold: 5 Upper Non-recoverable Threshold: 6 Handle 0x001F, DMI type 35, 11 bytes Management Device Component Description: Management Device Handle: 0x0019 Component Handle: 0x001C Threshold Handle: 0x001D Handle 0x0020, DMI type 27, 15 bytes Cooling Device Temperature Probe Handle: 0x001D Type: <OUT OF SPEC> Status: <OUT OF SPEC> Cooling Unit Group: 1 OEM-specific Information: 0x00000000 Nominal Speed: Unknown Or Non-rotating Description: Cooling Dev 1 Handle 0x0021, DMI type 36, 16 bytes Management Device Threshold Data Lower Non-critical Threshold: 1 Upper Non-critical Threshold: 2 Lower Critical Threshold: 3 Upper Critical Threshold: 4 Lower Non-recoverable Threshold: 5 Upper Non-recoverable Threshold: 6 Handle 0x0022, DMI type 35, 11 bytes Management Device Component Description: Management Device Handle: 0x0019 Component Handle: 0x001F Threshold Handle: 0x0020 Handle 0x0023, DMI type 27, 15 bytes Cooling Device Temperature Probe Handle: 0x001D Type: <OUT OF SPEC> Status: <OUT OF SPEC> Cooling Unit Group: 1 OEM-specific Information: 0x00000000 Nominal Speed: Unknown Or Non-rotating Description: Not Specified Handle 0x0024, DMI type 36, 16 bytes Management Device Threshold Data Lower Non-critical Threshold: 1 Upper Non-critical Threshold: 2 Lower Critical Threshold: 3 Upper Critical Threshold: 4 Lower Non-recoverable Threshold: 5 Upper Non-recoverable Threshold: 6 Handle 0x0025, DMI type 35, 11 bytes Management Device Component Description: Management Device Handle: 0x0019 Component Handle: 0x0022 Threshold Handle: 0x0023 Handle 0x0026, DMI type 29, 22 bytes Electrical Current Probe Description: ABC Location: <OUT OF SPEC> Status: <OUT OF SPEC> Maximum Value: Unknown Minimum Value: Unknown Resolution: Unknown Tolerance: Unknown Accuracy: Unknown OEM-specific Information: 0x00000000 Nominal Value: Unknown Handle 0x0027, DMI type 36, 16 bytes Management Device Threshold Data Handle 0x0028, DMI type 35, 11 bytes Management Device Component Description: Management Device Handle: 0x0019 Component Handle: 0x0025 Threshold Handle: 0x0023 Handle 0x0029, DMI type 26, 22 bytes Voltage Probe Description: LM78A Location: Power Unit Status: OK Maximum Value: Unknown Minimum Value: Unknown Resolution: Unknown Tolerance: Unknown Accuracy: Unknown OEM-specific Information: 0x00000000 Nominal Value: Unknown Handle 0x002A, DMI type 28, 22 bytes Temperature Probe Description: LM78A Location: Power Unit Status: OK Maximum Value: Unknown Minimum Value: Unknown Resolution: Unknown Tolerance: Unknown Accuracy: Unknown OEM-specific Information: 0x00000000 Nominal Value: Unknown Handle 0x002B, DMI type 27, 15 bytes Cooling Device Temperature Probe Handle: 0x002A Type: Power Supply Fan Status: OK Cooling Unit Group: 1 OEM-specific Information: 0x00000000 Nominal Speed: Unknown Or Non-rotating Description: Cooling Dev 1 Handle 0x002C, DMI type 29, 22 bytes Electrical Current Probe Description: ABC Location: Power Unit Status: OK Maximum Value: Unknown Minimum Value: Unknown Resolution: Unknown Tolerance: Unknown Accuracy: Unknown OEM-specific Information: 0x00000000 Nominal Value: Unknown Handle 0x002D, DMI type 39, 22 bytes System Power Supply Power Unit Group: 1 Location: Name: Manufacturer: Serial Number: Asset Tag: Model Part Number: Standard Efficiency Revision: Max Power Capacity: Unknown Status: Present, OK Type: Switching Input Voltage Range Switching: Auto-switch Plugged: Yes Hot Replaceable: No Input Voltage Probe Handle: 0x0029 Cooling Device Handle: 0x002B Input Current Probe Handle: 0x002C Handle 0x002E, DMI type 41, 11 bytes Onboard Device Reference Designation: Onboard IGD Type: Video Status: Enabled Type Instance: 1 Bus Address: 0000:00:02.0 Handle 0x002F, DMI type 41, 11 bytes Onboard Device Reference Designation: Onboard LAN Type: Ethernet Status: Enabled Type Instance: 1 Bus Address: 0000:04:00.0 Handle 0x0030, DMI type 41, 11 bytes Onboard Device Reference Designation: Onboard AUDIO Type: Sound Status: Enabled Type Instance: 1 Bus Address: 0000:00:1b.0 Handle 0x0031, DMI type 7, 19 bytes Cache Information Socket Designation: CPU Internal L2 Configuration: Enabled, Not Socketed, Level 2 Operational Mode: Write Through Location: Internal Installed Size: 1 MB Maximum Size: 1 MB Supported SRAM Types: Unknown Installed SRAM Type: Unknown Speed: Unknown Error Correction Type: Multi-bit ECC System Type: Unified Associativity: 8-way Set-associative Handle 0x0032, DMI type 7, 19 bytes Cache Information Socket Designation: CPU Internal L1 Configuration: Enabled, Not Socketed, Level 1 Operational Mode: Write Through Location: Internal Installed Size: 256 kB Maximum Size: 256 kB Supported SRAM Types: Unknown Installed SRAM Type: Unknown Speed: Unknown Error Correction Type: Parity System Type: Data Associativity: 8-way Set-associative Handle 0x0033, DMI type 7, 19 bytes Cache Information Socket Designation: CPU Internal L3 Configuration: Enabled, Not Socketed, Level 3 Operational Mode: Write Back Location: Internal Installed Size: 6 MB Maximum Size: 6 MB Supported SRAM Types: Unknown Installed SRAM Type: Unknown Speed: Unknown Error Correction Type: Multi-bit ECC System Type: Unified Associativity: 12-way Set-associative Handle 0x0034, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: None Maximum Capacity: 16 GB Error Information Handle: Not Provided Number Of Devices: 2 Handle 0x0035, DMI type 4, 42 bytes Processor Information Socket Designation: SOCKET 0 Type: Central Processor Family: Core i5 Manufacturer: Intel(R) Corporation ID: A7 06 02 00 FF FB EB BF Signature: Type 0, Family 6, Model 42, Stepping 7 Flags: FPU (Floating-point unit on-chip) VME (Virtual mode extension) DE (Debugging extension) PSE (Page size extension) TSC (Time stamp counter) MSR (Model specific registers) PAE (Physical address extension) MCE (Machine check exception) CX8 (CMPXCHG8 instruction supported) APIC (On-chip APIC hardware supported) SEP (Fast system call) MTRR (Memory type range registers) PGE (Page global enable) MCA (Machine check architecture) CMOV (Conditional move instruction supported) PAT (Page attribute table) PSE-36 (36-bit page size extension) CLFSH (CLFLUSH instruction supported) DS (Debug store) ACPI (ACPI supported) MMX (MMX technology supported) FXSR (FXSAVE and FXSTOR instructions supported) SSE (Streaming SIMD extensions) SSE2 (Streaming SIMD extensions 2) SS (Self-snoop) HTT (Multi-threading) TM (Thermal monitor supported) PBE (Pending break enabled) Version: Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz Voltage: 5.0 V 2.9 V External Clock: 100 MHz Max Speed: 3800 MHz Current Speed: 3100 MHz Status: Populated, Enabled Upgrade: Socket BGA1155 L1 Cache Handle: 0x0032 L2 Cache Handle: 0x0031 L3 Cache Handle: 0x0033 Serial Number: Not Specified Asset Tag: Fill By OEM Part Number: Fill By OEM Core Count: 4 Core Enabled: 4 Thread Count: 4 Characteristics: 64-bit capable Handle 0x0036, DMI type 17, 34 bytes Memory Device Array Handle: 0x0034 Error Information Handle: Not Provided Total Width: Unknown Data Width: Unknown Size: No Module Installed Form Factor: DIMM Set: None Locator: ChannelA-DIMM0 Bank Locator: BANK 0 Type: Unknown Type Detail: None Handle 0x0037, DMI type 17, 34 bytes Memory Device Array Handle: 0x0034 Error Information Handle: Not Provided Total Width: 64 bits Data Width: 64 bits Size: 8 GB Form Factor: DIMM Set: None Locator: ChannelB-DIMM0 Bank Locator: BANK 2 Type: DDR3 Type Detail: Synchronous Speed: 1333 MT/s Manufacturer: Hynix/Hyundai Serial Number: 1529E4F2 Asset Tag: 9876543210 Part Number: HMT41GU6DFR8A-PB Rank: 2 Configured Memory Speed: 1333 MT/s Handle 0x0038, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x001FFFFFFFF Range Size: 8 GB Physical Device Handle: 0x0037 Memory Array Mapped Address Handle: 0x0039 Partition Row Position: Unknown Interleave Position: 1 Interleaved Data Depth: 1 Handle 0x0039, DMI type 19, 31 bytes Memory Array Mapped Address Starting Address: 0x00000000000 Ending Address: 0x001FFFFFFFF Range Size: 8 GB Physical Array Handle: 0x0034 Partition Width: 2 Handle 0x003D, DMI type 208, 5 bytes OEM-specific Type Header and Data: D0 05 3D 00 00 Handle 0x003E, DMI type 13, 22 bytes BIOS Language Information Language Description Format: Abbreviated Installable Languages: 12 enUS frFR esES deDE itIT daDK fiFI nlNL nbNO ptBR svFI jaJP Currently Installed Language: jaJP Handle 0x003F, DMI type 131, 64 bytes OEM-specific Type Header and Data: 83 40 3F 00 14 00 00 00 00 00 00 00 00 00 00 00 F8 00 5C 1C 00 00 00 00 01 00 00 00 01 00 08 00 88 05 28 00 00 00 00 00 C8 00 FF FF 00 00 00 00 00 00 00 00 26 00 00 00 76 50 72 6F 00 00 00 00 Handle 0x0040, DMI type 127, 4 bytes End Of Table Anything else?
Adam, Not that I could get. I did try booting both default entries. The USB drive would go active and then a blank screen. When removing rhgb/quiet, GRUB would note "booting a command list", the USB drive would go active and then hang there.
Thanks...do you happen to know what's the most recent image that *does* boot for you?
Adam, I'm bisecting through now to find the last working. Glad to report that Fedora-Workstation-Live-x86_64-39-20230830.n.0.iso is not affected by this.
Adam, Last working image is Fedora-Workstation-Live-Rawhide-20230829.n.0. I have re-verified Fedora-Workstation-Live-Rawhide-20230830.n.0 isn't working on the affected hardware. From the Fedora-Workstation-Live-Rawhide-20230830.n.0 compose report these updated packages stuck out to investigate: anaconda-39.33-1.fc40 glibc-2.38.9000-6.fc40 kernel-6.6.0-0.rc0.20230829git1c59d383390f.59.fc40 lorax-39.4-2.fc40 It doesn't seem like the kernel comes alive so that is what I'm suspecting right now.
Isn't this a grub issue?
For you? No idea, since we don't have enough information. for Jonathan? Almost certainly not, as grub did not change in the last day. It's almost certainly the kernel for Jonathan. Can you try booting https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230828.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230828.n.0.iso and see if it works? If it does, the problem is almost certainly the kernel for you too. Jonathan, you can also try booting with 'debug' on the off chance that gives us more info (as well as removing 'rhgb' and 'quiet')...
Adam, No change booting with kernel arg 'debug' and removing the other options. However, I did a spin with all updates except the latest kernel and it works as expected. 6.5.0-57.fc40.x86_64 Rawhide guest fully updated, a simple restricted mock config, minor kickstart modifications and the produced live image works. mock (for this case I don't need the kernel in the chroot and the guest kernel matches): include('/etc/mock/fedora-rawhide-x86_64.cfg') config_opts['dnf.conf'] += """ [main] excludepkgs=kernel*6.6.0* """ kickstart (5383203 is the first build root I randomly found that was recent and had the last kernel): repo --name="rawhide" --mirrorlist=https://mirrors.fedoraproject.org/mirrorlist?repo=rawhide&arch=$basearch --excludepkgs=kernel*6.6.0* --cost=1 repo --name="koji-5383203" --baseurl=https://kojipkgs.fedoraproject.org/repos/f40-build/5383203/x86_64/ --cost=10
Current testing: * Fedora-Workstation-Live-x86_64-Rawhide-20230831.n.0.iso w/6.6.0-0.rc0.20230830git6c1b980a7e79.1.fc40.x86_64 FAILS * Fedora-Workstation-Live-x86_64-Rawhide-20230830.n.0.iso w/6.6.0-0.rc0.20230829git1c59d383390f.59.fc40.x86_64 FAILS * Fedora-Workstation-Live-Rawhide-20230829.n.0 w/6.5.0-57.fc40.x86_64 WORKS * Custom Rawhide w/6.5.0-57.fc40.x86_64 fully updated 20230830 WORKS * Custom Rawhide w/6.6.0-0.rc0.20230831gitb97d64c72259 fully updated 20230831 FAILS
I've installed F38 and upgraded to Rawhide on the impacted hardware. I'm booting kernel-6.4.12-200.fc38.x86_64 and have been installing and testing the Rawhide 6.6.x series and so far none of those kernels are booting.
(In reply to Jonathan Steffan from comment #13) > I've installed F38 and upgraded to Rawhide on the impacted hardware. I'm > booting kernel-6.4.12-200.fc38.x86_64 and have been installing and testing > the Rawhide 6.6.x series and so far none of those kernels are booting. To be clear, I'm using kernel-6.4.12-200.fc38.x86_64 to manage the system. I have boot tested all of the kernel-6.6.0 builds for fc40 and it hangs just like the live image.
I tried booting with enabling as much debugging as I could find as kernel args and still no additional details. https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html debug loglevel=8 acpi.debug_layer=0x2 acpi.debug_level=0xffffffff apic=debug debug_boot_weak_hash debug_objects debugpat dynamic_debug.verbose=3 dyndbg="file init/* +p #cmt ; func parse_one +p" early_ioremap_debug earlyprintk=vga efi=debug initcall_debug keep_bootcon lsm.debug memblock=debug mminit_loglevel=4 mtrr=debug pnp.debug=1 sched_verbose show_lapic=all udbg-immortal unwind_debug I confirmed that CONFIG_EARLY_PRINTK and CONFIG_DYNAMIC_DEBUG are enabled. Is there anything else I should try?
Not that I know of, but at this point it's probably worth reporting this upstream, to bugzilla.kernel.org or LKML...someone there should hopefully be able to provide better guidance.
Okay, I'll start working on that. I can confirm that building mainline (6.5.0) with the kernel config from the running system (i.e. `make localmodconfig`) works. Using the same technique with linux-next (6.5.0-next-20230831) the system doesn't boot. I've still not gotten it to tell me why so I'll just go through the process of finding the last working commit and report upstream.
I've been able to reproduce the boot failure with kernel 6.5.0-rc5-next-20230809. The last working kernel for this hardware is 6.5.0-rc5-next-20230808.
Ryan, It would be good to know which boot media (e.g. the kernel on the image) is working on your hardware. Please let us know which of these ISOs boot on your hardware: https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230828.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230828.n.0.iso https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230829.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230829.n.0.iso https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230830.n.0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide-20230830.n.0.iso
(In reply to Jonathan Steffan from comment #19) > Ryan, > > It would be good to know which boot media (e.g. the kernel on the image) is > working on your hardware. Please let us know which of these ISOs boot on > your hardware: > > https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230828.n. > 0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide- > 20230828.n.0.iso > https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230829.n. > 0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide- > 20230829.n.0.iso > https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20230830.n. > 0/compose/Workstation/x86_64/iso/Fedora-Workstation-Live-x86_64-Rawhide- > 20230830.n.0.iso Okay I will test them when they finsih downloading. Just tested the current iso for today will try when they finish downloading. Ty.
None of them work. All get stuck at grub> command line.
Ryan, Hmm, I would have expected at least one of them to work. It looks like your hardware boots with 6.5.0-0.rc7.20230821gitf7757129e3de.50.fc39.x86_64 (https://koji.fedoraproject.org/koji/buildinfo?buildID=2277075) which was tagged after the 6.5.0-rc5-next-20230809 version my hardware stopped booting with. I'll try this tag specifically in a bit. Since you have a booting system, will you please let us know what the last kernel is that is working for you and when it stopped booting? Locally installing and boot testing the kernel is enough as this does not appear to be live image related (at least not for me).
I am not using the worksation iso, I am using the server or custom iso. I can't get it to work when I use it via the usb, it only goes to grub.
Ryan, Your issue is unrelated to what I'm working on debugging it seems. We should split that out to a different BZ.
I've been building and then trying to boot each linux-next tag since next-20230809 and started looking at all of the commits merged between next-20230808..next-20230809. I was able to get some debugging information after learning more about kernel debugging during "The Linux Foundation's: Mentorship Session: Linux Kernel Debugging Tricks of the Trade" from today. The main thing that got it booting was "debug kgdboc=kbd nokaslr" and I added more to try to get as much detail from the boot as possible. I've built 6.6.0-rc1 and booted it with: BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.6.0-rc1 root=UUID=88bfe01c-0878-4864-bf50-6c372bd1ffef ro rootflags=subvol=root crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M debug loglevel=8 acpi.debug_layer=0x2 acpi.debug_level=0xffffffff apic=debug debug_boot_weak_hash debug_objects early_ioremap_debug earlyprintk=vga efi=debug initcall_debug keep_bootcon memblock=debug kgdboc=kbd nokaslr The main findings are multiple mm/early_ioremap.c warnings (see attached full boot log for them all): Sep 12 18:38:22 fedora kernel: __early_ioremap(0x0000000088ff7000, 00001000) [1] => 00000000 + ffffffffff240000 Sep 12 18:38:22 fedora kernel: WARNING: CPU: 0 PID: 0 at mm/early_ioremap.c:155 __early_ioremap+0x183/0x1c0 Sep 12 18:38:22 fedora kernel: Modules linked in: Sep 12 18:38:22 fedora kernel: CPU: 0 PID: 0 Comm: swapper Tainted: G W 6.6.0-rc1 #4 Sep 12 18:38:22 fedora kernel: Hardware name: Apple Inc. MacBookPro14,1/Mac-B4831CEBD52A0C4C, BIOS 447.40.12.0.0 09/25/2021 Sep 12 18:38:22 fedora kernel: RIP: 0010:__early_ioremap+0x183/0x1c0 Sep 12 18:38:22 fedora kernel: Code: ff ff ff 34 dd 40 7a a6 83 4d 89 e9 41 89 e8 4c 89 e1 48 c7 c6 f0 14 28 82 48 c7 c7 40 c8 8e 82 48 8d 54 24 08 e8 2d 5b 80 fd <0f> 0b 58 eb 9f > Sep 12 18:38:22 fedora kernel: RSP: 0000:ffffffff83003cc0 EFLAGS: 00010082 ORIG_RAX: 0000000000000000 Sep 12 18:38:22 fedora kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000000 Sep 12 18:38:22 fedora kernel: RDX: ffffffff83144108 RSI: 0000000000000086 RDI: ffffffff83144100 Sep 12 18:38:22 fedora kernel: RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000ffff10 Sep 12 18:38:22 fedora kernel: R10: 000000000000000f R11: 000000000000000f R12: 0000000000001000 Sep 12 18:38:22 fedora kernel: R13: 0000000000000000 R14: 00000000000005be R15: 0000000000000000 Sep 12 18:38:22 fedora kernel: FS: 0000000000000000(0000) GS:ffffffff83873000(0000) knlGS:0000000000000000 Sep 12 18:38:22 fedora kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 12 18:38:22 fedora kernel: CR2: ffff88826efff000 CR3: 0000000007020000 CR4: 00000000000200f0 Sep 12 18:38:22 fedora kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 12 18:38:22 fedora kernel: DR3: 0000000000000080 DR6: 00000000fffe0ff0 DR7: 0000000020000400 Sep 12 18:38:22 fedora kernel: Call Trace: Sep 12 18:38:22 fedora kernel: <TASK> Sep 12 18:38:22 fedora kernel: ? __early_ioremap+0x183/0x1c0 Sep 12 18:38:22 fedora kernel: ? __warn+0x81/0x130 Sep 12 18:38:22 fedora kernel: ? __early_ioremap+0x183/0x1c0 Sep 12 18:38:22 fedora kernel: ? report_bug+0x171/0x1a0 Sep 12 18:38:22 fedora kernel: ? early_fixup_exception+0x96/0xb0 Sep 12 18:38:22 fedora kernel: ? early_idt_handler_common+0x2f/0x3a Sep 12 18:38:22 fedora kernel: ? __early_ioremap+0x183/0x1c0 Sep 12 18:38:22 fedora kernel: ? acpi_tb_validate_table+0x3d/0x60 Sep 12 18:38:22 fedora kernel: ? acpi_tb_get_table+0x43/0x70 Sep 12 18:38:22 fedora kernel: ? acpi_get_table+0x63/0xa0 Sep 12 18:38:22 fedora kernel: ? acpi_table_parse_entries_array+0x8f/0x3b0 Sep 12 18:38:22 fedora kernel: ? _printk+0x64/0x80 Sep 12 18:38:22 fedora kernel: ? acpi_table_parse_madt+0x6d/0x90 Sep 12 18:38:22 fedora kernel: ? __pfx_acpi_parse_nmi_src+0x10/0x10 Sep 12 18:38:22 fedora kernel: ? acpi_boot_init+0x47c/0x620 Sep 12 18:38:22 fedora kernel: ? __pfx_acpi_parse_lapic+0x10/0x10 Sep 12 18:38:22 fedora kernel: ? __pfx_acpi_parse_x2apic+0x10/0x10 Sep 12 18:38:22 fedora kernel: ? setup_arch+0xadd/0x10b0 Sep 12 18:38:22 fedora kernel: ? slab_is_available+0x9/0x20 Sep 12 18:38:22 fedora kernel: ? start_kernel+0x6d/0xa90 Sep 12 18:38:22 fedora kernel: ? x86_64_start_reservations+0x18/0x30 Sep 12 18:38:22 fedora kernel: ? x86_64_start_kernel+0x96/0xa0 Sep 12 18:38:22 fedora kernel: ? secondary_startup_64_no_verify+0x17d/0x18b Sep 12 18:38:22 fedora kernel: </TASK> Sep 12 18:38:22 fedora kernel: ---[ end trace 0000000000000000 ]--- And then later in the boot a kernel/iomem.c warning: Sep 12 18:38:22 fedora kernel: ------------[ cut here ]------------ Sep 12 18:38:22 fedora kernel: memremap attempted on mixed range 0x0000000000000000 size: 0x10000 Sep 12 18:38:22 fedora kernel: WARNING: CPU: 1 PID: 1 at kernel/iomem.c:78 memremap+0x178/0x1c0 Sep 12 18:38:22 fedora kernel: Modules linked in: Sep 12 18:38:22 fedora kernel: CPU: 1 PID: 1 Comm: swapper/0 Tainted: G W 6.6.0-rc1 #4 Sep 12 18:38:22 fedora kernel: Hardware name: Apple Inc. MacBookPro14,1/Mac-B4831CEBD52A0C4C, BIOS 447.40.12.0.0 09/25/2021 Sep 12 18:38:22 fedora kernel: RIP: 0010:memremap+0x178/0x1c0 Sep 12 18:38:22 fedora kernel: Code: 06 ff ff ff 80 3d 48 3c f8 01 00 0f 85 73 ff ff ff 48 89 ea 48 89 e6 48 c7 c7 78 4e 8e 82 c6 05 2e 3c f8 01 01 e8 d8 59 dc ff <0f> 0b 31 c0 eb > Sep 12 18:38:22 fedora kernel: RSP: 0000:ffffc90000047d18 EFLAGS: 00010286 Sep 12 18:38:22 fedora kernel: RAX: 0000000000000000 RBX: 0000000000000001 RCX: c00000010004f89b Sep 12 18:38:22 fedora kernel: RDX: 0000000000000000 RSI: 000000010004f89b RDI: 0000000000000001 Sep 12 18:38:22 fedora kernel: RBP: 0000000000010000 R08: 0000000000000000 R09: ffffc90000047ba0 Sep 12 18:38:22 fedora kernel: R10: 0000000000000003 R11: ffffffff83138d68 R12: 000000000000000a Sep 12 18:38:22 fedora kernel: R13: ffffffff83ecc560 R14: ffff88810025c300 R15: 00000000000000a0 Sep 12 18:38:22 fedora kernel: FS: 0000000000000000(0000) GS:ffff888266c80000(0000) knlGS:0000000000000000 Sep 12 18:38:22 fedora kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 12 18:38:22 fedora kernel: CR2: 0000000000000000 CR3: 0000000007020001 CR4: 00000000003706e0 Sep 12 18:38:22 fedora kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Sep 12 18:38:22 fedora kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Sep 12 18:38:22 fedora kernel: Call Trace: Sep 12 18:38:22 fedora kernel: <TASK> Sep 12 18:38:22 fedora kernel: ? memremap+0x178/0x1c0 Sep 12 18:38:22 fedora kernel: ? __warn+0x81/0x130 Sep 12 18:38:22 fedora kernel: ? memremap+0x178/0x1c0 Sep 12 18:38:22 fedora kernel: ? report_bug+0x171/0x1a0 Sep 12 18:38:22 fedora kernel: ? prb_read_valid+0x1b/0x30 Sep 12 18:38:22 fedora kernel: ? handle_bug+0x3c/0x80 Sep 12 18:38:22 fedora kernel: ? exc_invalid_op+0x17/0x70 Sep 12 18:38:22 fedora kernel: ? asm_exc_invalid_op+0x1a/0x20 Sep 12 18:38:22 fedora kernel: ? memremap+0x178/0x1c0 Sep 12 18:38:22 fedora kernel: ? memremap+0x178/0x1c0 Sep 12 18:38:22 fedora kernel: efisubsys_init+0x605/0x6e0 Sep 12 18:38:22 fedora kernel: ? __pfx_efisubsys_init+0x10/0x10 Sep 12 18:38:22 fedora kernel: do_one_initcall+0x5a/0x320 Sep 12 18:38:22 fedora kernel: kernel_init_freeable+0x31b/0x470 Sep 12 18:38:22 fedora kernel: ? __pfx_kernel_init+0x10/0x10 Sep 12 18:38:22 fedora kernel: kernel_init+0x1a/0x1c0 Sep 12 18:38:22 fedora kernel: ret_from_fork+0x31/0x50 Sep 12 18:38:22 fedora kernel: ? __pfx_kernel_init+0x10/0x10 Sep 12 18:38:22 fedora kernel: ret_from_fork_asm+0x1b/0x30 Sep 12 18:38:22 fedora kernel: </TASK> Sep 12 18:38:22 fedora kernel: ---[ end trace 0000000000000000 ]---
Created attachment 1988546 [details] 6.6.0-rc1 debug boot log
Booting the latest rawhide kernel 6.6.0-0.rc0.20230904git708283abf896.6.fc40.x86_64 fails to boot without the debug options and shows the same warning.
Created attachment 1988547 [details] 6.6.0-0.rc0.20230904git708283abf896.6.fc40.x86_64 debug boot log
This hardware boots correctly with the 6.6.2-201.fc39.x86_64 kernel in updates-testing. It seems like this issue has been resolved.