Bug 1783733

Summary: Kernel 5.4.x instantly hangs or reboots on dual epyc 7742 at bootup unless edac_report=off is set on the kernel command-line
Product: [Fedora] Fedora Reporter: Gregory Maxwell <gmaxwell>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 31CC: airlied, bskeggs, hdegoede, ichavero, itamar, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, masami256, mchehab, mjg59, steved
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: ---
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-25 22:30:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
journalctl --no-hostname -k > dmesg.txt #for a successful boot with 5.1.15 none

Description Gregory Maxwell 2019-12-15 03:17:37 UTC 
Created attachment 1645252 [details]
journalctl --no-hostname -k > dmesg.txt #for a successful boot with 5.1.15

1. Please describe the problem:


Two process Epyc 7742 system, MBD-H11DSI-B motherboard

Upgraded from F30 to F31 and the system *instantly* reboots after the edd probing line. 


2. What is the Version-Release number of the kernel:

5.3.15-300.fc31

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

5.1.15-300.fc30 boots and runs,  5.3.15-200.fc30 fails in the same way.

I was 5.1.15 before the upgrade and had just never rebooted into 5.3.15 (or anything else later than 5.1.15). I believe it would have failed if I had and that the upgrade isn't at fault

5.0.9-301.fc30.x86_64 also works.

If it would be useful I could try bisecting older versions.

I tried edd=off noapic and apci=off individually and combined with no luck. nomodeset also has no effect (unsurprisingly, the failure is long before modesetting would happen).

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Yes, turn on the computer. :) Fails every time unless I select a 5.1 or 5.0 kernel in grub.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Looks like its the same kernel in rawhide right now?

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

There is no journal records for the failed boots the restart is very fast after the edd message (fast enough that I couldn't read it if I didn't know what it said).

Logs of a _successful_ boot attached.

I'm happy to conduct exploratory tests, attach a serial console, whatever.
Comment 1 Gregory Maxwell 2020-01-06 05:16:05 UTC 
5.3.16-300.fc31 fails the same way.
Comment 2 Gregory Maxwell 2020-01-08 02:06:56 UTC 
kernel 5.4.7-200.fc31.x86_64 also fails the same way.
Comment 3 Gregory Maxwell 2020-01-23 16:47:12 UTC 
5.4.12-200.fc31 also fails the same way.

I would be totally open to making exploratory changes, running patches, or doing additional diagnostics. Since it restarts so early in the boot with no output I am at a loss as to where to start.
Comment 4 Gregory Maxwell 2020-01-26 07:26:41 UTC 
Today was going to be another useless update-- "Kernel 5.4.13 no change"-- but the fates intervened.

With month and a half of no response here at all I've realized now that some mysterious force has removed all other humans from the earth and that I'm truly alone in this. So I decided to try a little harder this time. I replaced the ram. No joy.

I noticed that there was a new bios version for my motherboard (2.0b rather than 2.0) and so I installed it to see if it would help. Unfortunately, it didn't and the older kernel became unbootable too.  But now instead of instantly rebooting the system hang. Since going back would probably required contacting supermicro support to get the older version I was stuck.

In desperation I went through all the kernel command-line options in kernel-parameters.txt and added every one that seemed like it could possibly help or at least increase the amount of information I got.  Sure enough, it booted!

After much laborious bisection I found that adding edac_report=off is what makes it boot.

I notice that HPE tells users to set edac_report=off on RHEL for their systems: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00041567en_us&docLocale=en_US

As stated before, I'd be happy to help diagnose this further should mankind return to the earth. Until then, I better go find some provisions before the power goes out.
Comment 5 Justin M. Forbes 2020-03-03 16:23:11 UTC 
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 31 kernel bugs.

Fedora 31 has now been rebased to 5.5.7-200.fc31.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 32, and are still experiencing this issue, please change the version to Fedora 32.

If you experience different issues, please open a new bug report for those.
Comment 6 Justin M. Forbes 2020-03-25 22:30:33 UTC 
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.
Comment 7 Red Hat Bugzilla 2023-09-14 05:48:46 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days