Bug 1783733 - Kernel 5.4.x instantly hangs or reboots on dual epyc 7742 at bootup unless edac_report=off is set on the kernel command-line
Summary: Kernel 5.4.x instantly hangs or reboots on dual epyc 7742 at bootup unless ed...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-15 03:17 UTC by Gregory Maxwell
Modified: 2023-09-14 05:48 UTC (History)
17 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2020-03-25 22:30:33 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
journalctl --no-hostname -k > dmesg.txt #for a successful boot with 5.1.15 (215.76 KB, text/plain)
2019-12-15 03:17 UTC, Gregory Maxwell
no flags Details

Description Gregory Maxwell 2019-12-15 03:17:37 UTC
Created attachment 1645252 [details]
journalctl --no-hostname -k > dmesg.txt #for a successful boot with 5.1.15

1. Please describe the problem:


Two process Epyc 7742 system, MBD-H11DSI-B motherboard

Upgraded from F30 to F31 and the system *instantly* reboots after the edd probing line. 


2. What is the Version-Release number of the kernel:

5.3.15-300.fc31

3. Did it work previously in Fedora? If so, what kernel version did the issue
   *first* appear?  Old kernels are available for download at
   https://koji.fedoraproject.org/koji/packageinfo?packageID=8 :

5.1.15-300.fc30 boots and runs,  5.3.15-200.fc30 fails in the same way.

I was 5.1.15 before the upgrade and had just never rebooted into 5.3.15 (or anything else later than 5.1.15). I believe it would have failed if I had and that the upgrade isn't at fault

5.0.9-301.fc30.x86_64 also works.

If it would be useful I could try bisecting older versions.

I tried edd=off noapic and apci=off individually and combined with no luck. nomodeset also has no effect (unsurprisingly, the failure is long before modesetting would happen).

4. Can you reproduce this issue? If so, please provide the steps to reproduce
   the issue below:

Yes, turn on the computer. :) Fails every time unless I select a 5.1 or 5.0 kernel in grub.

5. Does this problem occur with the latest Rawhide kernel? To install the
   Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by
   ``sudo dnf update --enablerepo=rawhide kernel``:

Looks like its the same kernel in rawhide right now?

6. Are you running any modules that not shipped with directly Fedora's kernel?:

No.

7. Please attach the kernel logs. You can get the complete kernel log
   for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the
   issue occurred on a previous boot, use the journalctl ``-b`` flag.

There is no journal records for the failed boots the restart is very fast after the edd message (fast enough that I couldn't read it if I didn't know what it said).

Logs of a _successful_ boot attached.

I'm happy to conduct exploratory tests, attach a serial console, whatever.

Comment 1 Gregory Maxwell 2020-01-06 05:16:05 UTC
5.3.16-300.fc31 fails the same way.

Comment 2 Gregory Maxwell 2020-01-08 02:06:56 UTC
kernel 5.4.7-200.fc31.x86_64 also fails the same way.

Comment 3 Gregory Maxwell 2020-01-23 16:47:12 UTC
5.4.12-200.fc31 also fails the same way.

I would be totally open to making exploratory changes, running patches, or doing additional diagnostics. Since it restarts so early in the boot with no output I am at a loss as to where to start.

Comment 4 Gregory Maxwell 2020-01-26 07:26:41 UTC
Today was going to be another useless update-- "Kernel 5.4.13 no change"-- but the fates intervened.

With month and a half of no response here at all I've realized now that some mysterious force has removed all other humans from the earth and that I'm truly alone in this. So I decided to try a little harder this time. I replaced the ram. No joy.

I noticed that there was a new bios version for my motherboard (2.0b rather than 2.0) and so I installed it to see if it would help. Unfortunately, it didn't and the older kernel became unbootable too.  But now instead of instantly rebooting the system hang. Since going back would probably required contacting supermicro support to get the older version I was stuck.

In desperation I went through all the kernel command-line options in kernel-parameters.txt and added every one that seemed like it could possibly help or at least increase the amount of information I got.  Sure enough, it booted!

After much laborious bisection I found that adding edac_report=off is what makes it boot.

I notice that HPE tells users to set edac_report=off on RHEL for their systems: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00041567en_us&docLocale=en_US

As stated before, I'd be happy to help diagnose this further should mankind return to the earth. Until then, I better go find some provisions before the power goes out.

Comment 5 Justin M. Forbes 2020-03-03 16:23:11 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 31 kernel bugs.

Fedora 31 has now been rebased to 5.5.7-200.fc31.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you have moved on to Fedora 32, and are still experiencing this issue, please change the version to Fedora 32.

If you experience different issues, please open a new bug report for those.

Comment 6 Justin M. Forbes 2020-03-25 22:30:33 UTC
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.

Comment 7 Red Hat Bugzilla 2023-09-14 05:48:46 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days


Note You need to log in before you can comment on or make changes to this bug.