Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
For bugs related to Red Hat Enterprise Linux 4 product line. The current stable release is 4.9. For Red Hat Enterprise Linux 6 and above, please visit Red Hat JIRA https://issues.redhat.com/secure/CreateIssue!default.jspa?pid=12332745 to report new issues.

Bug 570833

Summary: Kernel panic: EDAC MC0: INTERNAL ERROR: channel-b out of range
Product: Red Hat Enterprise Linux 4 Reporter: Bryn M. Reeves <bmr>
Component: kernelAssignee: Jacob Tanenbaum <jtanenba>
Status: CLOSED WONTFIX QA Contact: Red Hat Kernel QE team <kernel-qe>
Severity: high Docs Contact:
Priority: urgent    
Version: 4.8CC: clalance, dhoward, fhirtz, jwest, pbatkowski, pbonzini, rdassen, sforsber, syeghiay, tao, tom, xen-maint
Target Milestone: rcKeywords: Patch
Target Release: 4.9   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 533391 Environment:
Last Closed: 2012-10-25 18:46:41 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 533391    
Bug Blocks:    
Attachments:
Description Flags
118f3e1afd5534c15f9701f33514186cfc841a27 re-diffed for EL4 none

Description Bryn M. Reeves 2010-03-05 15:06:51 UTC 
+++ This bug was initially created as a clone of Bug #533391 +++

Created an attachment (id=367831)
Full console log

EDAC MC0: INTERNAL ERROR: channel-b out of range (4 >= 4)
Kernel panic - not syncing: EDAC MC0: Uncorrected Error
 (XEN) Domain 0 crashed: 'noreboot' set - not rebooting.

Or on EL4:

EDAC MC0: INTERNAL ERROR: channel-b out of range (4 >= 4)
Kernel panic - not syncing: MC0: Uncorrected Error
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at panic:77
invalid operand: 0000 [1] SMP  
CPU 4  

Kernel is 2.6.9-89.0.11.EL but the code is unchanged in later EL4 kernels.

Server board is a Supermicro X7DBi+ (Intel 5000P chipset) with 16GB RAM and two quad-core Xeon CPUs.

This is the first time it happened; server was rebooted about a week ago.

--- Additional comment from tom on 2009-11-06 12:05:29 EST ---

The bug is probably in the edac_mc.c function: i5000_process_nonfatal_error_info()

The i5000X data sheet says that FERR_NF_FBD bit 28 has no significance for M4Err through M12Err (=FERR_NF_UNCORRECTABLE bits). (it's on page 211 of the PDF)
The channel number is taken from bits 29:28 by this macro:
EXTRACT_FBDCHAN_INDX(x) (((x)>>28) & 0x3)

ue_errors = allErrors & FERR_NF_UNCORRECTABLE;
if (ue_errors) {
    debugf0("\tUncorrected bits= 0x%x\n", ue_errors);
    branch = EXTRACT_FBDCHAN_INDX(info->ferr_nf_fbd);
    channel = branch;
    [...]
    /* Call the helper to output message */
    edac_mc_handle_fbd_ue(mci, rank, channel, channel + 1, msg);
}

If both of bits 29:28 are set then channel+1 in the last line above yields 4, which is out of range.

The fix would be to replace the line:
    channel = branch;
with:
    channel = branch & 2;

--- Additional comment from tom on 2009-11-09 10:40:10 EST ---

Also submitted as: http://bugzilla.kernel.org/show_bug.cgi?id=14568

--- Additional comment from mchehab on 2009-12-09 06:33:30 EST ---

Created an attachment (id=377139)
Fix i5000 error when reporting the first fatal errors (FERR_FAT_FBD)

As reported, bit 28 of FERR_FAT_FBD is not reported properly by the chipset. Due to that, it may happen that both bits 28 and 29 to be one, giving an out-of-range value.

This patch fixes it on RHEL5 kernel.

The bug is also present upstream. I've sent a patch upstream for it.

--- Additional comment from mchehab on 2009-12-09 06:56:28 EST ---

(In reply to comment #4)
> Created an attachment (id=377139) [details]
> Fix i5000 error when reporting the first fatal errors (FERR_FAT_FBD)
> 
> As reported, bit 28 of FERR_FAT_FBD is not reported properly by the chipset.

In time: 
The issue is at FERR_NF_FBD (the first non-fatal error). The patch is correct,
I just made a typo on comment #4.

Let's wait for upstream to commit it, before adding it on RHEL5.

--- Additional comment from mchehab on 2010-01-18 09:14:41 EST ---

Patch were committed upstream on Jan, 15 at changeset 118f3e1afd5534c15f9701f33514186cfc841a27.

Patch posted at the ML.
Comment 1 Bryn M. Reeves 2010-03-05 15:10:36 UTC 
Created attachment 398061 [details]
118f3e1afd5534c15f9701f33514186cfc841a27 re-diffed for EL4
Comment 2 Bryn M. Reeves 2010-03-09 12:02:01 UTC 
Have a report that this is still failing with the patch from comment #1:

EDAC MC0: INTERNAL ERROR: channel-b out of range (4 >= 4)


[...network console startup...]
EDAC MC0: INTERNAL ERROR: channel-b out of range (4 >= 4)
Kernel panic - not syncing: MC0: Uncorrected Error
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at panic:77
invalid operand: 0000 [1] SMP
CPU 4
Modules linked in: sr_mod nfsd exportfs lp netconsole netdump autofs4 nfs lockd nfs_acl parport_pc parport sunrpc ds yenta_socket pcmcia_core cpufreq_powersave ib_srp ib_sdp ib_ipoib inet_lro rdma_ucm rdma_cm iw_cm ib_addr ib_umad ib_ucm ib_uverbs ib_cm ib_sa ib_mad ib_core dm_mirror dm_multipath dm_mod button battery ac nvidia(U) i2c_core md5 ipv6 uhci_hcd ehci_hcd i5000_edac edac_mc hw_random snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_hwdep snd soundcore tg3 floppy ext3 jbd ata_piix mptscsih mptsas mptspi mptscsi mptbase ahci libata sd_mod scsi_mod
Pid: 2812, comm: kedac Tainted: P      2.6.9-89.0.11.0.it311629.ELsmp
RIP: 0010:[<ffffffff8013899a>] <ffffffff8013899a>{panic+211}
RSP: 0000:000001080ed2bb18  EFLAGS: 00010286
RAX: 0000000000000036 RBX: ffffffffa01cb885 RCX: 0000000000000246
RDX: 0000000000006330 RSI: 0000000000000246 RDI: ffffffff803f6b00
RBP: 000001080ed2bce8 R08: 0000000000000004 R09: ffffffffa01cb885
R10: 0000000000000000 R11: 0000000000000000 R12: 00000000000003c0
R13: 000001080ed2bc10 R14: 0000000000000004 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffffffff80504b80(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000002a95557000 CR3: 0000000037e5c000 CR4: 00000000000006e0
Process kedac (pid: 2812, threadinfo 000001080ed2a000, task 0000010811c167f0)
Stack: 0000003000000010 000001080ed2bbf8 000001080ed2bb38 000001080ed2bc08
     000001080ed2bb48 0000000000000000 ffffffff803f6ae8 ffffffff803f6ae8
     ffffffff803f6ae8 000001080eee2000
Call Trace:<ffffffff801f0fe3>{vsnprintf+1406} <ffffffff801f0db5>{vsnprintf+848}
[snip]

Not sure how that's happening as we now have:

channel = branch & 2;
channelb = branch + 1;
channelb == 4??

Not sure how we ended up extracting a channel of 3 when we're masking & 2..
Comment 3 Bryn M. Reeves 2010-03-09 12:02:28 UTC 
that should be channelb = channel + 1;
Comment 4 Bryn M. Reeves 2010-03-09 12:13:34 UTC 
Rebuild kernel that BUGed in comment #2 with CONFIG_EDAC_DEBUG and will ask to re-test.
Comment 5 Tamas Vincze 2010-03-18 19:55:04 UTC 
The function i5000_process_fatal_error_info() still has:

  branch = EXTRACT_FBDCHAN_INDX(info->ferr_fat_fbd);
  channel = branch;
  [...]
  edac_mc_handle_fbd_ue(mci, rank, channel, channel + 1, msg);

Someone should check if both bits 29:28 of ferr_fat_fbd are valid
for FERR_FAT_M1ERR, FERR_FAT_M2ERR and FERR_FAT_M3ERR.

The original patch clears bit 28 for nonfatal errors only.
Comment 6 Bryn M. Reeves 2010-04-28 16:34:48 UTC 
Additional edac_debug_level=2 output from one of the affected systems:

390 MC: i5000_probe1(): Number of - Channels= 4  DIMMS= 8  CSROWS= 16
391 MC: drivers/edac/i5000_edac.c: i5000_probe1(): mci = 000001080f13e000
392 System Address, processor bus- PCI Bus ID: 0000:00:10.0  8086:25f0
393 Branchmap, control and errors - PCI Bus ID: 0000:00:10.1  8086:25f0
394 FSB Error Regs - PCI Bus ID: 0000:00:10.2  8086:25f0
395 AMBASE= 0xfe000000  MAXCH= 4  MAX-DIMM-Per-CH= 8
396  
397 TOLM (number of 256M regions) =13 (0xd)
398 Actual TOLM byte addr=3489660928 (0xd0000000)
399 MIR0: limit= 0x80  WAY1= 2  WAY0= 1
400 MIR1: limit= 0x80  WAY1= 0  WAY0= 0
401 MIR2: limit= 0x80  WAY1= 0  WAY0= 0
402 MTR0 where=0x80 B0 value=0x195
403 MTR0 where=0x80 B1 value=0x195  
404 MTR1 where=0x84 B0 value=0x195
405 MTR1 where=0x84 B1 value=0x195  
406 MTR2 where=0x88 B0 value=0x195
407 MTR2 where=0x88 B1 value=0x195
408 MTR3 where=0x8c B0 value=0x195
409 MTR3 where=0x8c B1 value=0x195  
...
...
467 -----------------------------------------------------------
468 csrow 15       0 MB   |    0 MB   |    0 MB   |    0 MB   |
469 csrow 14       0 MB   |    0 MB   |    0 MB   |    0 MB   |
470 -----------------------------------------------------------
471 csrow 13       0 MB   |    0 MB   |    0 MB   |    0 MB   |
472 csrow 12       0 MB   |    0 MB   |    0 MB   |    0 MB   |
473 -----------------------------------------------------------
474 csrow 11       0 MB   |    0 MB   |    0 MB   |    0 MB   |
475 csrow 10       0 MB   |    0 MB   |    0 MB   |    0 MB   |
476 -----------------------------------------------------------
477 csrow  9       0 MB   |    0 MB   |    0 MB   |    0 MB   |
478 csrow  8       0 MB   |    0 MB   |    0 MB   |    0 MB   |
479 -----------------------------------------------------------
480 csrow  7    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
481 csrow  6    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
482 -----------------------------------------------------------
483 csrow  5    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
484 csrow  4    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
485 -----------------------------------------------------------
486 csrow  3    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
487 csrow  2    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
488 -----------------------------------------------------------
489 csrow  1    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
490 csrow  0    1024 MB   | 1024 MB   | 1024 MB   | 1024 MB   |
491 -----------------------------------------------------------
492             channel 0 | channel 1 | channel 2 | channel 3 |
493  
494 MC: Enable error reporting now

Waiting on an additional file with the lines from the moments leading up to a crash...
Comment 7 James M. Leddy 2010-08-03 14:56:06 UTC 
(In reply to comment #5)
> The function i5000_process_fatal_error_info() still has:
> 
>   branch = EXTRACT_FBDCHAN_INDX(info->ferr_fat_fbd);
>   channel = branch;
>   [...]
>   edac_mc_handle_fbd_ue(mci, rank, channel, channel + 1, msg);
> 
> Someone should check if both bits 29:28 of ferr_fat_fbd are valid
> for FERR_FAT_M1ERR, FERR_FAT_M2ERR and FERR_FAT_M3ERR.
> 
> The original patch clears bit 28 for nonfatal errors only.    

Only for correctable errors though. Since comment #2 specifies uncorrectable error, this is still impossible.