Bug 463144

Created attachment 317347 [details]
output from the serial console

Search for "Trigger a crashdump" to find the start of Kdump Kernel boot.

This sounds alot like bz 458047.  Do you happen to to know the last good kernel that could dump on this system?  Is it possible for you to test this under kernel 2.6.18-92, to see if the last update worked on this system (test with the most recent kexec-tools if you would please).  Thanks!

I have tried 2.6.18-92.el5 without luck. This is a new system just has been added to RHTS recently, so I don't know any known working Kernel for it. As this is the system with huge memory, I am reserving this system to see if restricting memory size (mem=<small size>) makes any difference.

Looks like this is very similar to BZ 458047. I have seen the same warning here while copying of vmcore,

hda: cdrom_pc_intr: The drive appears confused (ireason = 0x01). Trying to recover by ending request.

Read from anywhere onward on 944353280 will get EINVAL. If I opened the vmcore-incomplete with crash, I got this:

[root@hp-dl785g5-01 ~]# crash /usr/lib/debug/lib/modules/2.6.18-116.el5/vmlinux vmcore-incomplete 

crash 4.0-5.0.3
Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
WARNING: vmcore-incomplete: may be truncated or incomplete
         PT_LOAD p_offset: 21606980
                 p_filesz: 3335831552
           bytes required: 3357438532
            dumpfile size: 944357376

GNU gdb 6.1
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...

      KERNEL: /usr/lib/debug/lib/modules/2.6.18-116.el5/vmlinux
    DUMPFILE: vmcore-incomplete
        CPUS: 32
        DATE: Tue Sep 23 03:15:24 2008
      UPTIME: 00:06:45
LOAD AVERAGE: 0.13, 0.07, 0.02
       TASKS: 391
    NODENAME: hp-dl785g5-01.rhts.bos.redhat.com
     RELEASE: 2.6.18-116.el5
     VERSION: #1 SMP Thu Sep 18 18:06:03 EDT 2008
     MACHINE: x86_64  (2300 Mhz)
      MEMORY: 1 GB
       PANIC: "SysRq : Trigger a crashdump"
         PID: 5892
     COMMAND: "bash"
        TASK: ffff810037e1e040  [THREAD_INFO: ffff81003a92c000]
         CPU: 16
       STATE: TASK_RUNNING (SYSRQ)

crash>

Theres certainly no reason reading page number 230556.  Thats right in the middle of the bios provided e820 map.  The EINVAL return is definately interesting, that only gets returned while reading for a very limited number of reasons.  Either because the address we request is beyond the max_saved_pfn value in the kernel, or because we have no offset to paddr mapping for that address.  I'm going to guess that somehow in our kdump boot munging we've altered the max_saved_pfn value so its much smaller than it needs to be.

Cai, can you set this system up so tha on a kdump it returns us to a login prompt in the kernel ?  I'd like to run a stap script in both the nominal and  kdump kernels to compare the value of saved_max_pfn, as they should be the same.  If you would prefer just let me know when I can hop on it and I'll take a look myself.  Thanks!

Neil, I have configured the Kdump with an empty kdump.conf and "chkconfig kdump off", so if you manually "service kdump on" in normal Kernel and do a SysRq-C, it will run init and give you a login prompt.

Anyway, the machine has been reserved, and is currently running Kdump Kernel, so you could ssh to it now. When done, just reboot it to return to the normal Kernel.

prarit posted a fix for a bug in the parsing of user memory map specifications
recently, and I think this may be related to that.  I've built kernels here:
/mnt/redhat/brewroot/scratch/nhorman/task_1499612

If you could please, try that kernel on this system and see if it doesn't clear
up the problem.  I have  a feeling that it will

Neil, I have tried kernel-2.6.18-117.el5.memmapfix without luck, the same problem.

Does anybody know the status of this system?  I can't find it in RHTS.

I can't either, although in another bug (bz 458047) which involved  a sun system randomly resetting that the cpu stepping was backlevel (stepping 2).  Moving to a identical system with cpus at stepping 3 made the problem go away.  That said, I see the dmidecode here shows cpu stepping three, but I wonder if the cpuid instruction (as exposed via /proc/cpuinfo) would show different.  Cai, do you know where this system is so that we can validate it?

This system is still on the old rhts controller, that's why I didn't see it.  However, it appears to be wedged and needs a physical reset.  I asked mbrodeur to reset it next time he is in the lab.  I will investigate this then.

I will check the /proc/cpuinfo that Neil was asking about and will also see if we have newer firmware available.

Thank you Doug!

Here is the /proc/cpuinfo data, however I am pretty sure this is a firmware issue.  There is a fix in the latest version of firmware that sounds like it is for this particular issue.

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 2
model name      : Quad-Core AMD Opteron(tm) Processor 8356
stepping        : 3
cpu MHz         : 2300.083
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt pdpe1gb rdtscp lm
 3dnowext 3dnow constant_tsc pni cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_
legacy altmovcr8 abm sse4a misalignsse 3dnowprefetch osvw
bogomips        : 4605.40
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate [8]

doug, do you have a link to the firmware fix?

(In reply to comment #19)
> doug, do you have a link to the firmware fix?

working on getting it updated now, will set this back to needinfo from me until tha tis done.

Hi!

I've the same problem here, but with an DL585 G5

BIOS:
        Version: A07
        Release Date: 03/28/2008

uname -a:
Linux name 2.6.18-92.1.1.el5 #1 SMP Thu May 22 09:01:47 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux


(I did a "set -x" in the init script on kdump initrd)

last message on the serial interface is cp /prov/vmcore ... 
then the server resets.

seems to be the same CPU ...

cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 2
model name      : Quad-Core AMD Opteron(tm) Processor 8356
stepping        : 3
cpu MHz         : 2300.000
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc pni cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy altmovcr8 abm sse4a misalignsse 3dnowprefetch osvw
bogomips        : 4624.00
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate [8]



br,
bernhard

PS: This is besides the stupid bug https://bugzilla.redhat.com/show_bug.cgi?id=360111 which I also hit :(

I updated the firmware on the dl785 but still see the same behavior.  I tried on a system back at HP and hit a MCA instead of a reset which I assume is related.

I did not get a chance to look at the dl585g5 to confirm if that is related.  I hope to get to that on monday.

Does the mca provide any details on the error?  Let us know what you find out

I don't think it needs any information for me at the moment.

I tried this on hp-dl585g5-01 and it does indeed look like that platform has the identical problem where it resets during copy of /proc/vmcore.

When I tried this back at HP on another dl785 I got this MCA which _might_ be related.  The MCA happens at the same time as the system here at RH resets.

HARDWARE ERROR
CPU 0: Machine Check Exception:                4 Bank 4: fe0000020005001b
TSC 1daa8aa6fccb3 ADDR 24000000 MISC e00c0ffe01000000
This is not a software problem!
Run through mcelog --ascii to decode and contact your hardware vendor


Running this through mcalog gives us:

HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
CPU 0 4 northbridge TSC 1daa8aa6fccb3 
  Northbridge GART error
       bit33 = err cpu1
       bit57 = processor context corrupt
       bit59 = misc error valid
       bit61 = error uncorrected
       bit62 = error overflow (multiple errors)
  TLB error 'generic transaction, level generic'
STATUS fe0000020005001b MCGSTATUS 4

We've seen the incomplete core dumps on Proliant DL365 G5 and DL585 G5.
Thinking it was an OS at first, I was debugging on the DL365 G5.  I also
hit a Machine Check Exception 3 times.
    HARDWARE ERROR. This is *NOT* a software problem!
    Please contact your hardware vendor
    CPU 0 BANK 4 TSC ec60892977
    STATUS be0000010005001b MCGSTATUS 4

In my case it was on a RHEL5U1 system.

http://lkml.org/lkml/2008/10/15/326

As per our IRC converstaion, dhcapman is going to test with this patch .  I tried this patch on bz 458047 to no avail, but given the MCA's above, it might be useful here

dchapman created a kernel for me to try with the patch from:
    http://lkml.org/lkml/2008/10/15/326
It worked for me on a DL365G5.

This bug is potentially going to hold up a product release.  Support, 
understandably so, is very concerned about not being able to get crash
dumps from customers.  What is the plan to address the bug?

well, I'm not sure what we can do at this point.  The patch is effectively dead
upstream given that no one has a system to confirm that can be used to confirm
this wont break.  Given that Bob M ontgomery at HP was the one proposing this
patch, he may be interested to know the results here.  Kris, perhaps you should
contact him and guide him to this bz.   The results collected here I think
satisfy the tests he was looking for in his last post to the thread referenced
in comment #29.  That would likely allow him to get the patch in upstream, and
then I would feel comfortable posting this for RHEL

Neil, 

What kind of system would we need to test the upstream patch on to make them happy? I read the thread and I *think* the answer is: 

- AMD Opteron (any generation?) 
- >=4GB memory
- AGP graphics adapter 

Anything else? We've got a customer report of what I believe is the same problem now so we're going to need to resolve it somehow or another in RHEL5. We should be able to come up with an early Opteron system matching the above internally - I think a Sun W2100z or W1100z workstation should fit the bill. 

-David

David, yes, thats the test case, but the consensus is that Opteron systems with AGP graphics adapters likely don't exist (or are very rare), but if you have one, please give it a test.  Thanks

They do exist, but they're only the very first generation of Opteron systems. In later generations PCIe had taken over from AGP. I can probably find one internally that fits the bill however Doug mentioned on IRC that he was working on an alternative patch that might find easier upstream acceptance. 

Doug, please let me know if you'd like unearth one of those early Opterons for testing or if you'd prefer giving your alternative patch a go first.

David,

Either way I think testing on such a platform would be a good thing.  I am fairly confident that my solution will ease the concerns given upstream but if we have more testing data that is always helpful.

Created attachment 325708 [details]
[PATCH 1/2] cleanup of AMD GART code

This brings RHEL5 closer to the upstream code.  No functional changes, just cleans the code up to get rid of hard coded register offsets and bitmasks.

Created attachment 325709 [details]
[PATCH 2/2] fix amd gart kdump issue

per side conversation.. there is a proposal in user space to
address this issue. The change is posted today:

http://lists.infradead.org/pipermail/kexec/2008-December/003092.html

hence change component to user space, kexec-tools.

Even if we go with the userspace fix we still need a kernel change.  That fix relies on having the GART region listed in /proc/iomem which is not the case in RHEL5.  If we go that route we need to backport this upstream commit to the kernel:

commit 56dd669a138c40ea6cdae487f233430d12372767
Author: Aaron Durbin <adurbin>
Date:   Tue Sep 26 10:52:40 2006 +0200

    [PATCH] Insert GART region into resource map
    
    Patch inserts the GART region into the iomem resource map. The GART will the
    be visible within /proc/iomem. It will also allow for other users
    utilizing the GART to subreserve the region (agp or IOMMU).
    
    Signed-off-by: Aaron Durbin <adurbin>
    Signed-off-by: Andi Kleen <ak>

per last comment, in order to get the user space change working,
there is a need for a kernel change to export GART region
through /proc/iomem. 

The bug to track the kernel change is bug 475507, fyi.