Bug 442101

Stack trace and other info from crash: (the crashme segfault messages are normal)

crashme[7789]: segfault at 00007fff77ee1425 rip 0000000000401a34 rsp
00007fff77ee141d error 6
crashme[7791]: segfault at 000000000000000d rip 000000000efd122b rsp
0000000051fd42dc error 4
crashme[7796]: segfault at 000000000000000a rip 000000000c90f221 rsp
0000000000000000 error 4
crashme[7822]: segfault at 0000000000000074 rip 000000001f2d822e rsp
00000000000000f4 error 4
Unable to handle kernel paging request at 00000000005967ac RIP:
 [<0000000011180220>]
PGD 1944e067 PUD 3c5b8067 PMD 3cee6067 PTE 0
Oops: 0000 [1] SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 0
Modules linked in: nfs fscache nfsd exportfs lockd nfs_acl auth_rpcgss loop
autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api
dm_multipath video sbs backlight
i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev
ide_cd i2c_i801 e752x_edac i2c_core floppy edac_mc sg tg3 pcspkr shpchp
serio_raw cdrom dm_snapshot
 dm_zero dm_mirror dm_mod ata_piix libata aic79xx scsi_transport_spi sd_mod
scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 7825, comm: crashme Not tainted 2.6.18-89.el5 #1
RIP: 0033:[<0000000011180220>]  [<0000000011180220>]
RSP: 002b:00007fffa0f66428  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000009 RCX: ffffffffffffffff
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000a
RBP: 0000000000000007 R08: 00007fffa0f65f20 R09: 0000000000000000
R10: 0000000000000008 R11: 0000000000000206 R12: 0000000000000000
R13: 00007fffa0f66588 R14: 0000000000000000 R15: 0000000000000000
FS:  00002b8c09b53210(0000) GS:ffffffff8039e000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000005967ac CR3: 000000003d2ad000 CR4: 00000000000006e0
Process crashme (pid: 7825, threadinfo ffff81002c05a000, task ffff810016ce30c0)

RIP  [<0000000011180220>]
 RSP <00007fffa0f66428>

bt of crashme processes:
PID: 7790   TASK: ffff8100310c87e0  CPU: 2   COMMAND: "crashme"
 #0 [ffff81003ff1ff30] crash_nmi_callback at ffffffff80079a5a
 #1 [ffff81003ff1ff40] do_nmi at ffffffff800659cc
 #2 [ffff81003ff1ff50] nmi at ffffffff80064e47
    RIP: 00000000165fe220  RSP: 00007fff0e901dc8  RFLAGS: 00000282
    RAX: 00000000000000a8  RBX: 0000000000000004  RCX: ffffffffffffffff
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 000000000000000a
    RBP: 0000000000000007   R8: 00007fff0e9018c0   R9: 0000000000000000
    R10: 0000000000000008  R11: 0000000000000202  R12: 0000000000000000
    R13: 00007fff0e901f28  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
--- <exception stack> ---

PID: 7812   TASK: ffff81000df4d7a0  CPU: 2   COMMAND: "crashme"
 #0 [ffff810034355ea0] schedule at ffffffff80062efb
 #1 [ffff810034355f78] retint_careful at ffffffff8005d68b
    RIP: 0000000016cfb222  RSP: 000000000d2738ec  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: 0000000000000008  RCX: ffffffffffffffff
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 000000000000000a
    RBP: ffffffff8005d68b   R8: 00007fff1a4e6cd0   R9: 0000000000000000
    R10: 0000000000000008  R11: 0000000000000206  R12: 0000000000000007
    R13: 0000000000000000  R14: 0000000000000000  R15: 00007fff1a4e7338
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b
PID: 7825   TASK: ffff810016ce30c0  CPU: 0   COMMAND: "crashme"
 #0 [ffff81002c05bd60] crash_kexec at ffffffff800aa935
 #1 [ffff81002c05be20] __die at ffffffff800650af
 #2 [ffff81002c05be60] do_page_fault at ffffffff80066a8e
 #3 [ffff81002c05bf50] error_exit at ffffffff8005dde9
    RIP: 0000000011180220  RSP: 00007fffa0f66428  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: 0000000000000009  RCX: ffffffffffffffff
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 000000000000000a
    RBP: 0000000000000007   R8: 00007fffa0f65f20   R9: 0000000000000000
    R10: 0000000000000008  R11: 0000000000000206  R12: 0000000000000000
    R13: 00007fffa0f66588  R14: 0000000000000000  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0033  SS: 002b

Forbidden

You don't have permission to access
/qa/qa/vmcores/bz442101/ibm-himalaya.rhts-20080410/vmcore-89-ibm-himalaya.rhts.compressed
on this server.
Apache/2.0.46 (Red Hat) Server at porkchop.redhat.com Port 80

oops.. I forgot to fix the permissions on the dump file, give it another try, it
should be fixed now.

I just started the same set of tests that produced this crash with the 0402.0
tree which includes the -88.el5 kernel.

Thanks, I'll take a look at the dump...

By just looking at the dmesg output, though, this certainly rings a bell,
where a page fault on a user address is mistakenly handled as a kernel
page fault.

crashme was running, generated a page fault on user address 5967ac,
while running in user-mode at text address 0033:0000000011180220,
but the kernel has been confused into thinking its a kernel-mode
page fault:

Unable to handle kernel paging request at 00000000005967ac RIP: 
 [<0000000011180220>]
PGD 1944e067 PUD 3c5b8067 PMD 3cee6067 PTE 0
Oops: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 0 
Modules linked in: nfs fscache nfsd exportfs lockd nfs_acl auth_rpcgss loop
autofs4 hidp rfcomm l2cap bluetooth sunrpc ipv6 xfrm_nalgo crypto_api
dm_multipath video sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug
ac parport_pc lp parport joydev ide_cd i2c_i801 e752x_edac i2c_core floppy
edac_mc sg tg3 pcspkr shpchp serio_raw cdrom dm_snapshot dm_zero dm_mirror
dm_mod ata_piix libata aic79xx scsi_transport_spi sd_mod scsi_mod ext3 jbd
uhci_hcd ohci_hcd ehci_hcd
Pid: 7825, comm: crashme Not tainted 2.6.18-89.el5 #1
RIP: 0033:[<0000000011180220>]  [<0000000011180220>]
RSP: 002b:00007fffa0f66428  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000009 RCX: ffffffffffffffff
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000000000a
RBP: 0000000000000007 R08: 00007fffa0f65f20 R09: 0000000000000000
R10: 0000000000000008 R11: 0000000000000206 R12: 0000000000000000
R13: 00007fffa0f66588 R14: 0000000000000000 R15: 0000000000000000
FS:  00002b8c09b53210(0000) GS:ffffffff8039e000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000005967ac CR3: 000000003d2ad000 CR4: 00000000000006e0
Process crashme (pid: 7825, threadinfo ffff81002c05a000, task ffff810016ce30c0)

RIP  [<0000000011180220>]

Haven't we been down this road before?

Dave,
 It reminds me of 213313 or 186925

Jeff

Right you are -- thanks for digging them out.

Those BZ's were both Intel machines (as is this one) where the error_code
does not have the PF_USER bit set on a user-mode address page fault.
The "Oops: 0000" line above is the error_code that gets checked for
in do_page_fault(), where it should just kill the process with a
SIGSEGV:

        /* User mode accesses just cause a SIGSEGV */
        if (error_code & PF_USER) {
            ....

We had asked Intel (Suresh and his replacement?) to look into why
the hardware is behaving that way -- the kernel code is doing the
right thing.
  
 

I see that upstream has this additional piece in do_page_fault() for x86_64:

        /*
         * User-mode registers count as a user access even for any
         * potential system fault or CPU buglet.
         */
        if (user_mode_vm(regs))
                error_code |= PF_USER;

I'm not sure of the commit number because the containing file has changed
since they did the x86/x86_64 code-merge upstream, but here's Linus's
final word:

  http://kerneltrap.org/mailarchive/linux-kernel/2007/9/17/261898

It will fix the problem seen here.  I can whip up a kernel, and let the
powers decide whether it should be included in RHEL5.2.

Here it is:

commit dbe3ed1c078c193be34326728d494c5c4bc115e2
Author: Linus Torvalds <torvalds.org>
Date:   Wed Sep 19 11:37:14 2007 -0700

    x86-64: page faults from user mode are always user faults
   
    Randy Dunlap noticed an interesting "crashme" behaviour on his dual
    Prescott Xeon setup, where he gets page faults with the error code
    having a zero "user" bit, but the register state points back to user
    mode.
   
    This may be a CPU microcode buglet triggered by some strange instruction
    pattern that crashme generates, and loading a microcode update seems to
    possibly have fixed it.
   
    Regardless, we really should trust the register state more than the
    error code, since it's really the register state that determines whether
    we can actually send a signal, or whether we're in kernel mode and need
    to oops/kill the process in the case of a page fault.
   
    Cc: Randy Dunlap <rdunlap>
    Signed-off-by: Linus Torvalds <torvalds>

diff --git a/arch/x86_64/mm/fault.c b/arch/x86_64/mm/fault.c
index 327c9f2..54816ad 100644
--- a/arch/x86_64/mm/fault.c
+++ b/arch/x86_64/mm/fault.c
@@ -374,6 +374,13 @@ asmlinkage void __kprobes do_page_fault(struct pt_regs *regs,
        if (unlikely(in_atomic() || !mm))
                goto bad_area_nosemaphore;

+       /*
+        * User-mode registers count as a user access even for any
+        * potential system fault or CPU buglet.
+        */
+       if (user_mode_vm(regs))
+               error_code |= PF_USER;
+
  again:
        /* When running in the kernel we expect faults to occur only to
         * addresses in user space.  All other faults represent errors in the

I apperantly managed to hang the same test system with the -88 kernel, however
I'm not able to get a crash dump or anything response at all out of it. 

I went ahead and cancelled the job and started a new one with the -90.el5 kernel.

in kernel-2.6.18-90.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5

Confirmed the fix is in the -90 kernel, also the rhts job I started yesterday
with the -90 kernel has completed without any issues.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2008-0314.html