Bug 675997

we are trying to get  a crash file...

Reassigning to kernel. lksctp-tools component is only for the user space sctp tools (sctp_darn, sctp_test, ...)

we have a panic/crash file.
I will attempt to upload it.

The crash file is 297 meg.
The vmlinux is 123 meg
the system map is 2meg

How do I get these files to you ?

Here's the backtrace;

crash> sys
  SYSTEM MAP: /boot/System.map-2.6.32-71.el6.x86_64
DEBUG KERNEL: /usr/lib/debug/lib/modules/2.6.32-71.el6.x86_64/vmlinux (2.6.32-71.el6.x86_64)
    DUMPFILE: vmcore  [PARTIAL DUMP]
        CPUS: 4
        DATE: Mon Feb 14 15:45:09 2011
      UPTIME: 03:08:53
LOAD AVERAGE: 1.10, 1.04, 0.93
       TASKS: 384
    NODENAME: pslin2
     RELEASE: 2.6.32-71.el6.x86_64
     VERSION: #1 SMP Wed Sep 1 01:33:01 EDT 2010
     MACHINE: x86_64  (2793 Mhz)
      MEMORY: 2 GB
       PANIC: "kernel BUG at net/core/skbuff.c:127!"
crash> bt
PID: 0      TASK: ffff88007cd000c0  CPU: 1   COMMAND: "swapper"
 #0 [ffff880022243720] machine_kexec at ffffffff8103695b
 #1 [ffff880022243780] crash_kexec at ffffffff810b8f08
 #2 [ffff880022243850] oops_end at ffffffff814cbbd0
 #3 [ffff880022243880] die at ffffffff8101733b
 #4 [ffff8800222438b0] do_trap at ffffffff814cb4a4
 #5 [ffff880022243910] do_invalid_op at ffffffff81014ee5
 #6 [ffff8800222439b0] invalid_op at ffffffff81013f5b
    [exception RIP: skb_over_panic+93]
    RIP: ffffffff8140411d  RSP: ffff880022243a60  RFLAGS: 00010282
    RAX: 0000000000000089  RBX: 00000000000005d8  RCX: 0000000000005cd3
    RDX: 0000000000000000  RSI: 0000000000000046  RDI: 0000000000000246
    RBP: ffff880022243a80   R8: 000000000001118f   R9: 00000000fffffffa
    R10: 0000000000000002  R11: 0000000000000000  R12: ffff88007aaf4000
    R13: 00000000000005d4  R14: ffff8800799fba80  R15: ffff88007aa1d9a0
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff880022243a88] skb_put at ffffffff8140515c
 #8 [ffff880022243aa8] sctp_packet_transmit at ffffffffa0b30008
 #9 [ffff880022243b48] sctp_packet_transmit_chunk at ffffffffa0b307fe #10 [ffff880022243b78] sctp_outq_flush at ffffffffa0b2553c
#11 [ffff880022243c78] sctp_outq_uncork at ffffffffa0b255da
#12 [ffff880022243c88] sctp_do_sm at ffffffffa0b19456
#13 [ffff880022243e18] sctp_generate_heartbeat_event at ffffffffa0b1a611
#14 [ffff880022243e48] run_timer_softirq at ffffffff8107de07
#15 [ffff880022243ed8] __do_softirq at ffffffff81073bd7
#16 [ffff880022243f48] call_softirq at ffffffff810142cc
#17 [ffff880022243f60] do_softirq at ffffffff81015f35
#18 [ffff880022243f80] irq_exit at ffffffff810739d5
#19 [ffff880022243f90] smp_apic_timer_interrupt at ffffffff814cfa01 #20 [ffff880022243fb0] apic_timer_interrupt at ffffffff81013c93
--- <IRQ stack> ---
#21 [ffff88007cd09e38] apic_timer_interrupt at ffffffff81013c93
    [exception RIP: mwait_idle+113]
    RIP: ffffffff8101bc01  RSP: ffff88007cd09ee8  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: ffff88007cd09ef8  RCX: 0000000000000000
    RDX: 0000000000000000  RSI: ffff88007cd09fd8  RDI: ffffffff81a101e8
    RBP: ffffffff81013c8e   R8: 0000000000000000   R9: 0000000000000001
    R10: 0000000000000000  R11: 0000000100a85957  R12: ffffffff818a1f60
    R13: 0000000000000001  R14: 0000000000000001  R15: ffff88007cd09e68
    ORIG_RAX: ffffffffffffff10  CS: 0010  SS: 0018
#22 [ffff88007cd09f00] cpu_idle at ffffffff81011e96

Red Hat case number  00418965 has been created.

here are the commands we used for the crash tool:


crash /boot/System.map-2.6.32-71.el6.x86_64 /usr/lib/debug/lib/modules/2.6.32-71.el6.x86_64/vmlinux vmcore

vmcore file uploaded to dropbox.redhat.com as file

vmcore.gz.buzz.675997

Created attachment 478951 [details]
sosreport

sosreport of machine

update -

on another pair of machines, there problem is very repeatable.

Based on information from this
ticket ( https://access.redhat.com/support/cases/00418965 ),
I am now re-running the test using a 
2.6.32-71.18.2 RH kernel

The kernel from comment 14 ran successfully all weekend.

I am waiting to hear about the technical details of the issue/fix.

( from the RH ticket/case )
The latest RHEL 6 kernel errata (2.6.32-71.18.2.el6) should contain our official fix for this issue. You can advise your clients to update to this version. This issue was fixed as part of the work that addressed CVE-2010-3432. Here is the CVE notice from Mitre:

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-3432

Here is the public tracking bug from our Bugzilla:

https://bugzilla.redhat.com/show_bug.cgi?id=637675

this ticket can be closed from my point of view.

The reporter of this ticket asked to close this and the underlying support ticket is closed as well.

Closing this as dup of 637675.
thanks,
fbl

*** This bug has been marked as a duplicate of bug 637675 ***