Attachment 302544 Details for Bug 442661 – ibm-defiant crash read failure log

ibm-defiant crash read failure log

ibm-defiant-crash (text/plain), 77.17 KB, created by Qian Cai on 2008-04-16 04:11:59 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Qian Cai

Created: 2008-04-16 04:11:59 UTC

Size: 77.17 KB

patch

obsolete

>
>----------------------------------------------------------
>                        VMCORE SIZE                       
>----------------------------------------------------------
>
>-r-------- 1 root root 896M Apr 15 06:32 /mnt/tests/kernel/distribution/kexec-tools/crasher/ltp-kdump-20070917/results/vmcore
>
>----------------------------------------------------------
>                        READELF                           
>----------------------------------------------------------
>
>ELF Header:
>  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00 
>  Class:                             ELF64
>  Data:                              2's complement, little endian
>  Version:                           1 (current)
>  OS/ABI:                            UNIX - System V
>  ABI Version:                       0
>  Type:                              CORE (Core file)
>  Machine:                           Advanced Micro Devices X86-64
>  Version:                           0x1
>  Entry point address:               0x0
>  Start of program headers:          64 (bytes into file)
>  Start of section headers:          0 (bytes into file)
>  Flags:                             0x0
>  Size of this header:               64 (bytes)
>  Size of program headers:           56 (bytes)
>  Number of program headers:         4
>  Size of section headers:           0 (bytes)
>  Number of section headers:         0
>  Section header string table index: 0
>
>There are no sections in this file.
>
>There are no sections in this file.
>
>Program Headers:
>  Type           Offset             VirtAddr           PhysAddr
>                 FileSiz            MemSiz              Flags  Align
>  NOTE           0x0000000000000120 0x0000000000000000 0x0000000000000000
>                 0x00000000000006a8 0x00000000000006a8         0
>  LOAD           0x00000000000007c8 0xffff810000000000 0x0000000000000000
>                 0x00000000000a0000 0x00000000000a0000  RWE    0
>  LOAD           0x00000000000a07c8 0xffff810000100000 0x0000000000100000
>                 0x0000000001f00000 0x0000000001f00000  RWE    0
>  LOAD           0x0000000001fa07c8 0xffff81000a000000 0x000000000a000000
>                 0x0000000035fcbc00 0x0000000035fcbc00  RWE    0
>
>There is no dynamic section in this file.
>
>There are no relocations in this file.
>
>There are no unwind sections in this file.
>
>No version information found in this file.
>
>Notes at offset 0x00000120 with length 0x000006a8:
>  Owner		Data size	Description
>  CORE		0x00000150	NT_PRSTATUS (prstatus structure)
>  Xen		0x00000020	Unknown note type: (0x01000002)
>  Xen		0x00000048	Unknown note type: (0x01000001)
>  CORE		0x00000150	NT_PRSTATUS (prstatus structure)
>  Xen		0x00000020	Unknown note type: (0x01000002)
>  CORE		0x00000150	NT_PRSTATUS (prstatus structure)
>  Xen		0x00000020	Unknown note type: (0x01000002)
>  CORE		0x00000150	NT_PRSTATUS (prstatus structure)
>  Xen		0x00000020	Unknown note type: (0x01000002)
>
>----------------------------------------------------------
>                        CRASH                             
>----------------------------------------------------------
>
>
>crash 4.0-5.0.2
>Copyright (C) 2002, 2003, 2004, 2005, 2006, 2007, 2008  Red Hat, Inc.
>Copyright (C) 2004, 2005, 2006  IBM Corporation
>Copyright (C) 1999-2006  Hewlett-Packard Co
>Copyright (C) 2005, 2006  Fujitsu Limited
>Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
>Copyright (C) 2005  NEC Corporation
>Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
>Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
>This program is free software, covered by the GNU General Public License,
>and you are welcome to change it and/or distribute copies of it under
>certain conditions.  Enter "help copying" to see the conditions.
>This program has absolutely no warranty.  Enter "help warranty" for details.
> 
>    NOTE: stdin: not a tty
>GNU gdb 6.1
>Copyright 2004 Free Software Foundation, Inc.
>GDB is free software, covered by the GNU General Public License, and you are
>welcome to change it and/or distribute copies of it under certain conditions.
>Type "show copying" to see the conditions.
>There is absolutely no warranty for GDB.  Type "show warranty" for details.
>This GDB was configured as "x86_64-unknown-linux-gnu"...
>
>crash: read error: kernel virtual address: ffff880034204604  type: "array cache limit"
>crash: unable to initialize kmem slab cache subsystem
>
>
>WARNING: cannot access vmalloc'd module memory
>
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>
>crash: cannot read pid_hash node pid_link
>crash: read error: kernel virtual address: ffff8800347050c0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff8800346b7860  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff8800345f3820  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003449f820  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003463c080  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034579040  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034533820  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034530080  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003449d040  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034576860  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003468d820  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034709860  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003467a040  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff8800346000c0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034735100  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003449e080  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034688040  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff8800345797a0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034649040  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003452c7a0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034705820  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003449d7a0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff8800345330c0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003449f0c0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff8800345fc7e0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff88003449e7e0  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034576100  type: "fill_task_struct"
>crash: read error: kernel virtual address: ffff880034529100  type: "fill_task_struct"
>WARNING: active task ffff880029e380c0 on cpu 2 not found in PID hash
>
>      KERNEL: /usr/lib/debug/lib/modules/2.6.18-89.el5xen/vmlinux
>    DUMPFILE: /mnt/tests/kernel/distribution/kexec-tools/crasher/ltp-kdump-20070917/results/vmcore
>        CPUS: 4
>        DATE: Tue Apr 15 06:31:22 2008
>      UPTIME: 00:03:02
>LOAD AVERAGE: 0.16, 0.16, 0.06
>       TASKS: 46
>    NODENAME: ibm-defiant.rhts.boston.redhat.com
>     RELEASE: 2.6.18-89.el5xen
>     VERSION: #1 SMP Tue Apr 8 16:19:32 EDT 2008
>     MACHINE: x86_64  (2660 Mhz)
>      MEMORY: 848.7 MB
>       PANIC: "SysRq : Trigger a crashdump"
>         PID: 5431
>     COMMAND: "master"
>        TASK: ffff880029e380c0  [THREAD_INFO: ffff88001fda0000]
>         CPU: 2
>       STATE: TASK_RUNNING (SYSRQ)
>
>crash> mod
>mod: cannot access vmalloc'd module memory
>crash> mod -S
>mod: cannot access vmalloc'd module memory
>crash> runq
>RUNQUEUES[0]: ffff880001003460
> ACTIVE PRIO_ARRAY: ffff8800010034d8
> EXPIRED PRIO_ARRAY: ffff880001003db8
>RUNQUEUES[1]: ffff880001009860
> ACTIVE PRIO_ARRAY: ffff8800010098d8
> EXPIRED PRIO_ARRAY: ffff88000100a1b8
>RUNQUEUES[2]: ffff88000100fc60
> ACTIVE PRIO_ARRAY: ffff8800010105b8
>  [115] PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
> EXPIRED PRIO_ARRAY: ffff88000100fcd8
>RUNQUEUES[3]: ffff880001016060
> ACTIVE PRIO_ARRAY: ffff8800010160d8
> EXPIRED PRIO_ARRAY: ffff8800010169b8
>crash> foreach bt
>PID: 0      TASK: ffffffff804d6b00  CPU: 0   COMMAND: "swapper"
> #0 [ffffffff805eff88] xen_idle at ffffffff8026c502
> #1 [ffffffff805eff98] cpu_idle at ffffffff8024a982
>
>PID: 0      TASK: ffff880000004040  CPU: 1   COMMAND: "swapper"
> #0 [ffff880000ee3f38] xen_idle at ffffffff8026c502
> #1 [ffff880000ee3f48] cpu_idle at ffffffff8024a982
>
>PID: 0      TASK: ffff880000ec57e0  CPU: 2   COMMAND: "swapper"
> #0 [ffff880000ee7e68] schedule at ffffffff80262cdb
> #1 [ffff880000ee7f48] cpu_idle at ffffffff8024a9a3
>
>PID: 0      TASK: ffff880000ec5080  CPU: 3   COMMAND: "swapper"
> #0 [ffff880000ee9f38] xen_idle at ffffffff8026c502
> #1 [ffff880000ee9f48] cpu_idle at ffffffff8024a982
>
>PID: 1      TASK: ffff8800000047a0  CPU: 3   COMMAND: "init"
> #0 [ffff880000ec1940] schedule at ffffffff80262cdb
> #1 [ffff880000ec1a20] schedule_timeout at ffffffff80263624
> #2 [ffff880000ec1a70] do_select at ffffffff80211e24
> #3 [ffff880000ec1d90] core_sys_select at ffffffff802d3db6
> #4 [ffff880000ec1f20] sys_select at ffffffff80216b19
> #5 [ffff880000ec1f80] system_call at ffffffff80260102
>bt: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>    RIP: 00002acecdec9213  RSP: 00007fffdd3252e8  RFLAGS: 00000246
>    RAX: 0000000000000017  RBX: ffffffff80260102  RCX: ffffffff8026007c
>    RDX: 0000000000000000  RSI: 00007fffdd325410  RDI: 000000000000000b
>    RBP: 0000000000000004   R8: 00007fffdd325490   R9: 3333333333333333
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fffdd325490
>    R13: 0000000000000003  R14: 00007fffdd325410  R15: 00007fffdd325668
>    ORIG_RAX: 0000000000000017  CS: e033  SS: e02b
>
>PID: 2      TASK: ffff880000ec7820  CPU: 0   COMMAND: "migration/0"
> #0 [ffff880000eebdc8] schedule at ffffffff80262cdb
> #1 [ffff880000eebea8] migration_thread at ffffffff80246737
> #2 [ffff880000eebee8] kthread at ffffffff802339c8
> #3 [ffff880000eebf48] kernel_thread at ffffffff80260b24
>
>PID: 3      TASK: ffff880000ec70c0  CPU: 0   COMMAND: "ksoftirqd/0"
> #0 [ffff880000eefdf8] schedule at ffffffff80262cdb
> #1 [ffff880000eefed8] ksoftirqd at ffffffff8028ffc5
> #2 [ffff880000eefee8] kthread at ffffffff802339c8
> #3 [ffff880000eeff48] kernel_thread at ffffffff80260b24
>
>PID: 4      TASK: ffff880000ec9860  CPU: 0   COMMAND: "watchdog/0"
> #0 [ffff880000ef1de8] schedule at ffffffff80262cdb
> #1 [ffff880000ef1ec8] watchdog at ffffffff802b15ef
> #2 [ffff880000ef1ee8] kthread at ffffffff802339c8
> #3 [ffff880000ef1f48] kernel_thread at ffffffff80260b24
>
>PID: 5      TASK: ffff880000ec9100  CPU: 1   COMMAND: "migration/1"
> #0 [ffff880000ef3dc8] schedule at ffffffff80262cdb
> #1 [ffff880000ef3ea8] migration_thread at ffffffff80246737
> #2 [ffff880000ef3ee8] kthread at ffffffff802339c8
> #3 [ffff880000ef3f48] kernel_thread at ffffffff80260b24
>
>PID: 6      TASK: ffff880000ed57a0  CPU: 1   COMMAND: "ksoftirqd/1"
> #0 [ffff880000ef7df8] schedule at ffffffff80262cdb
> #1 [ffff880000ef7ed8] ksoftirqd at ffffffff8028ffc5
> #2 [ffff880000ef7ee8] kthread at ffffffff802339c8
> #3 [ffff880000ef7f48] kernel_thread at ffffffff80260b24
>
>PID: 7      TASK: ffff880000ed5040  CPU: 1   COMMAND: "watchdog/1"
> #0 [ffff880000efdde8] schedule at ffffffff80262cdb
> #1 [ffff880000efdec8] watchdog at ffffffff802b15ef
> #2 [ffff880000efdee8] kthread at ffffffff802339c8
> #3 [ffff880000efdf48] kernel_thread at ffffffff80260b24
>
>PID: 8      TASK: ffff880000ed77e0  CPU: 2   COMMAND: "migration/2"
> #0 [ffff880000041dc8] schedule at ffffffff80262cdb
> #1 [ffff880000041ea8] migration_thread at ffffffff80246737
> #2 [ffff880000041ee8] kthread at ffffffff802339c8
> #3 [ffff880000041f48] kernel_thread at ffffffff80260b24
>
>PID: 9      TASK: ffff880000ed7080  CPU: 2   COMMAND: "ksoftirqd/2"
> #0 [ffff88000006bdf8] schedule at ffffffff80262cdb
> #1 [ffff88000006bed8] ksoftirqd at ffffffff8028ffc5
> #2 [ffff88000006bee8] kthread at ffffffff802339c8
> #3 [ffff88000006bf48] kernel_thread at ffffffff80260b24
>
>PID: 11     TASK: ffff88000004d0c0  CPU: 3   COMMAND: "migration/3"
> #0 [ffff88000006fdc8] schedule at ffffffff80262cdb
> #1 [ffff88000006fea8] migration_thread at ffffffff80246737
> #2 [ffff88000006fee8] kthread at ffffffff802339c8
> #3 [ffff88000006ff48] kernel_thread at ffffffff80260b24
>
>PID: 12     TASK: ffff880000056860  CPU: 3   COMMAND: "ksoftirqd/3"
> #0 [ffff880000073df8] schedule at ffffffff80262cdb
> #1 [ffff880000073ed8] ksoftirqd at ffffffff8028ffc5
> #2 [ffff880000073ee8] kthread at ffffffff802339c8
> #3 [ffff880000073f48] kernel_thread at ffffffff80260b24
>
>PID: 13     TASK: ffff880000056100  CPU: 3   COMMAND: "watchdog/3"
> #0 [ffff88000007dde8] schedule at ffffffff80262cdb
> #1 [ffff88000007dec8] watchdog at ffffffff802b15ef
> #2 [ffff88000007dee8] kthread at ffffffff802339c8
> #3 [ffff88000007df48] kernel_thread at ffffffff80260b24
>
>PID: 1712   TASK: ffff88003072f100  CPU: 3   COMMAND: "kmpathd/3"
> #0 [ffff88002f935d98] schedule at ffffffff80262cdb
> #1 [ffff88002f935e78] worker_thread at ffffffff8024b966
> #2 [ffff88002f935ee8] kthread at ffffffff802339c8
> #3 [ffff88002f935f48] kernel_thread at ffffffff80260b24
>
>PID: 1788   TASK: ffff88003072d100  CPU: 0   COMMAND: "rc"
> #0 [ffff8800313f7e10] schedule at ffffffff80262cdb
> #1 [ffff8800313f7ef0] do_wait at ffffffff80228a83
> #2 [ffff8800313f7f80] system_call at ffffffff80260102
>    RIP: 00002b83523200d5  RSP: 00007fff58e423a0  RFLAGS: 00000246
>    RAX: 000000000000003d  RBX: ffffffff80260102  RCX: ffffffff8026007c
>    RDX: 000000000000000a  RSI: 00007fff58e423d4  RDI: ffffffffffffffff
>    RBP: 0000000010167700   R8: 0000000010137110   R9: 00000000101397d0
>    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000000
>    R13: 0000000000000000  R14: 00000000ffffffff  R15: 00007fff58e423d4
>    ORIG_RAX: 000000000000003d  CS: e033  SS: e02b
>
>PID: 1886   TASK: ffff880030fc5080  CPU: 0   COMMAND: "kondemand/0"
> #0 [ffff8800314c5d98] schedule at ffffffff80262cdb
> #1 [ffff8800314c5e78] worker_thread at ffffffff8024b966
> #2 [ffff8800314c5ee8] kthread at ffffffff802339c8
> #3 [ffff8800314c5f48] kernel_thread at ffffffff80260b24
>
>PID: 2255   TASK: ffff88002e9b2860  CPU: 0   COMMAND: "dhclient"
> #0 [ffff8800329b9940] schedule at ffffffff80262cdb
> #1 [ffff8800329b9a20] schedule_timeout at ffffffff80263624
> #2 [ffff8800329b9a70] do_select at ffffffff80211e24
> #3 [ffff8800329b9d90] core_sys_select at ffffffff802d3db6
> #4 [ffff8800329b9f20] sys_select at ffffffff80216b19
> #5 [ffff8800329b9f80] system_call at ffffffff80260102
>    RIP: 00002b6d7b047213  RSP: 00007fff2ffcd098  RFLAGS: 00000246
>    RAX: 0000000000000017  RBX: ffffffff80260102  RCX: ffffffff8026007c
>    RDX: 00007fff2ffcd130  RSI: 00007fff2ffcd1b0  RDI: 0000000000000006
>    RBP: 0000000000000005   R8: 00007fff2ffcd240   R9: 0000000000000000
>    R10: 00007fff2ffcd0b0  R11: 0000000000000246  R12: 00007fff2ffcd0b0
>    R13: 00007fff2ffcd130  R14: 00007fff2ffcd1b0  R15: 0000000000000001
>    ORIG_RAX: 0000000000000017  CS: e033  SS: e02b
>
>PID: 2337   TASK: ffff88002ea4b040  CPU: 0   COMMAND: "auditd"
> #0 [ffff8800304efc70] schedule at ffffffff80262cdb
> #1 [ffff8800304efd50] schedule_timeout at ffffffff802635b8
> #2 [ffff8800304efda0] do_futex at ffffffff8023fc27
> #3 [ffff8800304eff10] sys_futex at ffffffff8029eeb1
> #4 [ffff8800304eff80] system_call at ffffffff80260102
>    RIP: 00002b9548114496  RSP: 0000000040ec8c70  RFLAGS: 00000246
>    RAX: 00000000000000ca  RBX: ffffffff80260102  RCX: ffffffff8026007c
>    RDX: 0000000000000007  RSI: 0000000000000000  RDI: 00002b9547eed554
>    RBP: 00002b9547eed528   R8: 00002b9547eed528   R9: 0000000000000003
>    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000040ecc000
>    R13: 00002b9547eed440  R14: 0000000000000007  R15: 00002b9547eed520
>    ORIG_RAX: 00000000000000ca  CS: e033  SS: e02b
>
>PID: 2338   TASK: ffff880031be0820  CPU: 0   COMMAND: "audispd"
> #0 [ffff880032f99c70] schedule at ffffffff80262cdb
> #1 [ffff880032f99d50] schedule_timeout at ffffffff802635b8
> #2 [ffff880032f99da0] do_futex at ffffffff8023fc27
> #3 [ffff880032f99f10] sys_futex at ffffffff8029eeb1
> #4 [ffff880032f99f80] system_call at ffffffff80260102
>    RIP: 00002b2b63b89496  RSP: 00007fff47358c70  RFLAGS: 00000246
>    RAX: 00000000000000ca  RBX: ffffffff80260102  RCX: ffffffff8026007c
>    RDX: 0000000000000007  RSI: 0000000000000000  RDI: 00002b2b639623c4
>    RBP: 00002b2b6ba1d300   R8: 00002b2b63962380   R9: 0000000000000003
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00000000000000b5
>    R13: 00007fff47359050  R14: 0000000000000007  R15: 00002b2b6ba1d2f0
>    ORIG_RAX: 00000000000000ca  CS: e033  SS: e02b
>
>PID: 2339   TASK: ffff88002ee35080  CPU: 0   COMMAND: "audispd"
> #0 [ffff88002f8cda60] schedule at ffffffff80262cdb
> #1 [ffff88002f8cdb40] schedule_timeout at ffffffff80263624
> #2 [ffff88002f8cdb90] do_sys_poll at ffffffff80230bd0
> #3 [ffff88002f8cdf60] sys_poll at ffffffff8024dc8c
> #4 [ffff88002f8cdf80] system_call at ffffffff80260102
>    RIP: 00002b2b63e621d6  RSP: 00000000414af100  RFLAGS: 00000202
>    RAX: 0000000000000007  RBX: ffffffff80260102  RCX: ffffffff8026007c
>    RDX: 0000000000004e20  RSI: 0000000000000002  RDI: 00002b2b63962320
>    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000004000001
>    R10: 0000000000000001  R11: 0000000000000202  R12: 00000000414af0f0
>    R13: 0000000000001000  R14: ffffffff8024dc8c  R15: 0000000000001000
>    ORIG_RAX: 0000000000000007  CS: e033  SS: e02b
>
>PID: 2366   TASK: ffff88002f7e5820  CPU: 3   COMMAND: "klogd"
> #0 [ffff880029dd3d90] schedule at ffffffff80262cdb
> #1 [ffff880029dd3e70] do_syslog at ffffffff8028d8da
> #2 [ffff880029dd3ef0] kmsg_read at ffffffff802f2862
> #3 [ffff880029dd3f10] vfs_read at ffffffff8020b88d
> #4 [ffff880029dd3f40] sys_read at ffffffff80211fd9
> #5 [ffff880029dd3f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac690026290  RSP: 00007fff1af69bb8  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000fff  RSI: 00002ac68fd44920  RDI: 0000000000000000
>    RBP: 0000000000000000   R8: 0000000000000000   R9: ffffffff00000000
>    R10: 00007fff1af68d43  R11: 0000000000000246  R12: 0000000000000000
>    R13: 00002ac6902b0d58  R14: 0000000000000000  R15: 00000000902b0d50
>    ORIG_RAX: 0000000000000000  CS: 0033  SS: e02b
>
>PID: 2377   TASK: ffff88002f02c100  CPU: 2   COMMAND: "irqbalance"
> #0 [ffff880029df3de0] schedule at ffffffff80262cdb
> #1 [ffff880029df3ec0] do_nanosleep at ffffffff80263a76
> #2 [ffff880029df3ed0] hrtimer_nanosleep at ffffffff8025c842
> #3 [ffff880029df3f50] sys_nanosleep at ffffffff80256733
> #4 [ffff880029df3f80] tracesys at ffffffff802602f1 (via system_call)
>bt: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>    RIP: 00002b9d3a1534d0  RSP: 00007fff710adfa8  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000009  RSI: 0000000000000000  RDI: 00007fff710adfc0
>    RBP: 0000000000000000   R8: 0000000000000100   R9: 0000000000000004
>    R10: 0000000000000004  R11: 0000000000000246  R12: 000000000000000a
>    R13: 00007fff710ae0d0  R14: 000000003b6ce400  R15: 0000000000000009
>    ORIG_RAX: 0000000000000023  CS: 0033  SS: e02b
>
>PID: 2478   TASK: ffff880030bed040  CPU: 0   COMMAND: "dbus-daemon"
> #0 [ffff880029f67a60] schedule at ffffffff80262cdb
> #1 [ffff880029f67b40] schedule_timeout at ffffffff802635b8
> #2 [ffff880029f67b90] do_sys_poll at ffffffff80230bd0
> #3 [ffff880029f67f60] sys_poll at ffffffff8024dc8c
> #4 [ffff880029f67f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002acc1dbe51d6  RSP: 00007fff8dc17290  RFLAGS: 00000206
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: ffffffffffffffff  RSI: 0000000000000006  RDI: 00007fff8dc174e0
>    RBP: ffffffffffffffff   R8: 0000000000000000   R9: 0000000000000000
>    R10: 000000332e313a00  R11: 0000000000000206  R12: 00007fff8dc174e0
>    R13: 0000000004753510  R14: ffffffff8024dc8c  R15: 0000000004753518
>    ORIG_RAX: 0000000000000007  CS: 0033  SS: e02b
>
>PID: 2479   TASK: ffff88002fc1b7e0  CPU: 0   COMMAND: "dbus-daemon"
> #0 [ffff880029de3a70] schedule at ffffffff80262cdb
> #1 [ffff880029de3b50] schedule_timeout at ffffffff802635b8
> #2 [ffff880029de3ba0] skb_recv_datagram at ffffffff8025724f
> #3 [ffff880029de3c20] netlink_recvmsg at ffffffff8041f418
> #4 [ffff880029de3cc0] sock_recvmsg at ffffffff8023191c
> #5 [ffff880029de3e60] sys_recvfrom at ffffffff8022bf61
> #6 [ffff880029de3f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002acc1d4f50ac  RSP: 0000000041948c80  RFLAGS: 00000206
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000400  RSI: 0000000041948cd0  RDI: 0000000000000005
>    RBP: 0000000000001000   R8: 0000000041948cc0   R9: 0000000041948ccc
>    R10: 0000000000000000  R11: 0000000000000206  R12: 000000004194a000
>    R13: 0000000041948ccc  R14: 0000000041948cc0  R15: 0000000041948cd0
>    ORIG_RAX: 000000000000002d  CS: 0033  SS: e02b
>
>PID: 2493   TASK: ffff88003072d860  CPU: 1   COMMAND: "sdpd"
> #0 [ffff8800299718f0] schedule at ffffffff80262cdb
> #1 [ffff8800299719d0] schedule_timeout at ffffffff80263624
> #2 [ffff880029971a20] do_select at ffffffff80211e24
> #3 [ffff880029971d40] core_sys_select at ffffffff802d3db6
> #4 [ffff880029971ed0] sys_pselect7 at ffffffff802d3f73
> #5 [ffff880029971f60] sys_pselect6 at ffffffff802d40c3
> #6 [ffff880029971f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b611f1cd33e  RSP: 00007fff8bfdf8f0  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fff8bfdfa40  RDI: 0000000000000006
>    RBP: 00007fff8bfdfd48   R8: 0000000000000000   R9: 00007fff8bfdf990
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fff8bfdf990
>    R13: 0000000000000008  R14: ffffffff802d40c3  R15: 00007fff8bfdfa40
>    ORIG_RAX: 000000000000010e  CS: 0033  SS: e02b
>
>PID: 2580   TASK: ffff880029ea30c0  CPU: 3   COMMAND: "hidd"
> #0 [ffff88002945ba00] schedule at ffffffff80262cdb
> #1 [ffff88002945bae0] schedule_timeout at ffffffff802635b8
> #2 [ffff88002945bb30] do_sys_poll at ffffffff80230bd0
> #3 [ffff88002945bf00] sys_ppoll at ffffffff802d41c1
> #4 [ffff88002945bf80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002af977045297  RSP: 00007fff343e9560  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 0000000000000002  RDI: 00007fff343ea0f0
>    RBP: 00007fff343ea0f0   R8: 0000000000000008   R9: 0000000000000000
>    R10: 00007fff343ea070  R11: 0000000000000246  R12: 0000000000000002
>    R13: 00007fff343ea070  R14: 00007fff343ea070  R15: 0000000000000000
>    ORIG_RAX: 000000000000010f  CS: 0033  SS: e02b
>
>PID: 2598   TASK: ffff88002984b7a0  CPU: 0   COMMAND: "automount"
> #0 [ffff880029a75d80] schedule at ffffffff80262cdb
> #1 [ffff880029a75e60] schedule_timeout at ffffffff802635b8
> #2 [ffff880029a75eb0] sys_rt_sigtimedwait at ffffffff802957ce
> #3 [ffff880029a75f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b9ed9a67ba8  RSP: 00007fffd14ac528  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 00007fffd14ac570
>    RBP: 0000000000000000   R8: 00002b9ed982c380   R9: 0000000000000080
>    R10: 0000000000000008  R11: 0000000000000246  R12: 0000000000000000
>    R13: 00007fffd14ac570  R14: 00007fffd14ac5fc  R15: 00002b9ed982c360
>    ORIG_RAX: 0000000000000080  CS: 0033  SS: e02b
>
>PID: 2599   TASK: ffff88003072f860  CPU: 0   COMMAND: "automount"
> #0 [ffff88002944fc70] schedule at ffffffff80262cdb
> #1 [ffff88002944fd50] schedule_timeout at ffffffff80263624
> #2 [ffff88002944fda0] do_futex at ffffffff8023fc27
> #3 [ffff88002944ff10] sys_futex at ffffffff8029eeb1
> #4 [ffff88002944ff80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b9ed9a64687  RSP: 0000000041bd3080  RFLAGS: 00000202
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000009  RSI: 0000000000000000  RDI: 00002b9ed982c0a4
>    RBP: 0000000000001000   R8: 00002b9ed982c060   R9: 0000000000000004
>    R10: 0000000041bd3098  R11: 0000000000000202  R12: ffffffffffffff92
>    R13: 0000000041bd3100  R14: 0000000000000009  R15: 00002b9ef8c7b1d0
>    ORIG_RAX: 00000000000000ca  CS: 0033  SS: e02b
>
>PID: 2600   TASK: ffff88002f02c860  CPU: 0   COMMAND: "automount"
> #0 [ffff880029a89c70] schedule at ffffffff80262cdb
> #1 [ffff880029a89d50] schedule_timeout at ffffffff80263624
> #2 [ffff880029a89da0] do_futex at ffffffff8023fc27
> #3 [ffff880029a89f10] sys_futex at ffffffff8029eeb1
> #4 [ffff880029a89f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b9ed9a64687  RSP: 0000000041be4070  RFLAGS: 00000202
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000119  RSI: 0000000000000000  RDI: 00002b9ed982bfc4
>    RBP: 00002b9ed982bbe0   R8: 00002b9ed982bf80   R9: 000000000000008c
>    R10: 0000000041be4088  R11: 0000000000000202  R12: ffffffffffffff92
>    R13: 0000000041be40f0  R14: 0000000000000119  R15: 00002b9ed982bbe0
>    ORIG_RAX: 00000000000000ca  CS: 0033  SS: e02b
>
>PID: 2629   TASK: ffff88002f4677a0  CPU: 0   COMMAND: "sshd"
> #0 [ffff880029681940] schedule at ffffffff80262cdb
> #1 [ffff880029681a20] schedule_timeout at ffffffff802635b8
> #2 [ffff880029681a70] do_select at ffffffff80211e24
> #3 [ffff880029681d90] core_sys_select at ffffffff802d3db6
> #4 [ffff880029681f20] sys_select at ffffffff80216a87
> #5 [ffff880029681f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b171eb42213  RSP: 00007fff8e8e6508  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00002b172309ab80  RDI: 0000000000000004
>    RBP: 0000000000000000   R8: 0000000000000000   R9: 000000000000000a
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fff8e8e6af0
>    R13: 00002b172309ab80  R14: 0000000000000004  R15: 0000000000000008
>    ORIG_RAX: 0000000000000017  CS: 0033  SS: e02b
>
>PID: 2690   TASK: ffff880030fc57e0  CPU: 1   COMMAND: "gpm"
> #0 [ffff8800285b3940] schedule at ffffffff80262cdb
> #1 [ffff8800285b3a20] schedule_timeout at ffffffff80263624
> #2 [ffff8800285b3a70] do_select at ffffffff80211e24
> #3 [ffff8800285b3d90] core_sys_select at ffffffff802d3db6
> #4 [ffff8800285b3f20] sys_select at ffffffff80216b19
> #5 [ffff8800285b3f80] tracesys at ffffffff802602f1 (via system_call)
>bt: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>    RIP: 00002aad4ec4e213  RSP: 00007fff5c3c2d58  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fff5c3c2ed0  RDI: 0000000000000006
>    RBP: 0000000000616848   R8: 00007fff5c3c2f90   R9: 000000007fffffff
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fff5c3c30f0
>    R13: 0000000000000000  R14: 20c49ba5e353f7cf  R15: 0000000000000001
>    ORIG_RAX: 0000000000000017  CS: 0033  SS: e02b
>
>PID: 2737   TASK: ffff88002936c0c0  CPU: 0   COMMAND: "xfs"
> #0 [ffff8800280cb940] schedule at ffffffff80262cdb
> #1 [ffff8800280cba20] schedule_timeout at ffffffff80263624
> #2 [ffff8800280cba70] do_select at ffffffff80211e24
> #3 [ffff8800280cbd90] core_sys_select at ffffffff802d3db6
> #4 [ffff8800280cbf20] sys_select at ffffffff80216b19
> #5 [ffff8800280cbf80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b3b4f50b213  RSP: 00007fff5bd1a7a8  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 000000000061d0c0  RDI: 0000000000000080
>    RBP: 00007fff5bd1a900   R8: 00007fff5bd1a8b0   R9: 00007fff5bd1a730
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00007fff5bd1a900
>    R13: 0000011951a1751c  R14: 00007fff5bd1a970  R15: 00002b3b4efa9bc0
>    ORIG_RAX: 0000000000000017  CS: 0033  SS: e02b
>
>PID: 2762   TASK: ffff880029b47860  CPU: 0   COMMAND: "atd"
> #0 [ffff880028819de0] schedule at ffffffff80262cdb
> #1 [ffff880028819ec0] do_nanosleep at ffffffff80263a76
> #2 [ffff880028819ed0] hrtimer_nanosleep at ffffffff8025c842
> #3 [ffff880028819f50] sys_nanosleep at ffffffff80256733
> #4 [ffff880028819f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac8565434d0  RSP: 00007fff55046a28  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fff55046bd0  RDI: 00007fff55046bd0
>    RBP: 00000000ffffffff   R8: 0000000000000000   R9: 00007fff55046a30
>    R10: 0000000000000008  R11: 0000000000000246  R12: 00007fff55046b50
>    R13: 00007fff55046ad0  R14: 0000000000000000  R15: 000000000000012c
>    ORIG_RAX: 0000000000000023  CS: 0033  SS: e02b
>
>PID: 2786   TASK: ffff880031ede860  CPU: 0   COMMAND: "avahi-daemon"
> #0 [ffff880028835a60] schedule at ffffffff80262cdb
> #1 [ffff880028835b40] schedule_timeout at ffffffff80263624
> #2 [ffff880028835b90] do_sys_poll at ffffffff80230bd0
> #3 [ffff880028835f60] sys_poll at ffffffff8024dc8c
> #4 [ffff880028835f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b209308e19f  RSP: 00007fff18ba54f0  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000107  RSI: 0000000000000009  RDI: 00000000187638a0
>    RBP: 0000000000000000   R8: 00002b2091f3b620   R9: 0000000000000000
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00000000187638a0
>    R13: 0000000018747d40  R14: ffffffff8024dc8c  R15: 00007fff18ba5580
>    ORIG_RAX: 0000000000000007  CS: 0033  SS: e02b
>
>PID: 2797   TASK: ffff88002f7e50c0  CPU: 0   COMMAND: "hald"
> #0 [ffff880028675a60] schedule at ffffffff80262cdb
> #1 [ffff880028675b40] schedule_timeout at ffffffff80263624
> #2 [ffff880028675b90] do_sys_poll at ffffffff80230bd0
> #3 [ffff880028675f60] sys_poll at ffffffff8024dc8c
> #4 [ffff880028675f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b1b0497219f  RSP: 00007fffa6fb3830  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000007530  RSI: 000000000000000a  RDI: 0000000007d56f40
>    RBP: 0000000000000000   R8: 000000000000000a   R9: 0000000000000000
>    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000007d56f40
>    R13: 0000000007d56f40  R14: ffffffff8024dc8c  R15: 00002b1b04972170
>    ORIG_RAX: 0000000000000007  CS: 0033  SS: e02b
>
>PID: 2809   TASK: ffff880029ab27e0  CPU: 0   COMMAND: "hald-addon-stor"
> #0 [ffff8800274edde0] schedule at ffffffff80262cdb
> #1 [ffff8800274edec0] do_nanosleep at ffffffff80263a76
> #2 [ffff8800274eded0] hrtimer_nanosleep at ffffffff8025c842
> #3 [ffff8800274edf50] sys_nanosleep at ffffffff80256733
> #4 [ffff8800274edf80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b658a0c24d0  RSP: 00007fff210dacd8  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fff210dae80  RDI: 00007fff210dae80
>    RBP: 00000000ffffffff   R8: 0000000000000000   R9: 00007fff210dace0
>    R10: 0000000000000008  R11: 0000000000000246  R12: 00007fff210dae00
>    R13: 00007fff210dad80  R14: 0000000000000000  R15: 0000000000000002
>    ORIG_RAX: 0000000000000023  CS: 0033  SS: e02b
>
>PID: 2811   TASK: ffff88002860e7a0  CPU: 3   COMMAND: "hald-addon-stor"
> #0 [ffff880027475de0] schedule at ffffffff80262cdb
> #1 [ffff880027475ec0] do_nanosleep at ffffffff80263a76
> #2 [ffff880027475ed0] hrtimer_nanosleep at ffffffff8025c842
> #3 [ffff880027475f50] sys_nanosleep at ffffffff80256733
> #4 [ffff880027475f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b3ee050d4d0  RSP: 00007fffcac8ecb8  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fffcac8ee60  RDI: 00007fffcac8ee60
>    RBP: 00000000ffffffff   R8: 0000000000000000   R9: 00007fffcac8ecc0
>    R10: 0000000000000008  R11: 0000000000000246  R12: 00007fffcac8ede0
>    R13: 00007fffcac8ed60  R14: 0000000000000000  R15: 0000000000000002
>    ORIG_RAX: 0000000000000023  CS: 0033  SS: e02b
>
>PID: 2835   TASK: ffff88002984b040  CPU: 0   COMMAND: "yum-updatesd"
> #0 [ffff880026dc1a60] schedule at ffffffff80262cdb
> #1 [ffff880026dc1b40] schedule_timeout at ffffffff80263624
> #2 [ffff880026dc1b90] do_sys_poll at ffffffff80230bd0
> #3 [ffff880026dc1f60] sys_poll at ffffffff8024dc8c
> #4 [ffff880026dc1f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b38b5a5819f  RSP: 00007ffff5f0c230  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 000000000036e7bc  RSI: 0000000000000002  RDI: 000000000b95b0f0
>    RBP: 0000000000000000   R8: 0000000000000002   R9: 000000000000010c
>    R10: 0000000000000000  R11: 0000000000000246  R12: 000000000b95b0f0
>    R13: 000000000b95b0f0  R14: ffffffff8024dc8c  R15: 00002b38b5a58170
>    ORIG_RAX: 0000000000000007  CS: 0033  SS: e02b
>
>PID: 4591   TASK: ffff880024e4c820  CPU: 2   COMMAND: "rhts-test-runne"
> #0 [ffff880026583e10] schedule at ffffffff80262cdb
> #1 [ffff880026583ef0] do_wait at ffffffff80228a83
> #2 [ffff880026583f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b59b65600d5  RSP: 00007ffff4c00c20  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 000000000000000a  RSI: 00007ffff4c00c54  RDI: ffffffffffffffff
>    RBP: 000000000b5e3900   R8: 000000000b5e4650   R9: 000000000b5db338
>    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000000
>    R13: 0000000000000000  R14: 00000000ffffffff  R15: 00007ffff4c00c54
>    ORIG_RAX: 000000000000003d  CS: 0033  SS: e02b
>
>PID: 4625   TASK: ffff88002603f7e0  CPU: 1   COMMAND: "make"
> #0 [ffff88001e8f9e10] schedule at ffffffff80262cdb
> #1 [ffff88001e8f9ef0] do_wait at ffffffff80228a83
> #2 [ffff88001e8f9f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b9f8559e034  RSP: 00007fff257bd530  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fff257bd564  RDI: ffffffffffffffff
>    RBP: 0000000000000000   R8: 0000000000000000   R9: 00002b9f8561ca00
>    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000001
>    R13: 0000000000000001  R14: 00000000013c3080  R15: 0000000000000000
>    ORIG_RAX: 000000000000003d  CS: 0033  SS: e02b
>
>PID: 4928   TASK: ffff880024eb0040  CPU: 1   COMMAND: "master"
> #0 [ffff88001e581e10] schedule at ffffffff80262cdb
> #1 [ffff88001e581ef0] do_wait at ffffffff80228a83
> #2 [ffff88001e581f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac1894ba0d5  RSP: 00007fff21ca9870  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 00007fff21ca98a4  RDI: ffffffffffffffff
>    RBP: 0000000014cfc200   R8: 0000000014d061a0   R9: 0000000014cf0470
>    R10: 0000000000000000  R11: 0000000000000246  R12: 0000000000000000
>    R13: 0000000000000000  R14: 00000000ffffffff  R15: 00007fff21ca98a4
>    ORIG_RAX: 000000000000003d  CS: 0033  SS: e02b
>
>PID: 5430   TASK: ffff88002695b7e0  CPU: 3   COMMAND: "sleep"
> #0 [ffff88001df8bde0] schedule at ffffffff80262cdb
> #1 [ffff88001df8bec0] do_nanosleep at ffffffff80263a76
> #2 [ffff88001df8bed0] hrtimer_nanosleep at ffffffff8025c842
> #3 [ffff88001df8bf50] sys_nanosleep at ffffffff80256733
> #4 [ffff88001df8bf80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002b61804484d0  RSP: 00007fff2a913038  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 00007fff2a913040
>    RBP: 00007fff2a913100   R8: 3ff0000000000000   R9: 0000000000000000
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00002b6180704690
>    R13: 0000000000000001  R14: 0000000000000000  R15: 0000000000000001
>    ORIG_RAX: 0000000000000023  CS: 0033  SS: e02b
>
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
> #0 [ffff88001fda1df0] crash_kexec at ffffffff802a5183
> #1 [ffff88001fda1eb0] sysrq_handle_crashdump at ffffffff8038d62f
> #2 [ffff88001fda1ec0] __handle_sysrq at ffffffff8038d42e
> #3 [ffff88001fda1f00] write_sysrq_trigger at ffffffff802f30cb
> #4 [ffff88001fda1f10] vfs_write at ffffffff80216c4a
> #5 [ffff88001fda1f40] sys_write at ffffffff80217482
> #6 [ffff88001fda1f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac1894e6310  RSP: 00007fff21ca9e68  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000002  RSI: 00002ac18cd4c000  RDI: 0000000000000001
>    RBP: 0000000000000002   R8: 00000000ffffffff   R9: 00002ac189776e10
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00002ac189770760
>    R13: 00002ac18cd4c000  R14: 0000000000000002  R15: 0000000000000000
>    ORIG_RAX: 0000000000000001  CS: 0033  SS: e02b
>crash> foreach files
>PID: 0      TASK: ffffffff804d6b00  CPU: 0   COMMAND: "swapper"
>ROOT: /    CWD: /
>No open files
>
>PID: 0      TASK: ffff880000004040  CPU: 1   COMMAND: "swapper"
>ROOT: /    CWD: /
>No open files
>
>PID: 0      TASK: ffff880000ec57e0  CPU: 2   COMMAND: "swapper"
>ROOT: /    CWD: /
>No open files
>
>PID: 0      TASK: ffff880000ec5080  CPU: 3   COMMAND: "swapper"
>ROOT: /    CWD: /
>No open files
>
>PID: 1      TASK: ffff8800000047a0  CPU: 3   COMMAND: "init"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
> 10 ffff8800336de0c0 ffff8800338833d8 ffff880032e936e8 FIFO /dev/initctl
>
>PID: 2      TASK: ffff880000ec7820  CPU: 0   COMMAND: "migration/0"
>ROOT: /    CWD: /
>No open files
>
>PID: 3      TASK: ffff880000ec70c0  CPU: 0   COMMAND: "ksoftirqd/0"
>ROOT: /    CWD: /
>No open files
>
>PID: 4      TASK: ffff880000ec9860  CPU: 0   COMMAND: "watchdog/0"
>ROOT: /    CWD: /
>No open files
>
>PID: 5      TASK: ffff880000ec9100  CPU: 1   COMMAND: "migration/1"
>ROOT: /    CWD: /
>No open files
>
>PID: 6      TASK: ffff880000ed57a0  CPU: 1   COMMAND: "ksoftirqd/1"
>ROOT: /    CWD: /
>No open files
>
>PID: 7      TASK: ffff880000ed5040  CPU: 1   COMMAND: "watchdog/1"
>ROOT: /    CWD: /
>No open files
>
>PID: 8      TASK: ffff880000ed77e0  CPU: 2   COMMAND: "migration/2"
>ROOT: /    CWD: /
>No open files
>
>PID: 9      TASK: ffff880000ed7080  CPU: 2   COMMAND: "ksoftirqd/2"
>ROOT: /    CWD: /
>No open files
>
>PID: 11     TASK: ffff88000004d0c0  CPU: 3   COMMAND: "migration/3"
>ROOT: /    CWD: /
>No open files
>
>PID: 12     TASK: ffff880000056860  CPU: 3   COMMAND: "ksoftirqd/3"
>ROOT: /    CWD: /
>No open files
>
>PID: 13     TASK: ffff880000056100  CPU: 3   COMMAND: "watchdog/3"
>ROOT: /    CWD: /
>No open files
>
>PID: 1712   TASK: ffff88003072f100  CPU: 3   COMMAND: "kmpathd/3"
>ROOT: /    CWD: /
>No open files
>
>PID: 1788   TASK: ffff88003072d100  CPU: 0   COMMAND: "rc"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  1 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  2 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>255 ffff8800338da580 ffff88002f9bfc48 ffff8800322a9d30 REG  /etc/rc.d/rc
>
>PID: 1886   TASK: ffff880030fc5080  CPU: 0   COMMAND: "kondemand/0"
>ROOT: /    CWD: /
>No open files
>
>PID: 2255   TASK: ffff88002e9b2860  CPU: 0   COMMAND: "dhclient"
>ROOT: /    CWD: /etc/sysconfig/network-scripts
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff8800322fa380 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff880032985a80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff88002f9c20c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff88003405fa98  type: "fill_dentry_cache"
>
>PID: 2337   TASK: ffff88002ea4b040  CPU: 0   COMMAND: "auditd"
>ROOT: /    CWD: /
>files: read error: kernel virtual address: ffff88003464a3c0  type: "files_struct buffer"
>
>PID: 2338   TASK: ffff880031be0820  CPU: 0   COMMAND: "audispd"
>ROOT: /    CWD: /
>files: read error: kernel virtual address: ffff8800346cd980  type: "files_struct buffer"
>
>PID: 2339   TASK: ffff88002ee35080  CPU: 0   COMMAND: "audispd"
>ROOT: /    CWD: /
>files: read error: kernel virtual address: ffff8800346cd980  type: "files_struct buffer"
>
>PID: 2366   TASK: ffff88002f7e5820  CPU: 3   COMMAND: "klogd"
>ROOT: /    CWD: /
>files: read error: kernel virtual address: ffff8800346bcc80  type: "files_struct buffer"
>
>PID: 2377   TASK: ffff88002f02c100  CPU: 2   COMMAND: "irqbalance"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002eabedc0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff88002eabedc0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff88002eabedc0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>
>PID: 2478   TASK: ffff880030bed040  CPU: 0   COMMAND: "dbus-daemon"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff8800338de4c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff8800338de4c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff8800338de4c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff880034436280  type: "fill_file_cache"
>
>PID: 2479   TASK: ffff88002fc1b7e0  CPU: 0   COMMAND: "dbus-daemon"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff8800338de4c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff8800338de4c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff8800338de4c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff880034436280  type: "fill_file_cache"
>
>PID: 2493   TASK: ffff88003072d860  CPU: 1   COMMAND: "sdpd"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff880032968a80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff880032968a80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff880032968a80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff8800344a24b0  type: "fill_dentry_cache"
>
>PID: 2580   TASK: ffff880029ea30c0  CPU: 3   COMMAND: "hidd"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff880032445180 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff880032445180 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff880032445180 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff880034430380  type: "fill_file_cache"
>
>PID: 2598   TASK: ffff88002984b7a0  CPU: 0   COMMAND: "automount"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff88003470e150  type: "fill_dentry_cache"
>
>PID: 2599   TASK: ffff88003072f860  CPU: 0   COMMAND: "automount"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff88003470e150  type: "fill_dentry_cache"
>
>PID: 2600   TASK: ffff88002f02c860  CPU: 0   COMMAND: "automount"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff88002f27e6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff88003470e150  type: "fill_dentry_cache"
>
>PID: 2629   TASK: ffff88002f4677a0  CPU: 0   COMMAND: "sshd"
>ROOT: /    CWD: /
>files: read error: kernel virtual address: ffff8800345f8680  type: "files_struct buffer"
>
>PID: 2690   TASK: ffff880030fc57e0  CPU: 1   COMMAND: "gpm"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  1 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  2 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>files: read error: kernel virtual address: ffff880034740ac0  type: "fill_file_cache"
>
>PID: 2737   TASK: ffff88002936c0c0  CPU: 0   COMMAND: "xfs"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff880032e33bc0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff880032e33bc0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff880032e33bc0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff8800344a24b0  type: "fill_dentry_cache"
>
>PID: 2762   TASK: ffff880029b47860  CPU: 0   COMMAND: "atd"
>ROOT: /    CWD: /var/spool/at
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88003369a6c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff880034436c80  type: "fill_file_cache"
>
>PID: 2786   TASK: ffff880031ede860  CPU: 0   COMMAND: "avahi-daemon"
>ROOT: /etc/avahi    CWD: /etc/avahi
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002fac60c0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff88002ffcaec0 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff8800316b4080 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff8800344a24b0  type: "fill_dentry_cache"
>
>PID: 2797   TASK: ffff88002f7e50c0  CPU: 0   COMMAND: "hald"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>files: read error: kernel virtual address: ffff8800347d1c80  type: "fill_file_cache"
>
>PID: 2809   TASK: ffff880029ab27e0  CPU: 0   COMMAND: "hald-addon-stor"
>files: read error: kernel virtual address: ffff8800346da150  type: "fill_dentry_cache"
>
>PID: 2811   TASK: ffff88002860e7a0  CPU: 3   COMMAND: "hald-addon-stor"
>files: read error: kernel virtual address: ffff8800346da150  type: "fill_dentry_cache"
>
>PID: 2835   TASK: ffff88002984b040  CPU: 0   COMMAND: "yum-updatesd"
>ROOT: /    CWD: /
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff8800338dac80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  1 ffff8800338dac80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>  2 ffff8800338dac80 ffff8800327b6c48 ffff8800338e1768 CHR  /dev/null
>files: read error: kernel virtual address: ffff8800344a24b0  type: "fill_dentry_cache"
>
>PID: 4591   TASK: ffff880024e4c820  CPU: 2   COMMAND: "rhts-test-runne"
>ROOT: /    CWD: /mnt/tests/kernel/distribution/kexec-tools/crasher
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  1 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  2 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>files: read error: kernel virtual address: ffff8800346da150  type: "fill_dentry_cache"
>
>PID: 4625   TASK: ffff88002603f7e0  CPU: 1   COMMAND: "make"
>ROOT: /    CWD: /mnt/tests/kernel/distribution/kexec-tools/crasher
>files: read error: kernel virtual address: ffff8800347d7540  type: "fdtable buffer"
>
>PID: 4928   TASK: ffff880024eb0040  CPU: 1   COMMAND: "master"
>ROOT: /    CWD: /mnt/tests/kernel/distribution/kexec-tools/crasher/ltp-kdump-20070917
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  1 ffff8800278990c0 ffff880028536df8 ffff88002756a700 REG  /tmp/TESTOUT.log
>  2 ffff8800278990c0 ffff880028536df8 ffff88002756a700 REG  /tmp/TESTOUT.log
>255 ffff88001d4070c0 ffff88002263c588 ffff88002268ed30 REG  /mnt/tests/kernel/distribution/kexec-tools/crasher/ltp-kdump-20070917/master
>
>PID: 5430   TASK: ffff88002695b7e0  CPU: 3   COMMAND: "sleep"
>ROOT: /    CWD: /mnt/tests/kernel/distribution/kexec-tools/crasher
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  1 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  2 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
>ROOT: /    CWD: /mnt/tests/kernel/distribution/kexec-tools/crasher/ltp-kdump-20070917
> FD       FILE            DENTRY           INODE       TYPE PATH
>  0 ffff88002f953480 ffff8800338e2810 ffff8800338e1478 CHR  /dev/console
>  1 ffff88002656d9c0 ffff880020af4660 ffff88002faa0cf8 REG  /proc/sysrq-trigger
>  2 ffff8800278990c0 ffff880028536df8 ffff88002756a700 REG  /tmp/TESTOUT.log
> 10 ffff8800278990c0 ffff880028536df8 ffff88002756a700 REG  /tmp/TESTOUT.log
>files: read error: kernel virtual address: ffff88003463d7c0  type: "fill_file_cache"
>crash> mount
>mount: read error: kernel virtual address: ffff880034667128  type: "list entry"
>crash> mount -f 
>mount: read error: kernel virtual address: ffff880034667128  type: "list entry"
>crash> mount -i
>mount: read error: kernel virtual address: ffff880034667128  type: "list entry"
>crash> vm
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
>       MM               PGD          RSS    TOTAL_VM
>ffff88002f653800  ffff880024ddc000   956k    65996k 
>      VMA           START       END     FLAGS FILE
>ffff880024f59330     400000     4b2000   1875 /bin/bash
>vm: read error: kernel virtual address: ffff8800340bb3d8  type: "fill_vma_cache"
>crash> ascii
>
>      0    1   2   3   4   5   6   7
>    +-------------------------------
>  0 | NUL DLE  SP  0   @   P   '   p
>  1 | SOH DC1  !   1   A   Q   a   q
>  2 | STX DC2  "   2   B   R   b   r
>  3 | ETX DC3  #   3   C   S   c   s
>  4 | EOT DC4  $   4   D   T   d   t
>  5 | ENQ NAK  %   5   E   U   e   u
>  6 | ACK SYN  &   6   F   V   f   v
>  7 | BEL ETB  `   7   G   W   g   w
>  8 |  BS CAN  (   8   H   X   h   x
>  9 |  HT  EM  )   9   I   Y   i   y
>  A |  LF SUB  *   :   J   Z   j   z
>  B |  VT ESC  +   ;   K   [   k   {
>  C |  FF  FS  ,   <   L   \   l   |
>  D |  CR  GS  _   =   M   ]   m   }
>  E |  SO  RS  .   >   N   ^   n   ~
>  F |  SI  US  /   ?   O   -   o  DEL
>
>crash> net
>   NET_DEVICE     NAME   IP ADDRESS(ES)
>ffffffff80521d00  lo     127.0.0.1
>ffff880030c99000  eth0   10.12.4.48
>ffff880030df8000  eth1   
>ffff88003099a000  sit0   
>crash> set
>    PID: 5431
>COMMAND: "master"
>   TASK: ffff880029e380c0  [THREAD_INFO: ffff88001fda0000]
>    CPU: 2
>  STATE: TASK_RUNNING (SYSRQ)
>crash> set -v
>        scroll: on (none)
>         radix: 10 (decimal)
>       refresh: on
>     print_max: 256
>       console: (not assigned)
>         debug: 0
>          core: off
>          hash: on
>        silent: off
>          edit: vi
>      namelist: /usr/lib/debug/lib/modules/2.6.18-89.el5xen/vmlinux
>      dumpfile: /mnt/tests/kernel/distribution/kexec-tools/crasher/ltp-kdump-20070917/results/vmcore
>        unwind: off
> zero_excluded: off
>crash> bt
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
> #0 [ffff88001fda1df0] crash_kexec at ffffffff802a5183
> #1 [ffff88001fda1eb0] sysrq_handle_crashdump at ffffffff8038d62f
> #2 [ffff88001fda1ec0] __handle_sysrq at ffffffff8038d42e
> #3 [ffff88001fda1f00] write_sysrq_trigger at ffffffff802f30cb
> #4 [ffff88001fda1f10] vfs_write at ffffffff80216c4a
> #5 [ffff88001fda1f40] sys_write at ffffffff80217482
> #6 [ffff88001fda1f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac1894e6310  RSP: 00007fff21ca9e68  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000002  RSI: 00002ac18cd4c000  RDI: 0000000000000001
>    RBP: 0000000000000002   R8: 00000000ffffffff   R9: 00002ac189776e10
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00002ac189770760
>    R13: 00002ac18cd4c000  R14: 0000000000000002  R15: 0000000000000000
>    ORIG_RAX: 0000000000000001  CS: 0033  SS: e02b
>crash> bt -a
>PID: 0      TASK: ffffffff804d6b00  CPU: 0   COMMAND: "swapper"
> #0 [ffffffff805eff88] xen_idle at ffffffff8026c502
> #1 [ffffffff805eff98] cpu_idle at ffffffff8024a982
>
>PID: 0      TASK: ffff880000004040  CPU: 1   COMMAND: "swapper"
> #0 [ffff880000ee3f38] xen_idle at ffffffff8026c502
> #1 [ffff880000ee3f48] cpu_idle at ffffffff8024a982
>
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
> #0 [ffff88001fda1df0] crash_kexec at ffffffff802a5183
> #1 [ffff88001fda1eb0] sysrq_handle_crashdump at ffffffff8038d62f
> #2 [ffff88001fda1ec0] __handle_sysrq at ffffffff8038d42e
> #3 [ffff88001fda1f00] write_sysrq_trigger at ffffffff802f30cb
> #4 [ffff88001fda1f10] vfs_write at ffffffff80216c4a
> #5 [ffff88001fda1f40] sys_write at ffffffff80217482
> #6 [ffff88001fda1f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac1894e6310  RSP: 00007fff21ca9e68  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000002  RSI: 00002ac18cd4c000  RDI: 0000000000000001
>    RBP: 0000000000000002   R8: 00000000ffffffff   R9: 00002ac189776e10
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00002ac189770760
>    R13: 00002ac18cd4c000  R14: 0000000000000002  R15: 0000000000000000
>    ORIG_RAX: 0000000000000001  CS: 0033  SS: e02b
>
>PID: 0      TASK: ffff880000ec5080  CPU: 3   COMMAND: "swapper"
> #0 [ffff880000ee9f38] xen_idle at ffffffff8026c502
> #1 [ffff880000ee9f48] cpu_idle at ffffffff8024a982
>crash> bt -f
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
> #0 [ffff88001fda1df0] crash_kexec at ffffffff802a5183
>    ffff88001fda1df8: 0000000000000006 0000000000000000 
>    ffff88001fda1e08: 0000000000000000 0000000000000063 
>    ffff88001fda1e18: 0000000000000000 ffffffff8051dba0 
>    ffff88001fda1e28: 0000000000000000 0000000000000001 
>    ffff88001fda1e38: 0000000000005853 0000000000000014 
>    ffff88001fda1e48: 0000000000000000 0000000000000000 
>    ffff88001fda1e58: 0000000000000000 0000000000000000 
>    ffff88001fda1e68: 0000000000000000 0000000000000000 
>    ffff88001fda1e78: ffffffff802a5162 000000000000e033 
>    ffff88001fda1e88: 0000000000000246 ffff88001fda1df8 
>    ffff88001fda1e98: 000000000000e02b 0000000000000000 
>    ffff88001fda1ea8: ffffffff8051dba0 ffffffff8038d62f 
> #1 [ffff88001fda1eb0] sysrq_handle_crashdump at ffffffff8038d62f
>    ffff88001fda1eb8: 0000000000000063 ffffffff8038d42e 
> #2 [ffff88001fda1ec0] __handle_sysrq at ffffffff8038d42e
>    ffff88001fda1ec8: 0000000000000000 0000000000000002 
>    ffff88001fda1ed8: 0000000000000002 00002ac18cd4c000 
>    ffff88001fda1ee8: ffff88001fda1f50 0000000000000000 
>    ffff88001fda1ef8: 0000000000000000 ffffffff802f30cb 
> #3 [ffff88001fda1f00] write_sysrq_trigger at ffffffff802f30cb
>    ffff88001fda1f08: ffff88002656d9c0 ffffffff80216c4a 
> #4 [ffff88001fda1f10] vfs_write at ffffffff80216c4a
>    ffff88001fda1f18: c000003e00000001 ffff88002656d9c0 
>    ffff88001fda1f28: 0000000000000002 fffffffffffffff7 
>    ffff88001fda1f38: 00002ac18cd4c000 ffffffff80217482 
> #5 [ffff88001fda1f40] sys_write at ffffffff80217482
>    ffff88001fda1f48: 0000000000000000 0000000000000000 
>    ffff88001fda1f58: 0000000000000000 0000000000000002 
>    ffff88001fda1f68: 00002ac18cd4c000 00002ac189770760 
>    ffff88001fda1f78: 0000000000000002 ffffffff802602f1 
> #6 [ffff88001fda1f80] tracesys at ffffffff802602f1 (via system_call)
>    RIP: 00002ac1894e6310  RSP: 00007fff21ca9e68  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000002  RSI: 00002ac18cd4c000  RDI: 0000000000000001
>    RBP: 0000000000000002   R8: 00000000ffffffff   R9: 00002ac189776e10
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00002ac189770760
>    R13: 00002ac18cd4c000  R14: 0000000000000002  R15: 0000000000000000
>    ORIG_RAX: 0000000000000001  CS: 0033  SS: e02b
>crash> bt -e
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
>
>  USER-MODE EXCEPTION FRAME AT: ffff88001fda1f58
>    RIP: 00002ac1894e6310  RSP: 00007fff21ca9e68  RFLAGS: 00000246
>    RAX: ffffffffffffffda  RBX: ffffffff802602f1  RCX: ffffffffffffffff
>    RDX: 0000000000000002  RSI: 00002ac18cd4c000  RDI: 0000000000000001
>    RBP: 0000000000000002   R8: 00000000ffffffff   R9: 00002ac189776e10
>    R10: 0000000000000000  R11: 0000000000000246  R12: 00002ac189770760
>    R13: 00002ac18cd4c000  R14: 0000000000000002  R15: 0000000000000000
>    ORIG_RAX: 0000000000000001  CS: 0033  SS: e02b
>crash> bt -E
>CPU 0 IRQ STACK:
>(none found)
>
>CPU 1 IRQ STACK:
>(none found)
>
>CPU 2 IRQ STACK:
>(none found)
>
>CPU 3 IRQ STACK:
>(none found)
>
>crash> ps
>   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>>     0      0   0  ffffffff804d6b00  RU   0.0       0      0  [swapper]
>>     0      1   1  ffff880000004040  RU   0.0       0      0  [swapper]
>      0      1   2  ffff880000ec57e0  RU   0.0       0      0  [swapper]
>>     0      1   3  ffff880000ec5080  RU   0.0       0      0  [swapper]
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>      1      0   3  ffff8800000047a0  IN   0.1   65996    956  ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>init
>      2      1   0  ffff880000ec7820  IN   0.0       0      0  [migration/0]
>      3      1   0  ffff880000ec70c0  IN   0.0       0      0  [ksoftirqd/0]
>      4      1   0  ffff880000ec9860  IN   0.0       0      0  [watchdog/0]
>      5      1   1  ffff880000ec9100  IN   0.0       0      0  [migration/1]
>      6      1   1  ffff880000ed57a0  IN   0.0       0      0  [ksoftirqd/1]
>      7      1   1  ffff880000ed5040  IN   0.0       0      0  [watchdog/1]
>      8      1   2  ffff880000ed77e0  IN   0.0       0      0  [migration/2]
>      9      1   2  ffff880000ed7080  IN   0.0       0      0  [ksoftirqd/2]
>     11      1   3  ffff88000004d0c0  IN   0.0       0      0  [migration/3]
>     12      1   3  ffff880000056860  IN   0.0       0      0  [ksoftirqd/3]
>     13      1   3  ffff880000056100  IN   0.0       0      0  [watchdog/3]
>   1712     -1   3  ffff88003072f100  IN   0.0       0      0  [kmpathd/3]
>   1788      1   0  ffff88003072d100  IN   0.2   66004   1552  rc
>   1886     -1   0  ffff880030fc5080  IN   0.0       0      0  [kondemand/0]
>   2255      1   0  ffff88002e9b2860  IN   0.1    6720    748  dhclient
>   2337      1   0  ffff88002ea4b040  IN   0.1   18360    684  auditd
>   2338     -1   0  ffff880031be0820  IN   0.1   16228    716  audispd
>   2339     -1   0  ffff88002ee35080  IN   0.1   16228    716  audispd
>   2366      1   3  ffff88002f7e5820  IN   0.0    3788    428  klogd
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>   2377      1   2  ffff88002f02c100  IN   0.0    3788    428  ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>irqbalance
>   2478      1   0  ffff880030bed040  IN   0.1   31488   1100  dbus-daemon
>   2479      1   0  ffff88002fc1b7e0  IN   0.1   31488   1100  dbus-daemon
>   2493      1   1  ffff88003072d860  IN   0.1    5916    520  sdpd
>   2580      1   3  ffff880029ea30c0  IN   0.1    8500    476  hidd
>   2598      1   0  ffff88002984b7a0  IN   0.2   37960   1444  automount
>   2599      1   0  ffff88003072f860  IN   0.2   37960   1444  automount
>   2600      1   0  ffff88002f02c860  IN   0.2   37960   1444  automount
>   2629      1   0  ffff88002f4677a0  IN   0.1   60528   1192  sshd
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>   2690      1   1  ffff880030fc57e0  IN   0.1   60528   1192  ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>gpm
>   2737      1   0  ffff88002936c0c0  IN   0.1   20156   1132  xfs
>   2762      1   0  ffff880029b47860  IN   0.1   18700    452  atd
>   2786      1   0  ffff880031ede860  IN   0.2   24132   1468  avahi-daemon
>   2797      1   0  ffff88002f7e50c0  IN   0.4   30696   3908  hald
>   2809     -1   0  ffff880029ab27e0  IN   0.1   10192    724  hald-addon-stor
>   2811     -1   3  ffff88002860e7a0  IN   0.1   10196    724  hald-addon-stor
>   2835      1   0  ffff88002984b040  IN   1.7  240784  14916  yum-updatesd
>   4591     -1   2  ffff880024e4c820  IN   0.2   65872   1372  rhts-test-runne
>   4625   4591   1  ffff88002603f7e0  IN   0.1   59004    896  make
>   4928     -1   1  ffff880024eb0040  IN   0.2   65992   1428  master
>   5430   4591   3  ffff88002695b7e0  IN   0.1   58864    548  sleep
>>  5431   4928   2  ffff880029e380c0  RU   0.1   65996    956  master
>crash> ps -k
>   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>>     0      0   0  ffffffff804d6b00  RU   0.0       0      0  [swapper]
>>     0      1   1  ffff880000004040  RU   0.0       0      0  [swapper]
>      0      1   2  ffff880000ec57e0  RU   0.0       0      0  [swapper]
>>     0      1   3  ffff880000ec5080  RU   0.0       0      0  [swapper]
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>      2      1   0  ffff880000ec7820  IN   0.0       0      0  [migration/0]
>      3      1   0  ffff880000ec70c0  IN   0.0       0      0  [ksoftirqd/0]
>      4      1   0  ffff880000ec9860  IN   0.0       0      0  [watchdog/0]
>      5      1   1  ffff880000ec9100  IN   0.0       0      0  [migration/1]
>      6      1   1  ffff880000ed57a0  IN   0.0       0      0  [ksoftirqd/1]
>      7      1   1  ffff880000ed5040  IN   0.0       0      0  [watchdog/1]
>      8      1   2  ffff880000ed77e0  IN   0.0       0      0  [migration/2]
>      9      1   2  ffff880000ed7080  IN   0.0       0      0  [ksoftirqd/2]
>     11      1   3  ffff88000004d0c0  IN   0.0       0      0  [migration/3]
>     12      1   3  ffff880000056860  IN   0.0       0      0  [ksoftirqd/3]
>     13      1   3  ffff880000056100  IN   0.0       0      0  [watchdog/3]
>   1712     -1   3  ffff88003072f100  IN   0.0       0      0  [kmpathd/3]
>   1886     -1   0  ffff880030fc5080  IN   0.0       0      0  [kondemand/0]
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>crash> ps -u
>   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>      1      0   3  ffff8800000047a0  IN   0.1   65996    956  ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>init
>   1788      1   0  ffff88003072d100  IN   0.2   66004   1552  rc
>   2255      1   0  ffff88002e9b2860  IN   0.1    6720    748  dhclient
>   2337      1   0  ffff88002ea4b040  IN   0.1   18360    684  auditd
>   2338     -1   0  ffff880031be0820  IN   0.1   16228    716  audispd
>   2339     -1   0  ffff88002ee35080  IN   0.1   16228    716  audispd
>   2366      1   3  ffff88002f7e5820  IN   0.0    3788    428  klogd
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>   2377      1   2  ffff88002f02c100  IN   0.0    3788    428  ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>irqbalance
>   2478      1   0  ffff880030bed040  IN   0.1   31488   1100  dbus-daemon
>   2479      1   0  ffff88002fc1b7e0  IN   0.1   31488   1100  dbus-daemon
>   2493      1   1  ffff88003072d860  IN   0.1    5916    520  sdpd
>   2580      1   3  ffff880029ea30c0  IN   0.1    8500    476  hidd
>   2598      1   0  ffff88002984b7a0  IN   0.2   37960   1444  automount
>   2599      1   0  ffff88003072f860  IN   0.2   37960   1444  automount
>   2600      1   0  ffff88002f02c860  IN   0.2   37960   1444  automount
>   2629      1   0  ffff88002f4677a0  IN   0.1   60528   1192  sshd
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>   2690      1   1  ffff880030fc57e0  IN   0.1   60528   1192  ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>gpm
>   2737      1   0  ffff88002936c0c0  IN   0.1   20156   1132  xfs
>   2762      1   0  ffff880029b47860  IN   0.1   18700    452  atd
>   2786      1   0  ffff880031ede860  IN   0.2   24132   1468  avahi-daemon
>   2797      1   0  ffff88002f7e50c0  IN   0.4   30696   3908  hald
>   2809     -1   0  ffff880029ab27e0  IN   0.1   10192    724  hald-addon-stor
>   2811     -1   3  ffff88002860e7a0  IN   0.1   10196    724  hald-addon-stor
>   2835      1   0  ffff88002984b040  IN   1.7  240784  14916  yum-updatesd
>   4591     -1   2  ffff880024e4c820  IN   0.2   65872   1372  rhts-test-runne
>   4625   4591   1  ffff88002603f7e0  IN   0.1   59004    896  make
>   4928     -1   1  ffff880024eb0040  IN   0.2   65992   1428  master
>   5430   4591   3  ffff88002695b7e0  IN   0.1   58864    548  sleep
>>  5431   4928   2  ffff880029e380c0  RU   0.1   65996    956  master
>crash> ps -s
>   PID    PPID  CPU      KSTACKP      ST  %MEM     VSZ    RSS  COMM
>>     0      0   0         --         RU   0.0       0      0  [swapper]
>>     0      1   1         --         RU   0.0       0      0  [swapper]
>      0      1   2  ffff880000ee7e68  RU   0.0       0      0  [swapper]
>>     0      1   3         --         RU   0.0       0      0  [swapper]
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>      1      0   3  ffff880000ec1940  IN   0.1   65996    956  ps: read error: kernel virtual address: ffff8800346deac0  type: "fill_mm_struct"
>init
>      2      1   0  ffff880000eebdc8  IN   0.0       0      0  [migration/0]
>      3      1   0  ffff880000eefdf8  IN   0.0       0      0  [ksoftirqd/0]
>      4      1   0  ffff880000ef1de8  IN   0.0       0      0  [watchdog/0]
>      5      1   1  ffff880000ef3dc8  IN   0.0       0      0  [migration/1]
>      6      1   1  ffff880000ef7df8  IN   0.0       0      0  [ksoftirqd/1]
>      7      1   1  ffff880000efdde8  IN   0.0       0      0  [watchdog/1]
>      8      1   2  ffff880000041dc8  IN   0.0       0      0  [migration/2]
>      9      1   2  ffff88000006bdf8  IN   0.0       0      0  [ksoftirqd/2]
>     11      1   3  ffff88000006fdc8  IN   0.0       0      0  [migration/3]
>     12      1   3  ffff880000073df8  IN   0.0       0      0  [ksoftirqd/3]
>     13      1   3  ffff88000007dde8  IN   0.0       0      0  [watchdog/3]
>   1712     -1   3  ffff88002f935d98  IN   0.0       0      0  [kmpathd/3]
>   1788      1   0  ffff8800313f7e10  IN   0.2   66004   1552  rc
>   1886     -1   0  ffff8800314c5d98  IN   0.0       0      0  [kondemand/0]
>   2255      1   0  ffff8800329b9940  IN   0.1    6720    748  dhclient
>   2337      1   0  ffff8800304efc70  IN   0.1   18360    684  auditd
>   2338     -1   0  ffff880032f99c70  IN   0.1   16228    716  audispd
>   2339     -1   0  ffff88002f8cda60  IN   0.1   16228    716  audispd
>   2366      1   3  ffff880029dd3d90  IN   0.0    3788    428  klogd
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>   2377      1   2  ffff880029df3de0  IN   0.0    3788    428  ps: read error: kernel virtual address: ffff8800346de040  type: "fill_mm_struct"
>irqbalance
>   2478      1   0  ffff880029f67a60  IN   0.1   31488   1100  dbus-daemon
>   2479      1   0  ffff880029de3a70  IN   0.1   31488   1100  dbus-daemon
>   2493      1   1  ffff8800299718f0  IN   0.1    5916    520  sdpd
>   2580      1   3  ffff88002945ba00  IN   0.1    8500    476  hidd
>   2598      1   0  ffff880029a75d80  IN   0.2   37960   1444  automount
>   2599      1   0  ffff88002944fc70  IN   0.2   37960   1444  automount
>   2600      1   0  ffff880029a89c70  IN   0.2   37960   1444  automount
>   2629      1   0  ffff880029681940  IN   0.1   60528   1192  sshd
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>   2690      1   1  ffff8800285b3940  IN   0.1   60528   1192  ps: read error: kernel virtual address: ffff8800346e0100  type: "fill_mm_struct"
>gpm
>   2737      1   0  ffff8800280cb940  IN   0.1   20156   1132  xfs
>   2762      1   0  ffff880028819de0  IN   0.1   18700    452  atd
>   2786      1   0  ffff880028835a60  IN   0.2   24132   1468  avahi-daemon
>   2797      1   0  ffff880028675a60  IN   0.4   30696   3908  hald
>   2809     -1   0  ffff8800274edde0  IN   0.1   10192    724  hald-addon-stor
>   2811     -1   3  ffff880027475de0  IN   0.1   10196    724  hald-addon-stor
>   2835      1   0  ffff880026dc1a60  IN   1.7  240784  14916  yum-updatesd
>   4591     -1   2  ffff880026583e10  IN   0.2   65872   1372  rhts-test-runne
>   4625   4591   1  ffff88001e8f9e10  IN   0.1   59004    896  make
>   4928     -1   1  ffff88001e581e10  IN   0.2   65992   1428  master
>   5430   4591   3  ffff88001df8bde0  IN   0.1   58864    548  sleep
>>  5431   4928   2         --         RU   0.1   65996    956  master
>crash> dev
>CHRDEV    NAME            OPERATIONS   
>dev: read error: kernel virtual address: ffff880034739d40  type: "char_device_struct"
>crash> dev -p
>dev: read error: kernel virtual address: ffff8800345ac800  type: "first list entry"
>crash> kmem -i
>              PAGES        TOTAL      PERCENTAGE
> TOTAL MEM   215215     840.7 MB         ----
>      FREE   119770     467.9 MB   55% of TOTAL MEM
>      USED    95445     372.8 MB   44% of TOTAL MEM
>    SHARED     8746      34.2 MB    4% of TOTAL MEM
>   BUFFERS     4885      19.1 MB    2% of TOTAL MEM
>    CACHED    57475     224.5 MB   26% of TOTAL MEM
>      SLAB     6978      27.3 MB    3% of TOTAL MEM
>
>TOTAL HIGH        0            0    0% of TOTAL MEM
> FREE HIGH        0            0    0% of TOTAL HIGH
> TOTAL LOW   215215     840.7 MB  100% of TOTAL MEM
>  FREE LOW   119770     467.9 MB   55% of TOTAL LOW
>
>kmem: read error: kernel virtual address: ffff8800346838e8  type: "fill_dentry_cache"
>crash> kmem -s
>kmem: kmem cache slab subsystem not available
>crash> task
>PID: 5431   TASK: ffff880029e380c0  CPU: 2   COMMAND: "master"
>struct task_struct {
>  state = 0, 
>  thread_info = 0xffff88001fda0000, 
>  usage = {
>    counter = 2
>  }, 
>  flags = 4194368, 
>  lock_depth = -1, 
>  load_weight = 128, 
>  prio = 115, 
>  static_prio = 120, 
>  normal_prio = 115, 
>  run_list = {
>    next = 0xffff880001010d08, 
>    prev = 0xffff880001010d08
>  }, 
>  array = 0xffff8800010105b8, 
>  ioprio = 0, 
>  btrace_seq = 0, 
>  sleep_avg = 799958935, 
>  timestamp = 182853090448, 
>  last_ran = 182853078472, 
>  sched_time = 328523, 
>  sleep_type = SLEEP_NORMAL, 
>  policy = 0, 
>  cpus_allowed = {
>    bits = {15, 0, 0, 0}
>  }, 
>  time_slice = 4, 
>  first_time_slice = 4928, 
>  sched_info = {
>    cpu_time = 0, 
>    run_delay = 0, 
>    pcnt = 2, 
>    last_arrival = 4294938009, 
>    last_queued = 0
>  }, 
>  tasks = {
>    next = 0xffffffff804d6bd8, 
>    prev = 0xffff88002695b8b8
>  }, 
>  mm = 0xffff88002f653800, 
>  active_mm = 0xffff88002f653800, 
>  binfmt = 0xffffffff80509660, 
>  exit_state = 0, 
>  exit_code = 0, 
>  exit_signal = 17, 
>  pdeath_signal = 0, 
>  personality = 0, 
>  did_exec = 0, 
>  pid = 5431, 
>  tgid = 5431, 
>  parent = 0xffff880024eb0040, 
>  children = {
>    next = 0xffff880029e381f8, 
>    prev = 0xffff880029e381f8
>  }, 
>  sibling = {
>    next = 0xffff880024eb0178, 
>    prev = 0xffff880024eb0178
>  }, 
>  group_leader = 0xffff880029e380c0, 
>  pids = {{
>      node = {
>        next = 0x0, 
>        pprev = 0xffff8800346fe5f8
>      }, 
>      pid = 0xffff8800346fe5e0
>    }, {
>      node = {
>        next = 0xffff880024eb01b8, 
>        pprev = 0xffff880000005a80
>      }, 
>      pid = 0xffff880000005a60
>    }, {
>      node = {
>        next = 0xffff88002695b970, 
>        pprev = 0xffff880031ccd288
>      }, 
>      pid = 0xffff880031ccd260
>    }}, 
>  thread_group = {
>    next = 0xffff880029e38268, 
>    prev = 0xffff880029e38268
>  }, 
>  vfork_done = 0x0, 
>  set_child_tid = 0x2ac189776ea0, 
>  clear_child_tid = 0x2ac189776ea0, 
>  rt_priority = 0, 
>  utime = 0, 
>  stime = 0, 
>  nvcsw = 0, 
>  nivcsw = 1, 
>  start_time = {
>    tv_sec = 182, 
>    tv_nsec = 923116669
>  }, 
>  min_flt = 217, 
>  maj_flt = 0, 
>  it_prof_expires = 0, 
>  it_virt_expires = 0, 
>  it_sched_expires = 0, 
>  cpu_timers = {{
>      next = 0xffff880029e382f0, 
>      prev = 0xffff880029e382f0
>    }, {
>      next = 0xffff880029e38300, 
>      prev = 0xffff880029e38300
>    }, {
>      next = 0xffff880029e38310, 
>      prev = 0xffff880029e38310
>    }}, 
>  uid = 0, 
>  euid = 0, 
>  suid = 0, 
>  fsuid = 0, 
>  gid = 0, 
>  egid = 0, 
>  sgid = 0, 
>  fsgid = 0, 
>  group_info = 0xffffffff804ddde0, 
>  cap_effective = 4294967039, 
>  cap_inheritable = 0, 
>  cap_permitted = 4294967039, 
>  keep_capabilities = 0, 
>  user = 0xffffffff804ddd60, 
>  request_key_auth = 0x0, 
>  thread_keyring = 0x0, 
>  jit_keyring = 0 '\0', 
>  oomkilladj = 0, 
>  comm = "master\000.sh\000unne", 
>  link_count = 0, 
>  total_link_count = 0, 
>  sysvsem = {
>    undo_list = 0x0
>  }, 
>  thread = {
>    rsp0 = 18446612132848607232, 
>    rsp = 18446612132848606480, 
>    userrsp = 140733760316480, 
>    fs = 47010723360272, 
>    gs = 0, 
>    es = 0, 
>    ds = 0, 
>    fsindex = 0, 
>    gsindex = 0, 
>    debugreg0 = 0, 
>    debugreg1 = 0, 
>    debugreg2 = 0, 
>    debugreg3 = 0, 
>    debugreg6 = 0, 
>    debugreg7 = 0, 
>    cr2 = 0, 
>    trap_no = 0, 
>    error_code = 0, 
>    i387 = {
>      fxsave = {
>        cwd = 895, 
>        swd = 0, 
>        twd = 0, 
>        fop = 0, 
>        rip = 0, 
>        rdp = 0, 
>        mxcsr = 8064, 
>        mxcsr_mask = 65535, 
>        st_space = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 
>        xmm_space = {0, 3220176896, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 
>        padding = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}
>      }
>    }, 
>    ioperm = 0, 
>    io_bitmap_ptr = 0x0, 
>    io_bitmap_max = 0, 
>    tls_array = {0, 0, 0}, 
>    iopl = 0
>  }, 
>  fs = 0xffff880031c68b80, 
>  files = 0xffff880031be10c0, 
>  namespace = 0xffff880034896d00, 
>  signal = 0xffff8800341a5380, 
>  sighand = 0xffff880026940980, 
>  blocked = {
>    sig = {0}
>  }, 
>  real_blocked = {
>    sig = {0}
>  }, 
>  saved_sigmask = {
>    sig = {0}
>  }, 
>  pending = {
>    list = {
>      next = 0xffff880029e386a0, 
>      prev = 0xffff880029e386a0
>    }, 
>    signal = {
>      sig = {0}
>    }
>  }, 
>  sas_ss_sp = 0, 
>  sas_ss_size = 0, 
>  notifier = 0, 
>  notifier_data = 0x0, 
>  notifier_mask = 0x0, 
>  tux_info = 0x0, 
>  tux_exit = 0, 
>  security = 0xffff880031ad2860, 
>  audit_context = 0xffff880027407800, 
>  seccomp = <incomplete type>, 
>  utrace = 0x0, 
>  utrace_flags = 0, 
>  parent_exec_id = 10, 
>  self_exec_id = 10, 
>  alloc_lock = {
>    raw_lock = {
>      slock = 1
>    }
>  }, 
>  pi_lock = {
>    raw_lock = {
>      slock = 1
>    }
>  }, 
>  pi_waiters = {
>    prio_list = {
>      next = 0xffff880029e38720, 
>      prev = 0xffff880029e38720
>    }, 
>    node_list = {
>      next = 0xffff880029e38730, 
>      prev = 0xffff880029e38730
>    }
>  }, 
>  pi_blocked_on = 0x0, 
>  journal_info = 0x0, 
>  reclaim_state = 0x0, 
>  backing_dev_info = 0x0, 
>  io_context = 0x0, 
>  io_wait = 0x0, 
>  rchar = 6336, 
>  wchar = 24, 
>  syscr = 5, 
>  syscw = 2, 
>  acct_rss_mem1 = 0, 
>  acct_vm_mem1 = 0, 
>  acct_stimexpd = 0, 
>  cpuset = 0xffffffff804df900, 
>  mems_allowed = {
>    bits = {1}
>  }, 
>  cpuset_mems_generation = 0, 
>  cpuset_mem_spread_rotor = 0, 
>  robust_list = 0x0, 
>  compat_robust_list = 0x0, 
>  pi_state_list = {
>    next = 0xffff880029e387d0, 
>    prev = 0xffff880029e387d0
>  }, 
>  pi_state_cache = 0x0, 
>  fs_excl = {
>    counter = 0
>  }, 
>  rcu = {
>    next = 0x0, 
>    func = 0
>  }, 
>  ptracees = {
>    next = 0xffff880029e38800, 
>    prev = 0xffff880029e38800
>  }, 
>  splice_pipe = 0x0, 
>  delays = 0xffff8800301e3320
>}
>
>crash> exit

Actions: View

Attachments on bug 442661: 302542 | 302543 | 302544 | 302546