Attachment 283881 Details for Bug 419391 – kernel oops message

kernel oops message

20071211-kernel-oops.txt (text/plain), 15.97 KB, created by Reiner Rottmann on 2007-12-11 09:23:59 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Reiner Rottmann

Created: 2007-12-11 09:23:59 UTC

Size: 15.97 KB

patch

obsolete

>Dec 11 09:00:27 node02.mgmt  Unable to handle kernel NULL pointer dereference at 0000000000000000 RIP:
>Dec 11 09:00:27 node02.mgmt  <ffffffffa0179aef>{:gfs:gfs_glock_dq+191}
>Dec 11 09:00:27 node02.mgmt  PML4 ac071067 PGD 83528067 PMD 0
>Dec 11 09:00:27 node02.mgmt  Oops: 0000 [1] SMP
>Dec 11 09:00:27 node02.mgmt  CPU 1
>Dec 11 09:00:27 node02.mgmt  Modules linked in: parport_pc lp parport netconsole netdump autofs4 i2c_dev i2c_core sunrpc ds yenta_socket pcmcia_core ext3 jbd button battery ac ohci_hcd hw_random scsi_transport_fc md5 ipv6 lock_dlm(U) dlm(U) gfs(U) lock_harness(U) cman(U) 8021q bonding(U) qla2300(U) qla2xxx_conf(U) qla2xxx(U) sd_mod scsi_mod dm_snapshot dm_mirror dm_mod tg3
>Dec 11 09:00:27 node02.mgmt  Pid: 23773, comm: site_preview.pl Not tainted 2.6.9-42.0.3.ELsmp
>Dec 11 09:00:27 node02.mgmt  RIP: 0010:[<ffffffffa0179aef>] <ffffffffa0179aef>{:gfs:gfs_glock_dq+191}
>Dec 11 09:00:27 node02.mgmt  RSP: 0018:000001002f0f1ce8  EFLAGS: 00010203
>Dec 11 09:00:27 node02.mgmt  RAX: 000001004124d558 RBX: 0000000000000000 RCX: 00000100eda4e210
>Dec 11 09:00:27 node02.mgmt  RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000001004124d54c
>Dec 11 09:00:27 node02.mgmt  RBP: 000001004124d520 R08: 0000000000000000 R09: 0000000000000000
>Dec 11 09:00:27 node02.mgmt  R10: 0000000000000000 R11: 00000100eda4e1a0 R12: 000001004124d54c
>Dec 11 09:00:27 node02.mgmt  R13: 0000010043265da8 R14: ffffff00005a3000 R15: 000001003205b2a8
>Dec 11 09:00:27 node02.mgmt  FS:  0000002a95561a20(0000) GS:ffffffff804e5200(0000) knlGS:0000000000000000
>Dec 11 09:00:27 node02.mgmt  CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>Dec 11 09:00:27 node02.mgmt  CR2: 0000000000000000 CR3: 0000000080682000 CR4: 00000000000006e0
>Dec 11 09:00:27 node02.mgmt  Process site_preview.pl (pid: 23773, threadinfo 000001002f0f0000, task 00000100cb43b030)
>Dec 11 09:00:27 node02.mgmt  Stack: 000001002f0f1cf8 0000e32500000000 ffffffffa01b99c0 0000010043265da8
>Dec 11 09:00:27 node02.mgmt         0000010043265d80 0000010043265da8 0000010076b40178 0000000000000000
>Dec 11 09:00:27 node02.mgmt         0000000000000000 ffffffffa0179c08
>Dec 11 09:00:27 node02.mgmt  Call Trace:<ffffffffa0179c08>{:gfs:gfs_glock_dq_uninit+9} <ffffffffa018ef63>{:gfs:gfs_flock+222}
>Dec 11 09:00:28 node02.mgmt         <ffffffffa01e1781>{:lock_dlm:get_resource+84} <ffffffffa01e23d7>{:lock_dlm:lm_dlm_punlock+642}
>Dec 11 09:00:28 node02.mgmt         <ffffffff8018e5e4>{locks_remove_flock+97} <ffffffff8017a21d>{__fput+73}
>Dec 11 09:00:28 node02.mgmt         <ffffffff80178e48>{filp_close+103} <ffffffff80178ed1>{sys_close+130}
>Dec 11 09:00:28 node02.mgmt         <ffffffff8011026a>{system_call+126}
>Dec 11 09:00:28 node02.mgmt
>Dec 11 09:00:28 node02.mgmt  Code: 48 8b 02 0f 18 08 48 8d 45 38 48 39 c2 74 0c ff 44 24 0c 49
>Dec 11 09:00:28 node02.mgmt  RIP <ffffffffa0179aef>{:gfs:gfs_glock_dq+191} RSP <000001002f0f1ce8>
>Dec 11 09:00:28 node02.mgmt  CR2: 0000000000000000
>Dec 11 09:00:29 node05.mgmt clurgmgrd: [12433]: <info> Executing /etc/init.d/mysqld-multi status
>Dec 11 09:00:29 node05.mgmt clurgmgrd: [12433]: <info> Executing /etc/init.d/named-multi status
>Dec 11 09:00:29 node04.mgmt clurgmgrd: [14697]: <info> Executing /etc/init.d/keepalived status
>Dec 11 09:00:35 node06.mgmt clurgmgrd: [12455]: <info> Executing /etc/init.d/postfix-multi status
>Dec 11 09:00:44 node01.mgmt  CMAN: removing node node02 from the cluster : Missed too many heartbeats
>Dec 11 09:00:44 node03.mgmt fenced[13987]: fencing deferred to node01
>Dec 11 09:00:44 node06.mgmt fenced[11994]: fencing deferred to node01
>Dec 11 09:00:44 node04.mgmt fenced[14199]: fencing deferred to node01
>Dec 11 09:00:44 node01.mgmt fenced[16304]: node02 not a cluster member after 0 sec post_fail_delay
>Dec 11 09:00:44 node05.mgmt fenced[11980]: fencing deferred to node01
>Dec 11 09:00:44 node01.mgmt fenced[16304]: fencing node "node02"
>Dec 11 09:00:44 node01.mgmt kernel: CMAN: removing node node02 from the cluster : Missed too many heartbeats
>Dec 11 09:00:45 node04.mgmt kernel: CMAN: node node02 has been removed from the cluster : Missed too many heartbeats
>Dec 11 09:00:45 node06.mgmt kernel: CMAN: node node02 has been removed from the cluster : Missed too many heartbeats
>Dec 11 09:00:45 node03.mgmt kernel: CMAN: node node02 has been removed from the cluster : Missed too many heartbeats
>Dec 11 09:00:45 node05.mgmt kernel: CMAN: node node02 has been removed from the cluster : Missed too many heartbeats
>Dec 11 09:00:53 node01.mgmt fenced[16304]: fence "node02" success
>Dec 11 09:01:00 node04.mgmt clurgmgrd: [14697]: <info> Executing /etc/init.d/keepalived status
>Dec 11 09:01:00 node05.mgmt clurgmgrd: [12433]: <info> Executing /etc/init.d/mysqld-multi status
>Dec 11 09:01:00 node05.mgmt clurgmgrd: [12433]: <info> Executing /etc/init.d/named-multi status
>Dec 11 09:01:00 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Trying to acquire journal lock...
>Dec 11 09:01:00 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Trying to acquire journal lock...
>Dec 11 09:01:00 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Looking at journal...
>Dec 11 09:01:00 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Looking at journal...
>Dec 11 09:01:00 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Trying to acquire journal lock...
>Dec 11 09:01:00 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Trying to acquire journal lock...
>Dec 11 09:01:00 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Looking at journal...
>Dec 11 09:01:00 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Looking at journal...
>Dec 11 09:01:01 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt  GFS: fsid=web:backup.4: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node04.mgmt  GFS: fsid=web:backup.0: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node05.mgmt  GFS: fsid=web:lt_livedata.1: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node03.mgmt  GFS: fsid=web:backup.3: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Looking at journal...
>Dec 11 09:01:01 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Looking at journal...
>Dec 11 09:01:01 node03.mgmt  GFS: fsid=web:lt_livedata.3: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt  GFS: fsid=web:lt_livedata.4: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node01.mgmt  GFS: fsid=web:lt_livedata.5: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node05.mgmt  GFS: fsid=web:scratch.0: jid=1: Trying to acquire journal lock...
>Dec 11 09:01:01 node03.mgmt kernel: GFS: fsid=web:backup.3: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node03.mgmt kernel: GFS: fsid=web:lt_livedata.3: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node01.mgmt kernel: GFS: fsid=web:lt_livedata.5: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt kernel: GFS: fsid=web:backup.4: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node05.mgmt kernel: GFS: fsid=web:lt_livedata.1: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node04.mgmt kernel: GFS: fsid=web:backup.0: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Trying to acquire journal lock...
>Dec 11 09:01:01 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Looking at journal...
>Dec 11 09:01:01 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Looking at journal...
>Dec 11 09:01:01 node05.mgmt kernel: GFS: fsid=web:scratch.0: jid=1: Trying to acquire journal lock...
>Dec 11 09:01:01 node06.mgmt kernel: GFS: fsid=web:lt_livedata.4: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node01.mgmt  GFS: fsid=web:backup.5: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:01 node01.mgmt  GFS: fsid=web:scratch.3: jid=1: Trying to acquire journal lock...
>Dec 11 09:01:02 node01.mgmt  GFS: fsid=web:scratch.3: jid=1: Busy
>Dec 11 09:01:02 node01.mgmt kernel: GFS: fsid=web:backup.5: jid=2: Trying to acquire journal lock...
>Dec 11 09:01:02 node01.mgmt kernel: GFS: fsid=web:scratch.3: jid=1: Trying to acquire journal lock...
>Dec 11 09:01:02 node01.mgmt kernel: GFS: fsid=web:scratch.3: jid=1: Busy
>Dec 11 09:01:02 node03.mgmt  GFS: fsid=web:backup.3: jid=2: Busy
>Dec 11 09:01:02 node06.mgmt  GFS: fsid=web:backup.4: jid=2: Busy
>Dec 11 09:01:02 node05.mgmt  GFS: fsid=web:scratch.0: jid=1: Busy
>Dec 11 09:01:02 node03.mgmt kernel: GFS: fsid=web:backup.3: jid=2: Busy
>Dec 11 09:01:02 node01.mgmt  GFS: fsid=web:backup.5: jid=2: Busy
>Dec 11 09:01:02 node06.mgmt kernel: GFS: fsid=web:backup.4: jid=2: Busy
>Dec 11 09:01:02 node05.mgmt kernel: GFS: fsid=web:scratch.0: jid=1: Busy
>Dec 11 09:01:02 node01.mgmt kernel: GFS: fsid=web:backup.5: jid=2: Busy
>Dec 11 09:01:02 node03.mgmt  GFS: fsid=web:lt_livedata.3: jid=2: Busy
>Dec 11 09:01:02 node05.mgmt  GFS: fsid=web:lt_livedata.1: jid=2: Busy
>Dec 11 09:01:02 node03.mgmt kernel: GFS: fsid=web:lt_livedata.3: jid=2: Busy
>Dec 11 09:01:02 node06.mgmt  GFS: fsid=web:lt_livedata.4: jid=2: Busy
>Dec 11 09:01:02 node05.mgmt kernel: GFS: fsid=web:lt_livedata.1: jid=2: Busy
>Dec 11 09:01:02 node06.mgmt kernel: GFS: fsid=web:lt_livedata.4: jid=2: Busy
>Dec 11 09:01:02 node01.mgmt  GFS: fsid=web:lt_livedata.5: jid=2: Busy
>Dec 11 09:01:02 node01.mgmt kernel: GFS: fsid=web:lt_livedata.5: jid=2: Busy
>Dec 11 09:01:02 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Acquiring the transaction lock...
>Dec 11 09:01:02 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Acquiring the transaction lock...
>Dec 11 09:01:02 node04.mgmt  GFS: fsid=web:backup.0: jid=2: Busy
>Dec 11 09:01:02 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Looking at journal...
>Dec 11 09:01:02 node04.mgmt kernel: GFS: fsid=web:backup.0: jid=2: Busy
>Dec 11 09:01:02 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Looking at journal...
>Dec 11 09:01:02 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Acquiring the transaction lock...
>Dec 11 09:01:02 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Acquiring the transaction lock...
>Dec 11 09:01:02 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Acquiring the transaction lock...
>Dec 11 09:01:03 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Acquiring the transaction lock...
>Dec 11 09:01:03 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Acquiring the transaction lock...
>Dec 11 09:01:03 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Replaying journal...
>Dec 11 09:01:03 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Acquiring the transaction lock...
>Dec 11 09:01:03 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Acquiring the transaction lock...
>Dec 11 09:01:03 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Replaying journal...
>Dec 11 09:01:03 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Replaying journal...
>Dec 11 09:01:03 node06.mgmt clurgmgrd: [12455]: <info> Executing /etc/init.d/httpd-multi status
>Dec 11 09:01:03 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Replayed 0 of 0 blocks
>Dec 11 09:01:03 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: replays = 0, skips = 0, sames = 0
>Dec 11 09:01:03 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Replayed 0 of 0 blocks
>Dec 11 09:01:03 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: replays = 0, skips = 0, sames = 0
>Dec 11 09:01:03 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Journal replayed in 1s
>Dec 11 09:01:03 node06.mgmt  GFS: fsid=web:scratch.2: jid=1: Done
>Dec 11 09:01:03 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Journal replayed in 1s
>Dec 11 09:01:03 node06.mgmt kernel: GFS: fsid=web:scratch.2: jid=1: Done
>Dec 11 09:01:03 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Replayed 45 of 66 blocks
>Dec 11 09:01:03 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: replays = 45, skips = 10, sames = 11
>Dec 11 09:01:03 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Journal replayed in 1s
>Dec 11 09:01:03 node01.mgmt  GFS: fsid=web:lt_cmsdata.1: jid=0: Done
>Dec 11 09:01:03 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Replaying journal...
>Dec 11 09:01:03 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Replaying journal...
>Dec 11 09:01:03 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Replayed 35 of 35 blocks
>Dec 11 09:01:03 node05.mgmt  GFS: fsid=web:backup.1: jid=2: replays = 35, skips = 0, sames = 0
>Dec 11 09:01:03 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Replayed 35 of 35 blocks
>Dec 11 09:01:03 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: replays = 35, skips = 0, sames = 0
>Dec 11 09:01:03 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Journal replayed in 1s
>Dec 11 09:01:03 node05.mgmt  GFS: fsid=web:backup.1: jid=2: Done
>Dec 11 09:01:03 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Journal replayed in 1s
>Dec 11 09:01:04 node05.mgmt kernel: GFS: fsid=web:backup.1: jid=2: Done
>Dec 11 09:01:04 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Replaying journal...
>Dec 11 09:01:04 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Replayed 32 of 32 blocks
>Dec 11 09:01:04 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: replays = 32, skips = 0, sames = 0
>Dec 11 09:01:04 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Journal replayed in 1s
>Dec 11 09:01:04 node01.mgmt  GFS: fsid=web:cmsroot.1: jid=0: Done
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Acquiring the transaction lock...
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Replaying journal...
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Replayed 45 of 66 blocks
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: replays = 45, skips = 10, sames = 11
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Journal replayed in 1s
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:lt_cmsdata.1: jid=0: Done
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Replaying journal...
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Replayed 32 of 32 blocks
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: replays = 32, skips = 0, sames = 0
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Journal replayed in 1s
>Dec 11 09:01:04 node01.mgmt kernel: GFS: fsid=web:cmsroot.1: jid=0: Done
>Dec 11 09:01:04 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Replaying journal...
>Dec 11 09:01:04 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Replaying journal...
>Dec 11 09:01:04 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Replayed 22 of 34 blocks
>Dec 11 09:01:04 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: replays = 22, skips = 0, sames = 12
>Dec 11 09:01:04 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Replayed 22 of 34 blocks
>Dec 11 09:01:04 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: replays = 22, skips = 0, sames = 12
>Dec 11 09:01:04 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Journal replayed in 1s
>Dec 11 09:01:04 node04.mgmt  GFS: fsid=web:lt_livedata.0: jid=2: Done
>Dec 11 09:01:04 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Journal replayed in 1s
>Dec 11 09:01:04 node04.mgmt kernel: GFS: fsid=web:lt_livedata.0: jid=2: Done
>Dec 11 09:01:04 node03.mgmt clurgmgrd[14470]: <info> Magma Event: Membership Change
>Dec 11 09:01:05 node03.mgmt clurgmgrd[14470]: <info> State change: node02 DOWN
>Dec 11 09:01:05 node01.mgmt clurgmgrd[16775]: <info> Magma Event: Membership Change
>Dec 11 09:01:05 node06.mgmt clurgmgrd[12455]: <info> Magma Event: Membership Change
>Dec 11 09:01:05 node05.mgmt clurgmgrd[12433]: <info> Magma Event: Membership Change
>Dec 11 09:01:05 node01.mgmt clurgmgrd[16775]: <info> State change: node02 DOWN
>Dec 11 09:01:05 node06.mgmt clurgmgrd[12455]: <info> State change: node02 DOWN
>Dec 11 09:01:05 node05.mgmt clurgmgrd[12433]: <info> State change: node02 DOWN
>Dec 11 09:01:05 node04.mgmt clurgmgrd[14697]: <info> Magma Event: Membership Change
>Dec 11 09:01:05 node04.mgmt clurgmgrd[14697]: <info> State change: node02 DOWN

Actions: View

Attachments on bug 419391: 283881