Attachment 310704 Details for Bug 254225 – log and kern dump from taft-04

log and kern dump from taft-04

taft-04.log (text/plain), 151.20 KB, created by Corey Marthaler on 2008-07-01 18:49:27 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2008-07-01 18:49:27 UTC

Size: 151.20 KB

patch

obsolete

>Jul  1 11:07:42 taft-04 qarshd[11580]: Running cmdline: vgscan 
>Jul  1 11:07:43 taft-04 qarshd[11580]: That's enough 
>Jul  1 11:07:44 taft-04 qarshd[11584]: Talking to peer 10.15.80.47:33912
>Jul  1 11:07:44 taft-04 qarshd[11584]: Running cmdline: pvs --noheadings -o +pv_uuid 
>Jul  1 11:07:45 taft-04 qarshd[11584]: That's enough 
>Jul  1 11:07:46 taft-04 kernel: dm-cmirror: Creating 3P2k7cK1 (1)
>Jul  1 11:07:46 taft-04 kernel: dm-cmirror: start_server called
>Jul  1 11:07:46 taft-04 kernel: dm-cmirror: cluster_log_serverd ready for work
>Jul  1 11:08:00 taft-04 kernel: dm-cmirror: Node joining
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror: server_id=dead, server_valid=0, 3P2k7cK1
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror: trigger = LRT_GET_SYNC_COUNT
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror: cluster_resume: Setting recovery_halted = 0
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (3P2k7cK1)
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror:   starter     : 4
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror:   co-ordinator: 4
>Jul  1 11:08:05 taft-04 kernel: dm-cmirror:   node_count  : 0
>Jul  1 11:08:05 taft-04 [7083]: Monitoring mirror device helter_skelter-syncd_primary_2legs_1 for events 
>Jul  1 11:08:05 taft-04 lvm[7083]: Loading config file: /etc/lvm/lvm.conf
>Jul  1 11:08:05 taft-04 lvm[7083]: Opened /etc/lvm/lvm.conf RO
>Jul  1 11:08:05 taft-04 lvm[7083]: Closed /etc/lvm/lvm.conf
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/syslog to 1
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/level to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/verbose to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/indent to 1
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/prefix to   
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/command_names to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting global/test to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting log/overwrite to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: log/activation not found in config: defaulting to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: Logging initialised at Tue Jul  1 11:08:05 2008
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting global/umask to 63
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting devices/dir to /dev
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting global/proc to /proc
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting global/activation to 1
>Jul  1 11:08:05 taft-04 lvm[7083]: global/suffix not found in config: defaulting to 1
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting global/units to h
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting activation/readahead to auto
>Jul  1 11:08:05 taft-04 lvm[7083]: devices/preferred_names not found in config file: using built-in preferences
>Jul  1 11:08:05 taft-04 lvm[7083]: Matcher built with 3 dfa states
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting devices/ignore_suspended_devices to 0
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting devices/cache_dir to /etc/lvm/cache
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting devices/write_cache_state to 1
>Jul  1 11:08:05 taft-04 lvm[7083]: Opened /etc/lvm/cache/.cache RO
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0003000000000000-part1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram11: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/VolGroup00/LogVol01: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram10: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0004000000000000-part1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/VolGroup00/LogVol00: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram12: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram6: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0005000000000000-part1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram13: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram14: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0007000000000000-part1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram5: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/mapper/VolGroup00-LogVol00: Aliased to /dev/VolGroup00/LogVol00 in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:02:0e.0-scsi-0:2:0:0-part1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram1: Aliased to /dev/ram in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sda1: Aliased to /dev/disk/by-path/pci-0000:02:0e.0-scsi-0:2:0:0-part1 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sdb1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sdc1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sdg1: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sdd1: Aliased to /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0003000000000000-part1 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sde1: Aliased to /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0004000000000000-part1 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sdf1: Aliased to /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0005000000000000-part1 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sdh1: Aliased to /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0007000000000000-part1 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ramdisk: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/mapper/VolGroup00-LogVol01: Aliased to /dev/VolGroup00/LogVol01 in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram0: Aliased to /dev/ramdisk in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/root: Aliased to /dev/VolGroup00/LogVol00 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/dm-1: Aliased to /dev/VolGroup00/LogVol01 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:02:0e.0-scsi-0:2:0:0-part2: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram2: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/sda2: Aliased to /dev/disk/by-path/pci-0000:02:0e.0-scsi-0:2:0:0-part2 in device cache (preferred name)
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram8: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram9: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/dm-0: Aliased to /dev/root in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram15: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0001000000000000-part1: Aliased to /dev/sdb1 in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0006000000000000-part1: Aliased to /dev/sdg1 in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram3: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram4: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/ram7: Added to device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: /dev/disk/by-path/pci-0000:0b:02.0-fc-0x500805f3000a05b1:0x0002000000000000-part1: Aliased to /dev/sdc1 in device cache
>Jul  1 11:08:05 taft-04 lvm[7083]: Loaded persistent filter cache from /etc/lvm/cache/.cache
>Jul  1 11:08:05 taft-04 lvm[7083]: Closed /etc/lvm/cache/.cache
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting activation/reserved_stack to 256
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting activation/reserved_memory to 8192
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting activation/process_priority to -18
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised format: lvm1
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised format: pool
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised format: lvm2
>Jul  1 11:08:05 taft-04 lvm[7083]: global/format not found in config: defaulting to lvm2
>Jul  1 11:08:05 taft-04 lvm[7083]: lvmcache: initialised VG #orphans_lvm1
>Jul  1 11:08:05 taft-04 lvm[7083]: lvmcache: initialised VG #orphans_pool
>Jul  1 11:08:05 taft-04 lvm[7083]: lvmcache: initialised VG #orphans_lvm2
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised segtype: striped
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised segtype: zero
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised segtype: error
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting dmeventd/snapshot_library to libdevmapper-event-lvm2snapshot.so
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting global/library_dir to /usr/lib64
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised segtype: snapshot
>Jul  1 11:08:05 taft-04 lvm[7083]: Initialised segtype: mirror
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting backup/retain_days to 30
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting backup/retain_min to 10
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting backup/archive_dir to /etc/lvm/archive
>Jul  1 11:08:05 taft-04 lvm[7083]: Setting backup/backup_dir to /etc/lvm/backup
>Jul  1 11:08:05 taft-04 lvm[7083]: Locking memory
>Jul  1 11:08:05 taft-04 lvm[7083]: memlock_count inc to 1
>Jul  1 11:08:05 taft-04 lvm[7083]: Parsing: _memlock_inc
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: Creating 65rpnvNP (1)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: cluster_resume: Setting recovery_halted = 0
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 3
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 3
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: server_id=dead, server_valid=0, 65rpnvNP
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: trigger = LRT_GET_RESYNC_WORK
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 4
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 4
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 0
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_SELECTION(11): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 3
>Jul  1 11:08:06 taft-04 lvm[7083]: Monitoring mirror device helter_skelter-syncd_primary_2legs_2 for events 
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 2
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 2
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 2
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 3
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 4
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 4
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_MASTER_ASSIGN(12): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 3
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_SELECTION(11): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 2
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 2
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_SELECTION(11): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 3
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror: LRT_SELECTION(11): (65rpnvNP)
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   starter     : 4
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   co-ordinator: 1
>Jul  1 11:08:06 taft-04 kernel: dm-cmirror:   node_count  : 4
>Jul  1 11:09:17 taft-04 qarshd[11743]: Talking to peer 10.15.80.47:33940
>Jul  1 11:09:17 taft-04 qarshd[11743]: Running cmdline: lvs -o copy_percent --noheadings helter_skelter/syncd_primary_2legs_1 
>Jul  1 11:09:17 taft-04 qarshd[11743]: That's enough 
>Jul  1 11:09:17 taft-04 qarshd[11746]: Talking to peer 10.15.80.47:33941
>Jul  1 11:09:17 taft-04 qarshd[11746]: Running cmdline: lvs -o copy_percent --noheadings helter_skelter/syncd_primary_2legs_2 
>Jul  1 11:09:17 taft-04 qarshd[11746]: That's enough 
>Jul  1 11:09:34 taft-04 qarshd[11753]: Talking to peer 10.15.80.47:33957
>Jul  1 11:09:34 taft-04 qarshd[11753]: Running cmdline: mkdir -p /mnt/syncd_primary_2legs_1 
>Jul  1 11:09:34 taft-04 qarshd[11753]: That's enough 
>Jul  1 11:09:34 taft-04 qarshd[11755]: Talking to peer 10.15.80.47:33958
>Jul  1 11:09:34 taft-04 qarshd[11755]: Running cmdline: mount -t gfs /dev/helter_skelter/syncd_primary_2legs_1 /mnt/syncd_primary_2legs_1 
>Jul  1 11:09:34 taft-04 kernel: Lock_Harness 2.6.9-80.3 (built Jun  4 2008 11:59:08) installed
>Jul  1 11:09:34 taft-04 kernel: GFS 2.6.9-80.3 (built Jun  4 2008 11:59:21) installed
>Jul  1 11:09:34 taft-04 kernel: GFS: Trying to join cluster "lock_dlm", "TAFT:gfs1"
>Jul  1 11:09:34 taft-04 kernel: Lock_DLM (built Jun  4 2008 11:59:10) installed
>Jul  1 11:09:36 taft-04 kernel: GFS: fsid=TAFT:gfs1.3: Joined cluster. Now mounting FS...
>Jul  1 11:09:36 taft-04 kernel: GFS: fsid=TAFT:gfs1.3: jid=3: Trying to acquire journal lock...
>Jul  1 11:09:36 taft-04 kernel: GFS: fsid=TAFT:gfs1.3: jid=3: Looking at journal...
>Jul  1 11:09:36 taft-04 kernel: GFS: fsid=TAFT:gfs1.3: jid=3: Done
>Jul  1 11:09:36 taft-04 lvm[7083]: helter_skelter-syncd_primary_2legs_1 is now in-sync 
>Jul  1 11:09:36 taft-04 qarshd[11755]: That's enough 
>Jul  1 11:09:36 taft-04 qarshd[11784]: Talking to peer 10.15.80.47:33959
>Jul  1 11:09:36 taft-04 qarshd[11784]: Running cmdline: mkdir -p /mnt/syncd_primary_2legs_2 
>Jul  1 11:09:36 taft-04 qarshd[11784]: That's enough 
>Jul  1 11:09:36 taft-04 qarshd[11786]: Talking to peer 10.15.80.47:33960
>Jul  1 11:09:36 taft-04 qarshd[11786]: Running cmdline: mount -t gfs /dev/helter_skelter/syncd_primary_2legs_2 /mnt/syncd_primary_2legs_2 
>Jul  1 11:09:36 taft-04 kernel: GFS: Trying to join cluster "lock_dlm", "TAFT:gfs2"
>Jul  1 11:09:38 taft-04 kernel: GFS: fsid=TAFT:gfs2.3: Joined cluster. Now mounting FS...
>Jul  1 11:09:38 taft-04 kernel: GFS: fsid=TAFT:gfs2.3: jid=3: Trying to acquire journal lock...
>Jul  1 11:09:38 taft-04 kernel: GFS: fsid=TAFT:gfs2.3: jid=3: Looking at journal...
>Jul  1 11:09:38 taft-04 kernel: GFS: fsid=TAFT:gfs2.3: jid=3: Done
>Jul  1 11:09:38 taft-04 lvm[7083]: helter_skelter-syncd_primary_2legs_2 is now in-sync 
>Jul  1 11:09:38 taft-04 qarshd[11786]: That's enough 
>Jul  1 11:09:46 taft-04 qarshd[11819]: Talking to peer 10.15.80.47:33973
>Jul  1 11:09:46 taft-04 qarshd[11819]: Running cmdline: mkdir -p /mnt/syncd_primary_2legs_1/checkit 
>Jul  1 11:09:46 taft-04 qarshd[11819]: That's enough 
>Jul  1 11:09:46 taft-04 qarshd[11821]: Talking to peer 10.15.80.47:33974
>Jul  1 11:09:46 taft-04 qarshd[11821]: Running cmdline: /usr/tests/sts-rhel4.7/bin/checkit -w /mnt/syncd_primary_2legs_1/checkit -f /tmp/checkit_syncd_primary_2legs_1 -n 100 
>Jul  1 11:09:47 taft-04 qarshd[11821]: That's enough 
>Jul  1 11:09:47 taft-04 qarshd[11823]: Talking to peer 10.15.80.47:33975
>Jul  1 11:09:47 taft-04 qarshd[11823]: Running cmdline: mkdir -p /mnt/syncd_primary_2legs_2/checkit 
>Jul  1 11:09:47 taft-04 qarshd[11823]: That's enough 
>Jul  1 11:09:47 taft-04 qarshd[11825]: Talking to peer 10.15.80.47:33976
>Jul  1 11:09:47 taft-04 qarshd[11825]: Running cmdline: /usr/tests/sts-rhel4.7/bin/checkit -w /mnt/syncd_primary_2legs_2/checkit -f /tmp/checkit_syncd_primary_2legs_2 -n 100 
>Jul  1 11:09:48 taft-04 qarshd[11825]: That's enough 
>Jul  1 11:09:48 taft-04 qarshd[11827]: Talking to peer 10.15.80.47:33978
>Jul  1 11:09:48 taft-04 qarshd[11827]: Running cmdline: /usr/tests/sts-rhel4.7/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/syncd_primary_2legs_2/taft-04_load | /usr/tests/sts-rhel4.7/bin/xdoio -v 
>Jul  1 11:09:48 taft-04 qarshd[11831]: Talking to peer 10.15.80.47:33984
>Jul  1 11:09:48 taft-04 qarshd[11831]: Running cmdline: /usr/tests/sts-rhel4.7/bin/xiogen -f buffered -m random -s write,writev -t 100 -T 1000000  -F 1000000:/mnt/syncd_primary_2legs_1/taft-04_load | /usr/tests/sts-rhel4.7/bin/xdoio -v 
>Jul  1 11:09:51 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:09:51 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:09:54 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:09:54 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:09:57 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:09:57 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:10:00 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:10:00 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:10:03 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:10:03 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:10:06 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:10:06 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:10:06 taft-04 qarshd[11841]: Talking to peer 10.15.80.47:33988
>Jul  1 11:10:06 taft-04 qarshd[11841]: Running cmdline: echo offline > /sys/block/sde/device/state 
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 qarshd[11841]: That's enough 
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 lvm[7083]: Mirror device, 253:7, has failed. 
>Jul  1 11:10:06 taft-04 lvm[7083]: Device failure in helter_skelter-syncd_primary_2legs_2 
>Jul  1 11:10:06 taft-04 lvm[7083]: Parsing: vgreduce --config devices{ignore_suspended_devices=1} --removemissing helter_skelter
>Jul  1 11:10:06 taft-04 lvm[7083]: Reloading config files
>Jul  1 11:10:06 taft-04 lvm[7083]: Wiping internal VG cache
>Jul  1 11:10:06 taft-04 lvm[7083]: Loading config file: /etc/lvm/lvm.conf
>Jul  1 11:10:06 taft-04 lvm[7083]: WARNING: dev_open(/etc/lvm/lvm.conf) called while suspended
>Jul  1 11:10:06 taft-04 lvm[7083]: Opened /etc/lvm/lvm.conf RO
>Jul  1 11:10:06 taft-04 lvm[7083]: Closed /etc/lvm/lvm.conf
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/syslog to 1
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/level to 0
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/verbose to 0
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/indent to 1
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/prefix to   
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/command_names to 0
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/test to 0
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting log/overwrite to 0
>Jul  1 11:10:06 taft-04 lvm[7083]: log/activation not found in config: defaulting to 0
>Jul  1 11:10:06 taft-04 lvm[7083]: Logging initialised at Tue Jul  1 11:10:06 2008
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/umask to 63
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting devices/dir to /dev
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/proc to /proc
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/activation to 1
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 lvm[7083]: global/suffix not found in config: defaulting to 1
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/units to h
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting activation/readahead to auto
>Jul  1 11:10:06 taft-04 lvm[7083]: devices/preferred_names not found in config file: using built-in preferences
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 lvm[7083]: Matcher built with 3 dfa states
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting devices/ignore_suspended_devices to 1
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting devices/cache_dir to /etc/lvm/cache
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting devices/write_cache_state to 1
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised format: lvm1
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised format: pool
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised format: lvm2
>Jul  1 11:10:06 taft-04 lvm[7083]: global/format not found in config: defaulting to lvm2
>Jul  1 11:10:06 taft-04 lvm[7083]: lvmcache: initialised VG #orphans_lvm1
>Jul  1 11:10:06 taft-04 lvm[7083]: lvmcache: initialised VG #orphans_pool
>Jul  1 11:10:06 taft-04 lvm[7083]: lvmcache: initialised VG #orphans_lvm2
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised segtype: striped
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised segtype: zero
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised segtype: error
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting dmeventd/snapshot_library to libdevmapper-event-lvm2snapshot.so
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/library_dir to /usr/lib64
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised segtype: snapshot
>Jul  1 11:10:06 taft-04 lvm[7083]: Another thread is handling an event.  Waiting...
>Jul  1 11:10:06 taft-04 lvm[7083]: Initialised segtype: mirror
>Jul  1 11:10:06 taft-04 lvm[7083]: Processing: vgreduce --config devices{ignore_suspended_devices=1} --removemissing helter_skelter
>Jul  1 11:10:06 taft-04 lvm[7083]: O_DIRECT will be used
>Jul  1 11:10:06 taft-04 lvm[7083]: Setting global/locking_type to 3
>Jul  1 11:10:06 taft-04 lvm[7083]: Cluster locking selected.
>Jul  1 11:10:06 taft-04 lvm[7083]: Finding volume group "helter_skelter"
>Jul  1 11:10:06 taft-04 lvm[7083]: Locking VG V_helter_skelter PW B (0x4)
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 last message repeated 2 times
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 last message repeated 2 times
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 last message repeated 2 times
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 last message repeated 2 times
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 last message repeated 2 times
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:06 taft-04 last message repeated 2 times
>Jul  1 11:10:06 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:06 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 880/3P2k7cK1
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 871/65rpnvNP
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:07 taft-04 last message repeated 2 times
>Jul  1 11:10:07 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:07 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 881/3P2k7cK1
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 882/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 last message repeated 2 times
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 882/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 882/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 882/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 last message repeated 3 times
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 882/3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: LOG INFO:
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   uuid: LVM-3ch9Xihjw0QZaZmo6GdgncXiQK1GGfXDWUlsf7AkKeD0aNfVvPbOJdHe3P2k7cK1
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   uuid_ref    : 1
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   log type    : disk
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:  ?region_count: 1600
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:  ?sync_count  : 0
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:  ?sync_search : 0
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   in_sync     : YES
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   suspended   : NO
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   recovery_halted : NO
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   server_id   : 1
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror:   server_valid: YES
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: cluster_presuspend: recovery halted on 3P2k7cK1(1)
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: cluster_postsuspend
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:08 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:08 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:09 taft-04 qarshd[11827]: Nothing to do 
>Jul  1 11:10:09 taft-04 qarshd[11831]: Nothing to do 
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: LRT_MASTER_LEAVING(13): (3P2k7cK1)
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   starter     : 1
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   co-ordinator: 0
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   node_count  : 3
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: LRT_ELECTION(10): (3P2k7cK1)
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   starter     : 1
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   co-ordinator: 57005
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   node_count  : 3
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: LRT_SELECTION(11): (3P2k7cK1)
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   starter     : 1
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   co-ordinator: 57005
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   node_count  : 3
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: LRT_MASTER_ASSIGN(12): (3P2k7cK1)
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   starter     : 1
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   co-ordinator: 57005
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror:   node_count  : 1
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Removing 3P2k7cK1 (1)
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: 0 region user structures freed
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 872/65rpnvNP
>Jul  1 11:10:10 taft-04 last message repeated 2 times
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:10 taft-04 last message repeated 4 times
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:10 taft-04 last message repeated 2 times
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:10 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:10 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:11 taft-04 last message repeated 2 times
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 last message repeated 3 times
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:11 taft-04 last message repeated 2 times
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 last message repeated 3 times
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:11 taft-04 last message repeated 2 times
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: LOG INFO:
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   uuid: LVM-3ch9Xihjw0QZaZmo6GdgncXiQK1GGfXDFHuWUQA2h2mBe8QyrXSDLqBj65rpnvNP
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   uuid_ref    : 1
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   log type    : disk
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:  ?region_count: 1600
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:  ?sync_count  : 0
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:  ?sync_search : 0
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   in_sync     : YES
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   suspended   : NO
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   recovery_halted : NO
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   server_id   : 1
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror:   server_valid: YES
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: cluster_presuspend: recovery halted on 65rpnvNP(1)
>Jul  1 11:10:11 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: Notifying server(1) of sync change: 873/65rpnvNP
>Jul  1 11:10:11 taft-04 kernel: dm-cmirror: cluster_postsuspend
>Jul  1 11:10:30 taft-04 kernel: CMAN: Started transition, generation 16
>Jul  1 11:10:31 taft-04 kernel: CMAN: Finished transition, generation 16
>Jul  1 11:10:31 taft-04 fenced[6406]: fencing deferred to taft-02
>Jul  1 11:11:15 taft-04 kernel: dm-cmirror: A cluster mirror log member has failed.
>Jul  1 11:11:16 taft-04 kernel: GFS: fsid=TAFT:gfs2.3: jid=0: Trying to acquire journal lock...
>Jul  1 11:11:16 taft-04 kernel: GFS: fsid=TAFT:gfs1.3: jid=0: Trying to acquire journal lock...
>Jul  1 11:11:16 taft-04 kernel: GFS: fsid=TAFT:gfs1.3: jid=0: Busy
>Jul  1 11:11:16 taft-04 kernel: GFS: fsid=TAFT:gfs2.3: jid=0: Looking at journal...
>Jul  1 11:12:01 taft-04 kernel: scsi1 (0:4): rejecting I/O to offline device
>Jul  1 11:12:01 taft-04 kernel: dm-cmirror: Removing 65rpnvNP (1)
>Jul  1 11:12:01 taft-04 kernel: dm-cmirror: 0 region user structures freed
>Jul  1 12:02:19 taft-04 dhclient: DHCPREQUEST on eth0 to 10.15.89.100 port 67
>Jul  1 12:02:20 taft-04 dhclient: DHCPACK from 10.15.89.100
>Jul  1 12:02:20 taft-04 dhclient: bound to 10.15.89.70 -- renewal in 8794 seconds.
>Jul  1 13:18:41 taft-04 kernel: SysRq : Show State
>Jul  1 13:18:41 taft-04 kernel: 
>Jul  1 13:18:41 taft-04 kernel:                                                        sibling
>Jul  1 13:18:41 taft-04 kernel:   task                 PC          pid father child younger older
>Jul  1 13:18:41 taft-04 kernel: init          S 000000000000000b     0     1      0     2               (NOTLB)
>Jul  1 13:18:41 taft-04 kernel: 00000101fffa1d58 0000000000000002 0000010037e25200 0000010037e25200 
>Jul  1 13:18:41 taft-04 kernel:        0000000000000000 0000000000000246 00000101fffa1e58 0000000180191e2d 
>Jul  1 13:18:41 taft-04 kernel:        000001000693d7f0 0000000000000b4f 
>Jul  1 13:18:41 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:41 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:41 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:41 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:41 taft-04 kernel: migration/0   S 00000100010687e0     0     2      1             3       (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101fff19ec8 0000000000000046 00000100010687e0 0000001900000072 
>Jul  1 13:18:50 taft-04 kernel:        0000010215c507f0 0000000000000072 0000010001061a40 0000000000000001 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff817f0 00000000000001bf 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80135895>{migration_thread+323} <ffffffff80135752>{migration_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ksoftirqd/0   S 0000000000000000     0     3      1             4     2 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101fff1df08 0000000000000046 00000101fff817f0 0000001900000078 
>Jul  1 13:18:50 taft-04 kernel:        000001020b421030 0000000000000078 0000000000000000 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff81030 0000000000000112 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8013d9f0>{ksoftirqd+0} <ffffffff8013da2c>{ksoftirqd+60} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: migration/1   S 00000100010607e0     0     4      1             5     3 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101fff1fec8 0000000000000046 00000100010607e0 0000001900000075 
>Jul  1 13:18:50 taft-04 kernel:        000001020ad707f0 0000000000000075 0000010001069a40 0000000100000001 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff827f0 00000000000001d3 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80135895>{migration_thread+323} <ffffffff80135752>{migration_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ksoftirqd/1   S 0000000000000000     0     5      1             6     4 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101fff51f08 0000000000000046 00000101fff827f0 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff50000 0000000000000246 000001000106bcc0 0000000100000000 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff82030 0000000000000106 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8013d9f0>{ksoftirqd+0} <ffffffff8013da2c>{ksoftirqd+60} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: migration/2   S 00000100010607e0     0     6      1             7     5 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101fff55ec8 0000000000000046 00000100010607e0 0000001900000072 
>Jul  1 13:18:50 taft-04 kernel:        0000010215c507f0 0000000000000072 0000010001071a40 0000000200000001 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff297f0 00000000000001e3 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80135895>{migration_thread+323} <ffffffff80135752>{migration_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ksoftirqd/2   S 0000000000000000     0     7      1             8     6 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000100dff07f08 0000000000000046 00000101fff297f0 000000000000006d 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff31010 0000000000000246 0000010001073cc0 0000000200000000 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff29030 000000000000011b 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8013d9f0>{ksoftirqd+0} <ffffffff8013da2c>{ksoftirqd+60} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: migration/3   S 00000100010687e0     0     8      1             9     7 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000100dff09ec8 0000000000000046 00000100010607e0 0000001900000086 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff65030 0000000000000086 0000010001079a40 0000000300000001 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff657f0 000000000000019f 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80135895>{migration_thread+323} <ffffffff80135752>{migration_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ksoftirqd/3   S 0000000000000000     0     9      1            10     8 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000100dff4df08 0000000000000046 00000101fff657f0 ffffffffa0007411 
>Jul  1 13:18:50 taft-04 kernel:        0000010217867074 0000010217889bc0 0000000000000000 0000000300000000 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff65030 0000000000000125 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa0007411>{:scsi_mod:scsi_io_completion+497} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8013d9f0>{ksoftirqd+0} <ffffffff8013da2c>{ksoftirqd+60} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: events/0      S ffffffff8022f841     0    10      1    14      11     9 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101ffcb3e68 0000000000000046 ffffffff803e7e80 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000246 0000000000000246 0000000100d31342 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc907f0 00000000000001f1 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8022f841>{flush_to_ldisc+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: events/1      S ffffffff80164d4a     0    11      1  6380      12    10 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101ffcb5e68 0000000000000046 000001000693d030 000001000106bec0 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000005 0000000000000246 0000000100d3170b 0000000100000246 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc90030 0000000000001713 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80164d4a>{cache_reap+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: events/2      S ffffffff80164d4a     0    12      1   552      13    11 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101ffcc9e68 0000000000000046 00000101fff807f0 0000010001073ec0 
>Jul  1 13:18:50 taft-04 kernel:        000000000000000a 0000000000000246 0000000100d3131c 0000000200000246 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc927f0 0000000000000aa8 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80164d4a>{cache_reap+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: events/3      S ffffffff80164d4a     0    13      1  1113      67    12 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101ffccbe68 0000000000000046 00000101fff80030 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006a22758 0000000000000246 0000000100d31734 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc92030 0000000000000c55 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80164d4a>{cache_reap+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: khelper       S 0000010213707c18     0    14     10            15       (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000101ffccde68 0000000000000046 00000100010687e0 0000001900000064 
>Jul  1 13:18:50 taft-04 kernel:        00000102117897f0 0000000000000064 0000010001069a40 0000000100000001 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc937f0 0000000000000258 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80148129>{__call_usermodehelper+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: kacpid        S 00000100dff60040     0    15     10            63    14 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 00000100dfdf1e68 0000000000000046 0000000000000000 0000000000000071 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000002 ffffffff80483800 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc93030 0000000000000851 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134752>{__wake_up_common+67} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: kblockd/0     S 0000010217889da0     0    63     10            64    15 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037c4be68 0000000000000046 676e696e6e616373 0000001900000076 
>Jul  1 13:18:50 taft-04 kernel:        000001020bc5b7f0 0000000000000076 0000010001061a40 0000000017889bc0 
>Jul  1 13:18:50 taft-04 kernel:        00000100dfdd6030 000000000000062e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80254483>{blk_unplug_work+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: kblockd/1     S ffffffff80254483     0    64     10            65    63 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037c6de68 0000000000000046 0000010217818000 0000010037da11c0 
>Jul  1 13:18:50 taft-04 kernel:        00000102179da548 ffffffffa004786b 000001021795a080 0000000117888140 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c017f0 0000000000000483 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa004786b>{:qla2xxx:qla2x00_queuecommand+1212} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80254483>{blk_unplug_work+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: kblockd/2     S ffffffff80254483     0    65     10            66    64 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037c6fe68 0000000000000046 676e696e6e616373 0000000000000200 
>Jul  1 13:18:50 taft-04 kernel:        676e697070696b73 ffffffff80140a00 0000010006be7400 0000000217889bc0 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c01030 0000000000000710 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a00>{__mod_timer+237} <ffffffff80254483>{blk_unplug_work+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134752>{__wake_up_common+67} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: kblockd/3     S ffffffff80254483     0    66     10           102    65 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037c71e68 0000000000000046 0000010217818000 0000010037da11c0 
>Jul  1 13:18:50 taft-04 kernel:        00000102179da548 ffffffffa004786b 000001021795a080 0000000317888140 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c337f0 000000000000037e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa004786b>{:qla2xxx:qla2x00_queuecommand+1212} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80254483>{blk_unplug_work+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: khubd         S ffffffff804267c0     0    67      1           104    13 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037c95e78 0000000000000046 0000010215e13270 000000198029c3be 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff0f7f0 0000000100000076 0000010001079a40 0000000300000001 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c33030 000000000000839b 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80293e59>{hub_thread+2579} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80293446>{hub_thread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: pdflush       S ffffffff8014c590     0   102     10           103    66 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037c97ec8 0000000000000046 0000010037c647f0 000000000000006a 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c647f0 0000000001061120 ffffffff80483800 00000000010607e0 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c647f0 0000000000000588 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801349d2>{set_user_nice+259} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80161c10>{pdflush+191} <ffffffff80161b51>{pdflush+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: pdflush       D ffffffff8014c590     0   103     10           105   102 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037cb9c68 0000000000000046 000000000000ffff 0000000000190000 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000206 ffffffffa00b9d34 0000000000209ae1 0000000200000008 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c64030 0000000000001866 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa00b9d34>{:dm_mod:dm_request+335} <ffffffff8025445f>{generic_unplug_device+24} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa00bbd1a>{:dm_mod:dm_table_unplug_all+49} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8031869a>{io_schedule+38} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017ccd8>{__wait_on_buffer+125} <ffffffff8017cb5e>{bh_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017cb5e>{bh_wake_function+0} <ffffffffa029ed1f>{:gfs:gfs_logbh_wait+49} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa02b3ef2>{:gfs:disk_commit+794} <ffffffff8013303e>{try_to_wake_up+876} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa02b40c3>{:gfs:log_refund+111} <ffffffffa02b45da>{:gfs:log_flush_internal+510} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80181996>{sync_supers+167} <ffffffff801611cd>{wb_kupdate+39} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80161c94>{pdflush+323} <ffffffff801611a6>{wb_kupdate+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80161b51>{pdflush+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: aio/0         S 0000010037c48900     0   105     10           106   103 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037cdde68 0000000000000046 0000000000000000 000000000000006b 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000008 ffffffff80483800 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c77030 0000000000000723 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134752>{__wake_up_common+67} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: kswapd0       S 0000010037cbbf08     0   104      1           252    67 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037cbbeb8 0000000000000046 0000010037cbbe08 000000000000007d 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000000 ffffffff80483800 000000008013502e 
>Jul  1 13:18:50 taft-04 kernel:        0000010037c777f0 000000000000109e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801681db>{kswapd+231} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801680f4>{kswapd+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: aio/1         S 0000010037c48980     0   106     10           107   105 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037ce1e68 0000000000000046 0000000000000000 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000008 000000000000231e 0000000100000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010037cae7f0 0000000000000c99 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134752>{__wake_up_common+67} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: aio/2         S 0000010037c48a00     0   107     10           108   106 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037ce3e68 0000000000000046 0000000000000000 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000008 000000000000231e 0000000200000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010037cae030 0000000000000e44 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134752>{__wake_up_common+67} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: aio/3         S 0000010037c48a80     0   108     10           555   107 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037ce5e68 0000000000000046 0000000000000000 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000008 000000000000231e 0000000300000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010037caf7f0 0000000000000b45 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134752>{__wake_up_common+67} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: kseriod       S 0000000000000000     0   252      1           488   104 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010037d29ec8 0000000000000046 0000010037d29e18 0000010000000079 
>Jul  1 13:18:50 taft-04 kernel:        ffffffff80132b35 0000000000000000 ffffffff80483800 0000000037d17418 
>Jul  1 13:18:50 taft-04 kernel:        00000100dfdd67f0 000000000000115e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80132b35>{deactivate_task+21} <ffffffff80246e9c>{serio_thread+469} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff8013b570>{do_exit+3319} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80246cc7>{serio_thread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: scsi_eh_0     S 0000010037c1a7c0     0   488      1           512   252 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217871df8 0000000000000046 00000101fff20c00 00000100dff94bd2 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000018 0000000000000006 0000000000001b4e 00000002803f7d60 
>Jul  1 13:18:50 taft-04 kernel:        00000100dff947f0 0000000000003327 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803170fb>{__down_interruptible+203} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80318b1b>{__down_failed_interruptible+53} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0006a20>{:scsi_mod:.text.lock.scsi_error+45} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa0005bc6>{:scsi_mod:scsi_error_handler+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: scsi_eh_1     S 0000000000574730     0   512      1           513   488 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217887df8 0000000000000046 00000102179d8000 0000010006b64bd2 
>Jul  1 13:18:50 taft-04 kernel:        000000000000001e 0000000000000006 0000000000001b12 00000002803f7d60 
>Jul  1 13:18:50 taft-04 kernel:        0000010006b647f0 0000000000002894 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803170fb>{__down_interruptible+203} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80318b1b>{__down_failed_interruptible+53} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0006a20>{:scsi_mod:.text.lock.scsi_error+45} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa0005bc6>{:scsi_mod:scsi_error_handler+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: qla2300_1_dpc S 00000102179d8000     0   513      1           599   512 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217875e58 0000000000000046 00000102179da625 00000001a0072960 
>Jul  1 13:18:50 taft-04 kernel:        0000000500000000 ffffffff00000000 00000000040000a0 0000000300084000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006a997f0 000000000000039d 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa00521a3>{:qla2xxx:__qla2x00_marker+518} <ffffffff803170fb>{__down_interruptible+203} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80318b1b>{__down_failed_interruptible+53} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0047d9d>{:qla2xxx:.text.lock.qla_os+15} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0047244>{:qla2xxx:qla2x00_do_dpc+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ata/0         S 00000102178bf080     0   552     12           553       (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217b8be68 0000000000000046 0000000000000000 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        000001000693d7f0 0000000000000074 0000010001061a40 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006a887f0 0000000000000f6a 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: ata/1         S ffffffff8014c590     0   553     12           554   552 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217b83e68 0000000000000046 0000010217b83e18 ffffffff80132a85 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc909d0 0000000000000008 000000000000231e 0000000100000009 
>Jul  1 13:18:50 taft-04 kernel:        0000010006ad57f0 000000000000135c 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80132a85>{recalc_task_prio+337} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014874d>{worker_thread+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ata/2         S ffffffff8014c590     0   554     12           556   553 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217b89e68 0000000000000046 0000000100000001 0000000000000002 
>Jul  1 13:18:50 taft-04 kernel:        000001021792c9d0 0000000000000007 0000000000001f36 0000000200000100 
>Jul  1 13:18:50 taft-04 kernel:        0000010006afb030 0000000000000bb3 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: ata/3         S ffffffff8014c590     0   555     10                 108 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217b85e68 0000000000000046 0000000100000001 0000000000000002 
>Jul  1 13:18:50 taft-04 kernel:        000001021792c9d0 0000000000000009 0000000000002706 0000000300000100 
>Jul  1 13:18:50 taft-04 kernel:        000001021782a7f0 00000000000012b9 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: ata_aux       S ffffffff8014c590     0   556     12                 554 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010217b87e68 0000000000000046 0000000100000001 0000000000000002 
>Jul  1 13:18:50 taft-04 kernel:        000001021792c9d0 0000000000000007 0000000000001f36 0000000200000001 
>Jul  1 13:18:50 taft-04 kernel:        0000010006ad5030 0000000000000947 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014866b>{worker_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014874d>{worker_thread+226} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014866b>{worker_thread+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: kjournald     R  running task       0   599      1          2407   513 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: kauditd       S ffffffff8014c590     0  1113     13          6442       (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010216071ea8 0000000000000046 0000000407e98c35 00000101ffc92030 
>Jul  1 13:18:50 taft-04 kernel:        00000100010787e0 0000000000000003 00000102179267f0 0000000300000000 
>Jul  1 13:18:50 taft-04 kernel:        00000102179267f0 0000000000000b88 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014c590>{keventd_create_kthread+0} <ffffffff80155b20>{kauditd_thread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80155c9f>{kauditd_thread+383} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: udevd         S 00000100010687e0     0  2407      1          2985   599 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010215b7bd58 0000000000000006 00000101fff80030 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        0000010215dfc030 0000000000000074 0000010001079a40 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d3d7f0 00000000000014d3 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff802ba163>{datagram_poll+39} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e573>{do_select+939} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e8f2>{sys_select+820} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: kjournald     S 0000000000000000     0  2985      1          3535  2407 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010215147e78 0000000000000046 0000010215147dc8 00000102157dfaf8 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff59c98 0000000000000005 0000000000000000 00000003000003e8 
>Jul  1 13:18:50 taft-04 kernel:        000001021676a7f0 000000000000411f 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa00d2b54>{:jbd:kjournald+506} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffffa00d2954>{:jbd:commit_timeout+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa00d295a>{:jbd:kjournald+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: dhclient      S 0000000000000007     0  3535      1          3633  2985 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102147a7d58 0000000000000006 ffffffff803e7e80 00000102147a7e88 
>Jul  1 13:18:50 taft-04 kernel:        00000102147a7d58 0000001000000000 000000d014561980 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        000001021792c030 0000000000004ec8 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: syslogd       D 0000010214605ef8     0  3633      1          3637  3535 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010214605d98 0000000000000006 00000100010607e0 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010214605d18 ffffffff80132b13 0000000000000008 0000000200000005 
>Jul  1 13:18:50 taft-04 kernel:        0000010037caf030 0000000000000357 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80132b13>{activate_task+124} <ffffffffa00d2d7c>{:jbd:log_wait_commit+202} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa00cdaaf>{:jbd:journal_stop+536} <ffffffffa00cdb05>{:jbd:journal_force_commit+32} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa00ea949>{:ext3:ext3_force_commit+36} <ffffffff8019b724>{__writeback_single_inode+363} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8019ba63>{sync_inode+52} <ffffffffa00e1043>{:ext3:ext3_sync_file+191} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017d212>{sys_fsync+155} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: klogd         S 000000000000006b     0  3637      1          3650  3633 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010214603bc8 0000000000000006 0000010037caf030 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        000001020be9b7f0 0000000000000074 0000010214603b98 0000000380132a85 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff0f7f0 0000000000000626 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80163706>{cache_alloc_refill+390} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8013601c>{prepare_to_wait_exclusive+21} <ffffffff80312441>{unix_wait_for_peer+163} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801347a8>{__wake_up+54} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff80312f53>{unix_dgram_sendmsg+957} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b41f4>{sock_aio_write+306} <ffffffff8017bba7>{do_sync_write+178} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801389ca>{do_syslog+482} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801968e9>{dnotify_parent+34} <ffffffff8017bcb5>{vfs_write+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bd8a>{sys_write+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: irqbalance    S 0000000000000000     0  3650      1          3661  3637 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102144e9ec8 0000000000000006 0000000000000001 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        000001021fb19b70 ffffffff8016a3a4 0000000000001000 000000039556d000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006bef030 0000000000010cd0 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8016a3a4>{unmap_vmas+1345} <ffffffff80140a38>{__mod_timer+293} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80318863>{schedule_timeout+396} <ffffffff80141462>{process_timeout+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801415a5>{sys_nanosleep+192} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: portmap       S 7fffffffffffffff     0  3661      1          3680  3650 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021407fe68 0000000000000006 0000000000000000 ffffffff802b576e 
>Jul  1 13:18:50 taft-04 kernel:        000001021407e000 00000010802b389f 000000d01407fe48 0000000100000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010217829030 0000000000004d85 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff802b576e>{sys_sendmsg+534} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80135efc>{add_wait_queue+18} <ffffffff802dd5f3>{tcp_poll+44} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018ec97>{sys_poll+588} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: rpc.statd     S 0000000000000009     0  3680      1          3711  3661 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102145ffd58 0000000000000002 0000000000000246 ffffffffa00ec9bf 
>Jul  1 13:18:50 taft-04 kernel:        00000000000000d0 000001021454400c 000000d00000003b 0000000200000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215ddb7f0 0000000000007f1a 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa00ec9bf>{:ext3:ext3fs_dirhash+873} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802dd5f3>{tcp_poll+44} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80191eb2>{dput+189} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: rpc.idmapd    S 0000000000000000     0  3711      1          3805  3680 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010214329e58 0000000000000002 ffffffff803e7e80 0000001900000076 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d3d030 0000000000000076 0000010001069a40 00000000178297f0 
>Jul  1 13:18:50 taft-04 kernel:        00000102178297f0 0000000000000709 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff801a1079>{sys_epoll_wait+393} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141da6>{recalc_sigpending+15} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: acpid         S 0000000000000005     0  3805      1          3814  3711 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010213d03d58 0000000000000002 00000000486a3df2 0000000018044cd8 
>Jul  1 13:18:50 taft-04 kernel:        00000000486a3df2 0000000018044cd8 00000102147e6508 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        00000102159717f0 0000000000026516 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801968e9>{dnotify_parent+34} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: cupsd         S 0000000000000004     0  3814      1          3880  3805 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010213ccfd58 0000000000000002 00000000486a7408 000000002e7cdd18 
>Jul  1 13:18:50 taft-04 kernel:        00000000486a7408 0000000000000246 0000010213da28d0 0000000100000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010215c50030 0000000000030293 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80191eb2>{dput+189} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: sshd          S 0000000000000004     0  3880      1  6255    3901  3814 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010213a15d58 0000000000000002 ffffffff803e7e80 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        00000102167bd118 0000000000000246 000000d214648680 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        000001021676a030 00000000002cc507 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff802dd5f3>{tcp_poll+44} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e573>{do_select+939} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e8f2>{sys_select+820} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: xinetd        S 0000000000000007     0  3901      1  6576    3919  3880 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102139dbd58 0000000000000002 000001020c8817f0 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        0000010037caf030 0000000000000246 0000010001061a40 00000002802b3d83 
>Jul  1 13:18:50 taft-04 kernel:        00000102160d87f0 00000000000017b1 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff802dd5f3>{tcp_poll+44} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e573>{do_select+939} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e8f2>{sys_select+820} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: sendmail      S 0000000000000005     0  3919      1          3928  3901 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010213597d58 0000000000000002 0000010213597d28 ffffffff801ae9ab 
>Jul  1 13:18:50 taft-04 kernel:        fffffffffffffff4 0000000000000246 0000000000000000 000000010bc94015 
>Jul  1 13:18:50 taft-04 kernel:        000001021782a030 0000000000001483 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801ae9ab>{proc_root_lookup+49} <ffffffff80140a38>{__mod_timer+293} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80318863>{schedule_timeout+396} <ffffffff80141462>{process_timeout+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e573>{do_select+939} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e8f2>{sys_select+820} <ffffffff80191e2d>{dput+56} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: sendmail      S 0000000000000000     0  3928      1          3948  3919 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102136d9f68 0000000000000006 0000000100d7c326 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000246 ffffffff80140a38 0000000000000004 000000030036f0a4 
>Jul  1 13:18:50 taft-04 kernel:        0000010006bef7f0 000000000000ca8d 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff8013cbc2>{do_setitimer+332} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80145342>{sys_pause+23} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: gpm           S 0000000000000006     0  3948      1          4081  3928 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021300dd58 0000000000000006 0000000000000003 00000100010787e0 
>Jul  1 13:18:50 taft-04 kernel:        00000100010787e0 0000000000000002 00000001dfdd8918 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d1c030 00000000000081ff 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801968e9>{dnotify_parent+34} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: crond         S 0000000000000000     0  4081      1          4099  3948 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010213753ec8 0000000000000002 000000000000006e 00000102152ddad8 
>Jul  1 13:18:50 taft-04 kernel:        00000102152ddad8 ffffffff80191eb2 0000010213753e58 0000000313753ef8 
>Jul  1 13:18:50 taft-04 kernel:        000001021614f7f0 0000000000001eae 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80191eb2>{dput+189} <ffffffff80140a38>{__mod_timer+293} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80318863>{schedule_timeout+396} <ffffffff80141462>{process_timeout+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801415a5>{sys_nanosleep+192} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: xfs           S 0000000000000004     0  4099      1          4116  4081 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102137c7d58 0000000000000002 00000101fff80030 00000102137c7e88 
>Jul  1 13:18:50 taft-04 kernel:        00000102137c7d58 0000000000000000 0000000000000066 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        00000102154057f0 00000000000041dc 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: atd           S 0000000000000000     0  4116      1          4126  4099 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102132d9ec8 0000000000000002 0000010211a2c760 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        00000101fff3f800 0000000000000246 00000000078d3954 000000018018df2c 
>Jul  1 13:18:50 taft-04 kernel:        0000010215fc97f0 000000000000f1c2 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff801415a5>{sys_nanosleep+192} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: dbus-daemon-1 S 7fffffffffffffff     0  4126      1          4137  4116 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021324be68 0000000000000002 000001021324bda0 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        000000000058dae0 0000000000000800 000001021324bc68 0000000100000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d3d030 0000000000004ef1 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018ec97>{sys_poll+588} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017c241>{sys_writev+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: hald          S 000000000000069a     0  4137      1          4162  4126 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212c4de68 0000000000000002 0000000000000001 0000010037d8c410 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000246 0000000000000000 0000000200000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010215f727f0 00000000000007b5 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff802ba13c>{datagram_poll+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018ec97>{sys_poll+588} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: modclusterd   S 00000000000001f5     0  4162      1          4163  4137 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212ecfe68 0000000000000002 ffffffff803e7e80 8000000214076065 
>Jul  1 13:18:50 taft-04 kernel:        0000010212826b40 0000010216224030 0000002a95568888 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010216224030 00000000000002c8 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018ec97>{sys_poll+588} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: modclusterd   S 00000000000001f5     0  4163      1          4164  4162 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212d9de68 0000000000000002 ffffffff803e7e80 800000020b63c065 
>Jul  1 13:18:50 taft-04 kernel:        0000010211252050 0000010215c507f0 0000002a9580a040 0000000016718040 
>Jul  1 13:18:50 taft-04 kernel:        0000010215c507f0 00000000000001b0 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018ec97>{sys_poll+588} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: modclusterd   S 00000000000001f5     0  4164      1          4205  4163 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102128b3e68 0000000000000002 ffffffff803e7e80 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        000000004007f9f7 0000000000000246 00000102128b3e18 00000000000001b7 
>Jul  1 13:18:50 taft-04 kernel:        0000010213270030 00000000000008df 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018ec97>{sys_poll+588} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8017bd37>{sys_read+96} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: oddjobd       S 0000000000000004     0  4205      1          4225  4164 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212a8bd58 0000000000000006 00000101fff80030 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d3d030 0000000000000074 0000010001069a40 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d1c7f0 0000000000002349 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801968e9>{dnotify_parent+34} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: saslauthd     S 00000102130808d0     0  4225      1  4226    4237  4205 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212acde78 0000000000000002 0000010212859888 0000010217894030 
>Jul  1 13:18:50 taft-04 kernel:        0000000000511b90 0000010216719b40 00000102166d5688 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010217894030 0000000000003c16 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8018fdb1>{__posix_lock_file+1147} <ffffffff801910ca>{fcntl_setlk+473} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018d387>{sys_fcntl+736} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: saslauthd     S 0000000000000000     0  4226   4225          4227       (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212affd58 0000000000000002 0000002a95568b30 0000001900000078 
>Jul  1 13:18:50 taft-04 kernel:        0000010217894030 0000000000000078 0000010001061a40 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010217926030 000000000000ec92 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff8013601c>{prepare_to_wait_exclusive+21} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b9a39>{skb_recv_datagram+373} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff80312978>{unix_accept+99} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b4f6f>{sys_accept+201} <ffffffff80110e1d>{error_exit+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: saslauthd     S 000001021517a600     0  4227   4225          4228  4226 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212b21e78 0000000000000002 0000010212b07de0 0000001900000078 
>Jul  1 13:18:50 taft-04 kernel:        0000010217894030 0000000000000078 0000010001061a40 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215f73030 000000000000d5e6 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8018fdb1>{__posix_lock_file+1147} <ffffffff801910ca>{fcntl_setlk+473} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018d387>{sys_fcntl+736} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: saslauthd     S 000001021517a420     0  4228   4225          4229  4227 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212b43e78 0000000000000006 0000010212b18de0 000000190000007a 
>Jul  1 13:18:50 taft-04 kernel:        0000010217894030 000000000000007a 0000010001061a40 00000000000000d0 
>Jul  1 13:18:50 taft-04 kernel:        0000010215f737f0 000000000000cef3 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8018fdb1>{__posix_lock_file+1147} <ffffffff801910ca>{fcntl_setlk+473} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018d387>{sys_fcntl+736} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: saslauthd     S 0000010217a80d80     0  4229   4225                4228 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212b65e78 0000000000000002 0000010212b3bde0 000000190000007a 
>Jul  1 13:18:50 taft-04 kernel:        0000010217894030 000000000000007a 0000010001061a40 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010217b187f0 000000000000e0d0 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8018fdb1>{__posix_lock_file+1147} <ffffffff801910ca>{fcntl_setlk+473} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018d387>{sys_fcntl+736} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ricci         S 00000000000003e9     0  4237      1          4242  4225 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102129a9e68 0000000000000006 ffffffff803e7e80 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        0000010037caf030 0000000000000246 0000010001061a40 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        00000100dff94030 00000000000002fa 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018ec97>{sys_poll+588} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: login         S 0000000000000000     0  4242      1  9652    4243  4237 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102166f3eb8 0000000000000002 000001020bc56000 00000102167c0ff8 
>Jul  1 13:18:50 taft-04 kernel:        0000000100000000 0000000000000000 ffffffff8014925c 0000000200000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006be6030 00000000000077f5 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8014925c>{attach_pid+31} <ffffffff801ef299>{__up_read+16} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801248c9>{do_page_fault+577} <ffffffff8013c679>{do_wait+3413} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014522b>{sys_rt_sigaction+97} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: mingetty      S 000001021192ccc0     0  4243      1          4244  4242 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102128efd68 0000000000000006 0000000000000020 0000001900000078 
>Jul  1 13:18:50 taft-04 kernel:        0000010215dfc7f0 0000000000000078 0000010001079a40 00000003128efdd8 
>Jul  1 13:18:50 taft-04 kernel:        000001021792c7f0 000000000007d3ca 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80238c11>{complement_pos+12} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80135efc>{add_wait_queue+18} <ffffffff80231b7d>{read_chan+1080} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff801347a8>{__wake_up+54} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8022c766>{tty_read+230} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bac0>{vfs_read+207} <ffffffff8017bd1c>{sys_read+69} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: mingetty      S 00000100010687e0     0  4244      1          4245  4243 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010214c2fd68 0000000000000002 0000000000000020 000000190000006e 
>Jul  1 13:18:50 taft-04 kernel:        00000102119d87f0 000000000000006e 0000010001079a40 0000000314c2fdd8 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d4f7f0 0000000000007f1d 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80231b7d>{read_chan+1080} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80319010>{.text.lock.spinlock+2} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8022c766>{tty_read+230} <ffffffff8017bac0>{vfs_read+207} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bd1c>{sys_read+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: mingetty      S 00000102127560c0     0  4245      1          4246  4244 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010216641d68 0000000000000006 0000000000000020 000000190000007a 
>Jul  1 13:18:50 taft-04 kernel:        00000102158a37f0 000000000000007a 0000010001061a40 0000000016641dd8 
>Jul  1 13:18:50 taft-04 kernel:        0000010215971030 000000000001e8ed 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80231b7d>{read_chan+1080} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801347a8>{__wake_up+54} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8022c766>{tty_read+230} <ffffffff8017bac0>{vfs_read+207} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bd1c>{sys_read+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: mingetty      S 0000010211dc66c0     0  4246      1          4247  4245 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212aabd68 0000000000000002 0000000000000020 0000001900000076 
>Jul  1 13:18:50 taft-04 kernel:        000001021792c7f0 0000000000000076 0000010001079a40 0000000312aabdd8 
>Jul  1 13:18:50 taft-04 kernel:        00000102167db7f0 000000000003a5c7 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80231b7d>{read_chan+1080} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801347a8>{__wake_up+54} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8022c766>{tty_read+230} <ffffffff8017bac0>{vfs_read+207} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bd1c>{sys_read+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: mingetty      S 000001021192c2c0     0  4247      1          4248  4246 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212b89d68 0000000000000006 0000000000000020 0000001900000004 
>Jul  1 13:18:50 taft-04 kernel:        00000102158a37f0 0000000100000078 0000010001079a40 0000000312b89dd8 
>Jul  1 13:18:50 taft-04 kernel:        0000010215dfc7f0 000000000003dd4f 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80231b7d>{read_chan+1080} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8022c766>{tty_read+230} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bac0>{vfs_read+207} <ffffffff8017bd1c>{sys_read+69} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: mingetty      S ffffffff803173f8     0  4248      1          6316  4247 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021299fd68 0000000000000006 0000000000000020 0000000000000005 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 ffffffff8023d54e 0000000f00000067 000000001299fdd8 
>Jul  1 13:18:50 taft-04 kernel:        00000102158a37f0 0000000000027d32 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8023d54e>{do_con_write+7903} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80135efc>{add_wait_queue+18} <ffffffff80231b7d>{read_chan+1080} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff801347a8>{__wake_up+54} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8022c766>{tty_read+230} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bac0>{vfs_read+207} <ffffffff8017bd1c>{sys_read+69} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: sshd          S 000000000000000a     0  6255   3880  6257               (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021250fd58 0000000000000002 0000000500000001 0000010212a9bab0 
>Jul  1 13:18:50 taft-04 kernel:        0000010212a9bab0 0000000000000246 0000010212a9bab0 0000000000000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215f72030 0000000000001a2c 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff8022c225>{tty_ldisc_try+60} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8022c358>{tty_ldisc_deref+103} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801968e9>{dnotify_parent+34} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: bash          S 0000000000000001     0  6257   6255                     (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 00000102114c5d68 0000000000000002 0000000000000246 ffffffff80232525 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000010006be67f0 ffffffff80134701 0000000200000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006be67f0 000000000000a503 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80232525>{tty_wait_until_sent+237} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff803187d8>{schedule_timeout+257} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80231b7d>{read_chan+1080} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801347a8>{__wake_up+54} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8022c766>{tty_read+230} <ffffffff8017bac0>{vfs_read+207} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bd1c>{sys_read+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ccsd          S 000000000000000a     0  6316      1          6317  4248 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010211dc3d58 0000000000000006 0000010215ce9c80 0000000000000014 
>Jul  1 13:18:50 taft-04 kernel:        0000010211dc3d90 0000000000000246 0000000000000246 00000000802b708f 
>Jul  1 13:18:50 taft-04 kernel:        00000102160d8030 0000000000000ea1 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80191e2d>{dput+56} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: ccsd          S 000000000000000b     0  6317      1          6378  6316 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010211c0bd58 0000000000000006 0000000000000001 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010214ba7850 0000000000000246 0000010211c0be80 00000002114572c0 
>Jul  1 13:18:50 taft-04 kernel:        00000102132707f0 000000000002c1b7 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff802ba163>{datagram_poll+39} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e573>{do_select+939} <ffffffff8018e10d>{__pollwait+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e8f2>{sys_select+820} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: cman_comms    S 0000000000190000     0  6378      1          6379  6317 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212575d98 0000000000000046 00000101fff807f0 0000010215ddb030 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000001 0000010212575d38 00000002801347a8 
>Jul  1 13:18:50 taft-04 kernel:        0000010215ddb030 0000000000000467 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa0244c1f>{:cman:send_to_user_port+758} <ffffffffa0246829>{:cman:cluster_kthread+299} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8017bccb>{vfs_write+248} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa02466fe>{:cman:cluster_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: cman_serviced S ffffffff8014c590     0  6380     11          6443       (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021154ff08 0000000000000046 00000102117897f0 0000000000000008 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000000 0000000000000001 0000000000000246 0000000100000003 
>Jul  1 13:18:50 taft-04 kernel:        00000102162247f0 00000000000001e4 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa02537ed>{:cman:process_recoveries+556} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa024fb11>{:cman:serviced+0} <ffffffffa024fc39>{:cman:serviced+296} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: cman_memb     S 0000000000000003     0  6379      1          6389  6378 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010211293e38 0000000000000046 00000000fff80030 00000102126afc80 
>Jul  1 13:18:50 taft-04 kernel:        0000010207a187f0 0000000000000074 0000000000000003 00000003a02650e0 
>Jul  1 13:18:50 taft-04 kernel:        0000010212701030 0000000000000379 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa024e3e9>{:cman:membership_kthread+2891} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80133468>{schedule_tail+55} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa024d89e>{:cman:membership_kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: cman_hbeat    S 0000010211293dc8     0  6389      1          6406  6379 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212205f18 0000000000000046 0000000000000000 0000001900000073 
>Jul  1 13:18:50 taft-04 kernel:        000001020be6c7f0 00000102126afc80 ffffffffa0265400 000000010000006e 
>Jul  1 13:18:50 taft-04 kernel:        0000010215405030 000000000000095e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa024b858>{:cman:hello_kthread+478} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa024b67a>{:cman:hello_kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: fenced        S ffffffff801102f6     0  6406      1          6441  6389 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010211201f28 0000000000000006 0000010216370dc0 ffffff9111d6d030 
>Jul  1 13:18:50 taft-04 kernel:        0000000000507d7d 0000000000000000 0000010211201ec8 0000000000000010 
>Jul  1 13:18:50 taft-04 kernel:        0000010211d6d030 00000000000090b9 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801102f6>{system_call+126} <ffffffff8010fe2f>{sys_rt_sigsuspend+199} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8011066b>{ptregscall_common+103} 
>Jul  1 13:18:50 taft-04 kernel: clvmd         S 000000000000000c     0  6441      1          6469  6406 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021219fd58 0000000000000006 00000101fff80030 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        000005ee01eead6c 00000102117897f0 00000100010687e0 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010215dfc030 000000000000099e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: clvmd         S 0000000040a000d0     0  6469      1          6470  6441 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010211b21e58 0000000000000006 0000000000000246 0000000000000003 
>Jul  1 13:18:50 taft-04 kernel:        0000010211b21de8 ffffffff801347a8 ffffffff804a9c50 0000000000000206 
>Jul  1 13:18:50 taft-04 kernel:        0000010006a88030 0000000000000e9d 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801347a8>{__wake_up+54} <ffffffffa0269d23>{:dlm:dlm_read+259} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bac0>{vfs_read+207} <ffffffff8017bd1c>{sys_read+69} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: clvmd         D 00000100010787e0     0  6470      1          7083  6469 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010213707bf8 0000000000000006 00000000000718e1 0000001900000069 
>Jul  1 13:18:50 taft-04 kernel:        00000102162247f0 0000000000000069 0000010001069a40 0000000100000246 
>Jul  1 13:18:50 taft-04 kernel:        00000102117897f0 00000000000027d9 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa00bd364>{:dm_mod:dev_suspend+0} <ffffffff80317f68>{wait_for_completion+167} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa02545e1>{:cman:kcl_leave_service+249} <ffffffffa028ce13>{:dm_cmirror:cluster_dtr+660} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa01022a3>{:dm_mirror:dm_destroy_dirty_log+10} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0104749>{:dm_mirror:free_context+216} <ffffffffa00baf69>{:dm_mod:table_destroy+72} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa00ba486>{:dm_mod:dm_swap_table+119} <ffffffffa00bd364>{:dm_mod:dev_suspend+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa00bd60e>{:dm_mod:dev_suspend+682} <ffffffffa00be6a2>{:dm_mod:ctl_ioctl+602} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801847f2>{sys_newstat+32} <ffffffff8018db79>{sys_ioctl+853} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: clvmd         S 0000000000000001     0 11843      1         11835 11799 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020740fd78 0000000000000006 0000000000000016 0000000000000001 
>Jul  1 13:18:50 taft-04 kernel:        ffffffffa0288d50 0000000000000000 ffffffffa0288d58 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        000001020b0467f0 0000000000013d9a 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801347a8>{__wake_up+54} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8016e482>{find_extend_vma+22} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014db77>{do_futex+531} <ffffffffa026aa4c>{:dlm:dlm_write+2538} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff8014dfcd>{sys_futex+203} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bda5>{sys_write+96} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: dlm_astd      S 0000010037d95800     0  6442     13          6444  1113 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021125dec8 0000000000000046 ffffffff803e7e80 0000001900000072 
>Jul  1 13:18:50 taft-04 kernel:        0000010213270030 0000000000000072 0000010001061a40 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000010006b64030 000000000000010c 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffffa02ee2b3>{:lock_dlm:lock_bast+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0268a43>{:dlm:dlm_astd+314} <ffffffffa0268909>{:dlm:dlm_astd+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: dlm_recvd     S ffffffff8014c590     0  6443     11          6445  6380 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 0000010212479ef8 0000000000000046 ffffffffffffffff 0000000000000000 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000246 ffffffffa0270b9c 0000010001061a40 0000000000000072 
>Jul  1 13:18:50 taft-04 kernel:        0000010215d4f030 0000000000000286 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa0270b9c>{:dlm:receive_from_sock+849} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0271037>{:dlm:dlm_recvd+170} <ffffffffa0270f8d>{:dlm:dlm_recvd+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: dlm_sendd     S ffffffff802dda0d     0  6444     13         11776  6442 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021247be78 0000000000000046 0000000000000001 0000010211d07630 
>Jul  1 13:18:50 taft-04 kernel:        0000000000000072 ffffffff802b68ed 0000000000000072 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        0000010006afb7f0 0000000000000c1d 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff802b68ed>{release_sock+16} <ffffffff802dda0d>{tcp_sendpage+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0271342>{:dlm:dlm_sendd+218} <ffffffffa0271268>{:dlm:dlm_sendd+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: dlm_recoverd  S 0000000000000000     0  6445     11         11775  6443 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021247fea8 0000000000000046 0000000000000025 00000101fff245e8 
>Jul  1 13:18:50 taft-04 kernel:        00000000fffffffb 0000000000000000 0000000000000216 000000038013303e 
>Jul  1 13:18:50 taft-04 kernel:        0000010006a99030 0000000000000af6 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa02688a7>{:dlm:wake_astd+27} <ffffffffa0278613>{:dlm:dlm_recoverd+60} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa02785d7>{:dlm:dlm_recoverd+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: btimed        S ffffffff803173f8     0  6576   3901                     (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001021224baf8 0000000000000002 ffffffff803e7e80 0000010006a22000 
>Jul  1 13:18:50 taft-04 kernel:        0000010214b12c80 00000101ffc9d600 0000010006a221d4 0000000000001000 
>Jul  1 13:18:50 taft-04 kernel:        0000010217b18030 0000000000000103 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff802bcdb6>{dev_queue_xmit+541} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8013601c>{prepare_to_wait_exclusive+21} <ffffffff802b9a39>{skb_recv_datagram+373} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802f6dd0>{udp_recvmsg+118} <ffffffff802b7462>{sock_common_recvmsg+48} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b3f08>{sock_recvmsg+284} <ffffffff80189ad5>{link_path_walk+176} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80191eb2>{dput+189} <ffffffff8017c948>{fget+57} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff802b3af3>{sockfd_lookup+16} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b536b>{sys_recvfrom+182} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: dmeventd      S 0000000000000006     0  7083      1         11616  6470 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020c8f3d58 0000000000000002 00000101fff80030 0000001900000069 
>Jul  1 13:18:50 taft-04 kernel:        00000101ffc92030 0000000000000246 0000010001079a40 00000003ffffff00 
>Jul  1 13:18:50 taft-04 kernel:        000001021243b7f0 00000000000002f3 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffff8018e573>{do_select+939} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8018e10d>{__pollwait+0} <ffffffff8018e8f2>{sys_select+820} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel: dmeventd      S 0000000000000002     0 11660      1         11728 11616 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020c5a3d78 0000000000000002 00000100010787e0 0000000000000003 
>Jul  1 13:18:50 taft-04 kernel:        000001020c5a3cf8 ffffffff80132b13 00000100010787e0 0000000000000003 
>Jul  1 13:18:50 taft-04 kernel:        0000010213743030 0000000000004b52 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80132b13>{activate_task+124} <ffffffff803187d8>{schedule_timeout+257} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8016e482>{find_extend_vma+22} <ffffffff80135efc>{add_wait_queue+18} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014db77>{do_futex+531} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014dfcd>{sys_futex+203} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: dmeventd      S 0000000000000002     0 11728      1         11778 11660 (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020b489bb8 0000000000000002 00000102099b87f0 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        00000102099b87f0 0000000000000074 0000010001079a40 0000000315194380 
>Jul  1 13:18:50 taft-04 kernel:        000001020be9b030 0000000000001b1a 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff80135fc0>{prepare_to_wait+21} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff803138d0>{unix_stream_recvmsg+622} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff802b40ae>{sock_aio_read+297} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b41f4>{sock_aio_write+306} <ffffffff8017b9c5>{do_sync_read+178} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b5293>{sys_sendto+208} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801968e9>{dnotify_parent+34} <ffffffff8017bad3>{vfs_read+226} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8017bd1c>{sys_read+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: bash          R  running task       0  9652   4242                     (NOTLB)
>Jul  1 13:18:50 taft-04 kernel: cluster_log_s S 0000000000000000     0 11616      1         11660  7083 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020c43b9d8 0000000000000046 00000101fff807f0 0000001900000074 
>Jul  1 13:18:50 taft-04 kernel:        0000010207982030 0000000000000074 0000010001071a40 000000021255fb00 
>Jul  1 13:18:50 taft-04 kernel:        000001020bf42030 00000000000001d0 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff803187d8>{schedule_timeout+257} <ffffffff802cc2b8>{qdisc_restart+30} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8013601c>{prepare_to_wait_exclusive+21} <ffffffff802b9a39>{skb_recv_datagram+373} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff802db918>{ip_push_pending_frames+855} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff802f6dd0>{udp_recvmsg+118} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff802b7462>{sock_common_recvmsg+48} <ffffffff802b3f08>{sock_recvmsg+284} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8013303e>{try_to_wake_up+876} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80132a85>{recalc_task_prio+337} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80140a38>{__mod_timer+293} <ffffffffa028f154>{:dm_cmirror:my_recvmsg+265} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa028f03c>{:dm_cmirror:set_sigusr1+0} <ffffffffa028f03c>{:dm_cmirror:set_sigusr1+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa028dbce>{:dm_cmirror:cluster_log_serverd+784} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa028d8be>{:dm_cmirror:cluster_log_serverd+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: dlm_recoverd  S 00000100010687e0     0 11775     11         11777  6445 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020d3c9ea8 0000000000000046 0000000000000025 0000001900000073 
>Jul  1 13:18:50 taft-04 kernel:        0000010215ddb030 0000000000000073 0000010001079a40 0000000300000246 
>Jul  1 13:18:50 taft-04 kernel:        000001020b046030 0000000000014fe6 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff801ef299>{__up_read+16} <ffffffffa02688a7>{:dlm:wake_astd+27} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa0278613>{:dlm:dlm_recoverd+60} <ffffffffa02785d7>{:dlm:dlm_recoverd+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: lock_dlm1     S 0000010213d46c80     0 11776     13         11791  6444 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020b3d7e58 0000000000000046 000001020ad70030 0000001900000069 
>Jul  1 13:18:50 taft-04 kernel:        000001020ad70030 0000000000000069 0000010001069a40 000000010abb26e0 
>Jul  1 13:18:50 taft-04 kernel:        000001020bf887f0 0000000000000101 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa02a96c0>{:gfs:drop_bh+358} <ffffffffa02f2560>{:lock_dlm:dlm_async+356} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffffa02f23fc>{:lock_dlm:dlm_async+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c567>{kthread+200} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c49f>{kthread+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel:        
>Jul  1 13:18:50 taft-04 kernel: lock_dlm2     S 0000000000000000     0 11777     11         11789 11775 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020c547e58 0000000000000046 000001020bf887f0 0000001900000069 
>Jul  1 13:18:50 taft-04 kernel:        000001020bf887f0 0000000000000069 0000010001069a40 0000000108ff06e0 
>Jul  1 13:18:50 taft-04 kernel:        000001020ad70030 0000000000000094 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa02f2560>{:lock_dlm:dlm_async+356} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa02f23fc>{:lock_dlm:dlm_async+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: gfs_scand     S 0000000000000000     0 11778      1         11779 11728 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020b063ea8 0000000000000046 000001000693d030 000001020c3d3224 
>Jul  1 13:18:50 taft-04 kernel:        000001020c3d31f8 0000000000000001 0000000000000000 0000000100000000 
>Jul  1 13:18:50 taft-04 kernel:        000001020bf89030 0000000000029b84 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffffa029c6d1>{:gfs:gfs_scand+137} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029c648>{:gfs:gfs_scand+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:50 taft-04 kernel: gfs_glockd    S 0000000000000000     0 11779      1         11780 11778 (L-TLB)
>Jul  1 13:18:50 taft-04 kernel: 000001020cc6fed8 0000000000000046 0000000000000001 ffffffffa02a9867 
>Jul  1 13:18:50 taft-04 kernel:        0000010207c29800 000001020b02bda8 ffffffffa02ea660 000000030b02beb0 
>Jul  1 13:18:50 taft-04 kernel:        000001020bf88030 000000000000071e 
>Jul  1 13:18:50 taft-04 kernel: Call Trace:<ffffffffa02a9867>{:gfs:gfs_glock_drop_th+290} <ffffffffa02a8255>{:gfs:unlock_on_glock+37} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa029c7d6>{:gfs:gfs_glockd+185} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:50 taft-04 kernel:        <ffffffffa029c71d>{:gfs:gfs_glockd+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel:        
>Jul  1 13:18:51 taft-04 kernel: gfs_recoverd  S 0000000000000000     0 11780      1         11781 11779 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020b235ea8 0000000000000046 00000101fff80030 0000001900000069 
>Jul  1 13:18:51 taft-04 kernel:        00000102162247f0 0000000000000069 0000010001069a40 000000030000000d 
>Jul  1 13:18:51 taft-04 kernel:        000001020ca207f0 0000000000000189 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffffa029c897>{:gfs:gfs_recoverd+137} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029c80e>{:gfs:gfs_recoverd+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_logd      S 0000000000000000     0 11781      1         11782 11780 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020bcbbe38 0000000000000046 00000101fff807f0 ffffffffa02a8a9e 
>Jul  1 13:18:51 taft-04 kernel:        0000000300000000 000001020bcbbe30 000001020bcbbe38 0000000200000246 
>Jul  1 13:18:51 taft-04 kernel:        000001020ca20030 000000000000021f 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffffa02a8a9e>{:gfs:glock_wait_internal+710} <ffffffff80140a38>{__mod_timer+293} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80318863>{schedule_timeout+396} <ffffffff80141462>{process_timeout+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa029c9df>{:gfs:gfs_logd+252} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801d37a3>{dummy_d_instantiate+0} <ffffffffa029c8e3>{:gfs:gfs_logd+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_quotad    S 0000000000000000     0 11782      1         11783 11781 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020ce7fe98 0000000000000046 000001000693d030 0000001900000074 
>Jul  1 13:18:51 taft-04 kernel:        0000010207a187f0 0000000000000074 0000010001069a40 00000001010687e0 
>Jul  1 13:18:51 taft-04 kernel:        000001020bf897f0 000000000000014d 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffffa029cbde>{:gfs:gfs_quotad+428} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029ca32>{:gfs:gfs_quotad+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_inoded    S 0000000000000000     0 11783      1         11793 11782 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020c229ea8 0000000000000046 00000101fff80030 000001020c229ed8 
>Jul  1 13:18:51 taft-04 kernel:        0000000000000005 0000000000000000 0000000000000008 000000030c228000 
>Jul  1 13:18:51 taft-04 kernel:        000001020bc1a7f0 00000000000001be 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffffa029ccb8>{:gfs:gfs_inoded+137} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029cc2f>{:gfs:gfs_inoded+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: dlm_recoverd  S 0000000000000000     0 11789     11         11792 11777 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020b56bea8 0000000000000046 0000000000000025 0000010037dedde8 
>Jul  1 13:18:51 taft-04 kernel:        00000000fffffffb 0000000000000000 0000000000000216 000000018013303e 
>Jul  1 13:18:51 taft-04 kernel:        000001020b0b0030 00000000000109b4 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffffa02688a7>{:dlm:wake_astd+27} <ffffffffa0278613>{:dlm:dlm_recoverd+60} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02785d7>{:dlm:dlm_recoverd+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: lock_dlm1     S 0000000000000000     0 11791     13               11776 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020c241e58 0000000000000046 000001020cd4d790 0000001900000069 
>Jul  1 13:18:51 taft-04 kernel:        000001020be6c030 0000000000000069 0000010001061a40 0000000000000001 
>Jul  1 13:18:51 taft-04 kernel:        000001020be6c030 00000000000000c4 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffffa02f2560>{:lock_dlm:dlm_async+356} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02f23fc>{:lock_dlm:dlm_async+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: lock_dlm2     S 0000000000000000     0 11792     11               11789 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020b05fe58 0000000000000046 00000101fff80030 0000001900000069 
>Jul  1 13:18:51 taft-04 kernel:        000001021243b030 0000000000000069 0000010001079a40 0000000300000001 
>Jul  1 13:18:51 taft-04 kernel:        000001021243b030 0000000000000127 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffffa02f2560>{:lock_dlm:dlm_async+356} <ffffffff801360c4>{autoremove_wake_function+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801360c4>{autoremove_wake_function+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02f23fc>{:lock_dlm:dlm_async+0} <ffffffff8014c590>{keventd_create_kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8014c567>{kthread+200} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8014c590>{keventd_create_kthread+0} <ffffffff8014c49f>{kthread+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_scand     S 0000000000000000     0 11793      1         11794 11783 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020c4cfea8 0000000000000046 00000101fff80030 00000102086a78f4 
>Jul  1 13:18:51 taft-04 kernel:        00000102086a78c8 0000000000000001 0000000300000000 00000003107f2500 
>Jul  1 13:18:51 taft-04 kernel:        000001020be6c7f0 000000000002ad68 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff80318863>{schedule_timeout+396} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80141462>{process_timeout+0} <ffffffffa029c6d1>{:gfs:gfs_scand+137} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029c648>{:gfs:gfs_scand+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_glockd    D 0000010208e25b38     0 11794      1         11796 11793 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020cc1bd78 0000000000000046 000001021faf20a0 000001021faf20d8 
>Jul  1 13:18:51 taft-04 kernel:        000001021faf2110 000001021faf1e38 000001021faf1e70 000000031faf1ea8 
>Jul  1 13:18:51 taft-04 kernel:        000001020c92d030 000000000001edaa 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff803189bc>{__down_write+134} <ffffffffa02b440e>{:gfs:log_flush_internal+50} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02ab883>{:gfs:inode_go_sync+71} <ffffffffa02a9810>{:gfs:gfs_glock_drop_th+203} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02a8001>{:gfs:run_queue+314} <ffffffffa02a8255>{:gfs:unlock_on_glock+37} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02a834b>{:gfs:gfs_reclaim_glock+234} <ffffffffa029c75a>{:gfs:gfs_glockd+61} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80134701>{default_wake_function+0} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029c71d>{:gfs:gfs_glockd+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_recoverd  D 00000100010687e0     0 11796      1         11797 11794 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020d2df8c8 0000000000000046 000001020c519e40 0000001900000069 
>Jul  1 13:18:51 taft-04 kernel:        0000010037c337f0 0000000000000069 0000010001079a40 00000003a029ce54 
>Jul  1 13:18:51 taft-04 kernel:        000001020bf427f0 0000000000475cda 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80254426>{__generic_unplug_device+19} <ffffffff8025445f>{generic_unplug_device+24} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa00bbd1a>{:dm_mod:dm_table_unplug_all+49} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8031869a>{io_schedule+38} <ffffffff8017ccd8>{__wait_on_buffer+125} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8017cb5e>{bh_wake_function+0} <ffffffff8017cb5e>{bh_wake_function+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa029e383>{:gfs:gfs_dreread+154} <ffffffffa029e3ef>{:gfs:gfs_dread+40} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02c57c5>{:gfs:get_log_header+47} <ffffffff80132b13>{activate_task+124} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02c5875>{:gfs:find_good_lh+54} <ffffffffa02c5932>{:gfs:gfs_find_jhead+94} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801360cd>{autoremove_wake_function+9} <ffffffff80134752>{__wake_up_common+67} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801347a8>{__wake_up+54} <ffffffff80139134>{vprintk+498} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801391de>{printk+141} <ffffffffa02a939f>{:gfs:gfs_glock_nq_init+20} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02c5f79>{:gfs:gfs_recover_journal+190} <ffffffff80317d24>{thread_return+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80317d7c>{thread_return+88} <ffffffff80140b77>{del_timer+107} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80140c34>{del_singleshot_timer_sync+9} <ffffffff80318872>{schedule_timeout+411} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02c6567>{:gfs:gfs_check_journals+155} <ffffffffa029c850>{:gfs:gfs_recoverd+66} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029c80e>{:gfs:gfs_recoverd+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_logd      D 0000000000000000     0 11797      1         11798 11796 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020b08be38 0000000000000046 000001000693d030 0000001900000000 
>Jul  1 13:18:51 taft-04 kernel:        0000010207a187f0 0000000100000074 0000010001079a40 0000000100000002 
>Jul  1 13:18:51 taft-04 kernel:        00000101fff0f030 0000000000000fd3 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff803189bc>{__down_write+134} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02b3527>{:gfs:gfs_ail_empty+56} <ffffffffa029c930>{:gfs:gfs_logd+77} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8013b570>{do_exit+3319} <ffffffff8017bccb>{vfs_write+248} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80133468>{schedule_tail+55} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff801d37a3>{dummy_d_instantiate+0} <ffffffffa029c8e3>{:gfs:gfs_logd+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_quotad    D ffffffff803173f8     0 11798      1         11799 11797 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020b1b7e98 0000000000000046 000001000693d030 0000000000000000 
>Jul  1 13:18:51 taft-04 kernel:        000005cce382befc 000001020b0b07f0 00000100010687e0 0000000100000003 
>Jul  1 13:18:51 taft-04 kernel:        000001020c92d7f0 0000000000000ec2 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80140a38>{__mod_timer+293} <ffffffff803189bc>{__down_write+134} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02c541e>{:gfs:gfs_quota_sync+226} <ffffffffa029cb4d>{:gfs:gfs_quotad+283} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fd3>{child_rip+8} <ffffffffa029ca32>{:gfs:gfs_quotad+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel: gfs_inoded    D 0000000000000000     0 11799      1         11843 11798 (L-TLB)
>Jul  1 13:18:51 taft-04 kernel: 000001020bfe1e98 0000000000000046 000001020bfe1ea8 ffffffff80317d7c 
>Jul  1 13:18:51 taft-04 kernel:        000001000693d030 000001020be9b7f0 00000100010607e0 0000000100000002 
>Jul  1 13:18:51 taft-04 kernel:        00000102178947f0 0000000000000ea4 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80317d7c>{thread_return+88} <ffffffff803189bc>{__down_write+134} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02cbcad>{:gfs:unlinked_find+115} <ffffffffa02cc30c>{:gfs:gfs_unlinked_dealloc+25} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa029cc71>{:gfs:gfs_inoded+66} <ffffffff80110fd3>{child_rip+8} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa029cc2f>{:gfs:gfs_inoded+0} <ffffffff80110fcb>{child_rip+0} 
>Jul  1 13:18:51 taft-04 kernel:        
>Jul  1 13:18:51 taft-04 kernel: xdoio         D ffffff00107c2000     0 11835      1               11843 (NOTLB)
>Jul  1 13:18:51 taft-04 kernel: 0000010207aefbf8 0000000000000006 0000000000000001 0000000000000001 
>Jul  1 13:18:51 taft-04 kernel:        0000000000000000 0000000000000000 000001020b175dc0 0000000307aefbc0 
>Jul  1 13:18:51 taft-04 kernel:        000001020be9b7f0 0000000000037df9 
>Jul  1 13:18:51 taft-04 kernel: Call Trace:<ffffffff80316fc3>{__down+147} <ffffffff80134701>{default_wake_function+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8015e025>{generic_file_write_nolock+158} <ffffffff80318ae1>{__down_failed+53} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa029fa48>{:gfs:.text.lock.dio+95} <ffffffffa02cb4ec>{:gfs:gfs_trans_add_bh+205} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02bd7e3>{:gfs:do_write_buf+1138} <ffffffffa02bc657>{:gfs:walk_vm+278} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02bd371>{:gfs:do_write_buf+0} <ffffffffa02bd371>{:gfs:do_write_buf+0} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffffa02bda42>{:gfs:__gfs_write+201} <ffffffff8017bca2>{vfs_write+207} 
>Jul  1 13:18:51 taft-04 kernel:        <ffffffff8017bd8a>{sys_write+69} <ffffffff801102f6>{system_call+126} 
>Jul  1 13:18:51 taft-04 kernel:

Actions: View

Attachments on bug 254225: 172454 | 172456 | 251951 | 251961 | 251971 | 251981 | 251991 | 252001 | 252031 | 252041 | 252051 | 252061 | 310702 | 310703 | 310704