Attachment 934870 Details for Bug 1138452 – log file

log file

messages (text/plain), 50.57 KB, created by Corey Marthaler on 2014-09-05 16:55:49 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Corey Marthaler

Created: 2014-09-05 16:55:49 UTC

Size: 50.57 KB

patch

obsolete

>Sep  5 01:35:25 host-074 qarshd[683]: Running cmdline: echo offline > /sys/block/sdh/device/state
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 lvm[3854]: Log device 253:2 has failed (D).
>Sep  5 01:35:25 host-074 lvm[3854]: Device failure in revolution_9-mirror_1.
>Sep  5 01:35:25 host-074 xinetd[1619]: START: qarsh pid=685 from=::ffff:10.15.80.47
>Sep  5 01:35:25 host-074 xinetd[1619]: EXIT: qarsh status=0 pid=683 duration=0(sec)
>Sep  5 01:35:25 host-074 qarshd[685]: Talking to peer 10.15.80.47:56446 (IPv4)
>Sep  5 01:35:25 host-074 qarshd[685]: Running cmdline: echo offline > /sys/block/sdg/device/state
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 16104947712: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 16105054208: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 0: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 16104947712: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 16105054208: Input/output error
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 0: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 4096: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
>Sep  5 01:35:25 host-074 xinetd[1619]: EXIT: qarsh status=0 pid=685 duration=0(sec)
>Sep  5 01:35:25 host-074 xinetd[1619]: START: qarsh pid=687 from=::ffff:10.15.80.47
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 lvm[3854]: Couldn't find device with uuid P2DTMe-vQrr-ArL8-D3s8-jhzd-BLOy-vtM6Vp.
>Sep  5 01:35:25 host-074 lvm[3854]: Couldn't find device with uuid iHLb4R-mzpv-1zFj-HMI1-62tV-6ZR0-sulMkq.
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 qarshd[687]: Talking to peer 10.15.80.47:56447 (IPv4)
>Sep  5 01:35:25 host-074 qarshd[687]: Running cmdline: echo offline > /sys/block/sda/device/state
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 xinetd[1619]: EXIT: qarsh status=0 pid=687 duration=0(sec)
>Sep  5 01:35:25 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:25 host-074 lvm[3854]: /dev/sda1: read failed after 0 of 512 at 4096: Input/output error
>Sep  5 01:35:25 host-074 lvm[3854]: intermediate VG write failed.
>Sep  5 01:35:25 host-074 lvm[3854]: Repair of mirrored device revolution_9-mirror_1 failed.
>Sep  5 01:35:25 host-074 lvm[3854]: Failed to remove faulty devices in revolution_9-mirror_1.
>Sep  5 01:35:28 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:28 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:28 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 01:35:30 host-074 xinetd[1619]: START: qarsh pid=697 from=::ffff:10.15.80.47
>Sep  5 01:35:30 host-074 qarshd[697]: Talking to peer 10.15.80.47:56453 (IPv4)
>Sep  5 01:35:30 host-074 qarshd[697]: Running cmdline: tail -n1 /var/log/messages | cut -c 1-12
>Sep  5 01:35:30 host-074 xinetd[1619]: EXIT: qarsh status=0 pid=697 duration=0(sec)
>Sep  5 01:35:30 host-074 xinetd[1619]: START: qarsh pid=701 from=::ffff:10.15.80.47
>Sep  5 01:35:30 host-074 qarshd[701]: Talking to peer 10.15.80.47:56454 (IPv4)
>Sep  5 01:35:30 host-074 qarshd[701]: Running cmdline: dd if=/dev/zero of=/mnt/mirror_1/ddfile count=10 bs=4M
>Sep  5 01:35:30 host-074 xinetd[1619]: EXIT: qarsh status=0 pid=701 duration=0(sec)
>Sep  5 01:35:30 host-074 xinetd[1619]: START: qarsh pid=703 from=::ffff:10.15.80.47
>Sep  5 01:35:30 host-074 qarshd[703]: Talking to peer 10.15.80.47:56455 (IPv4)
>Sep  5 01:35:30 host-074 qarshd[703]: Running cmdline: sync
>Sep  5 01:39:14 host-074 kernel: INFO: task flush-253:6:2184 blocked for more than 120 seconds.
>Sep  5 01:39:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:39:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:39:14 host-074 kernel: flush-253:6   D 0000000000000000     0  2184      2 0x00000080
>Sep  5 01:39:14 host-074 kernel: ffff880020d8f840 0000000000000046 ffff880020d8f7d0 ffffffff81041e98
>Sep  5 01:39:14 host-074 kernel: ffff880000000000 0000fea2edc4482c ffff880020d8f810 ffff88003cfff670
>Sep  5 01:39:14 host-074 kernel: 000000000b517908 00000001077e5118 ffff88003b8bd098 ffff880020d8ffd8
>Sep  5 01:39:14 host-074 kernel: Call Trace:
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a22a>] __wait_on_bit_lock+0x5a/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a308>] out_of_line_wait_on_bit_lock+0x78/0x90
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4fd0>] ? end_buffer_async_write+0x0/0x190
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4e46>] __lock_buffer+0x36/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c6145>] __block_write_full_page+0x305/0x330
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4fd0>] ? end_buffer_async_write+0x0/0x190
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c6250>] block_write_full_page_endio+0xe0/0x120
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03cfd10>] ? buffer_unmapped+0x0/0x20 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c62a5>] block_write_full_page+0x15/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03d08bd>] ext3_ordered_writepage+0x1ed/0x240 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811380e7>] __writepage+0x17/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811393ad>] write_cache_pages+0x1fd/0x4c0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811380d0>] ? __writepage+0x0/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81139694>] generic_writepages+0x24/0x30
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811396d5>] do_writepages+0x35/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811bab3d>] writeback_single_inode+0xdd/0x290
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811baf3d>] writeback_sb_inodes+0xbd/0x170
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811bb09b>] writeback_inodes_wb+0xab/0x1b0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811bb493>] wb_writeback+0x2f3/0x410
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81088062>] ? del_timer_sync+0x22/0x30
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811bb66b>] wb_do_writeback+0xbb/0x240
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811bb853>] bdi_writeback_task+0x63/0x1b0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81148390>] ? bdi_start_fn+0x0/0x100
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81148416>] bdi_start_fn+0x86/0x100
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81148390>] ? bdi_start_fn+0x0/0x100
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
>Sep  5 01:39:14 host-074 kernel: INFO: task kjournald:2192 blocked for more than 120 seconds.
>Sep  5 01:39:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:39:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:39:14 host-074 kernel: kjournald     D 0000000000000000     0  2192      2 0x00000080
>Sep  5 01:39:14 host-074 kernel: ffff880008e15c50 0000000000000046 ffff880008e15be0 ffffffff81041e98
>Sep  5 01:39:14 host-074 kernel: ffff880000000000 0000fea3e7b879fe ffff880008e15c20 ffff88003cfff280
>Sep  5 01:39:14 host-074 kernel: 000000000b517980 ffffffffabf8928f ffff88003b257ab8 ffff880008e15fd8
>Sep  5 01:39:14 host-074 kernel: Call Trace:
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a45f>] __wait_on_bit+0x5f/0x90
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a508>] out_of_line_wait_on_bit+0x78/0x90
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c56>] __wait_on_buffer+0x26/0x30
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03b102f>] journal_commit_transaction+0x9ef/0x1300 [jbd]
>Sep  5 01:39:14 host-074 kernel: [<ffffffff810873dc>] ? lock_timer_base+0x3c/0x70
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03b6085>] kjournald+0xe5/0x240 [jbd]
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03b5fa0>] ? kjournald+0x0/0x240 [jbd]
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
>Sep  5 01:39:14 host-074 kernel: INFO: task xdoio:679 blocked for more than 120 seconds.
>Sep  5 01:39:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:39:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:39:14 host-074 kernel: xdoio         D 0000000000000000     0   679    678 0x00000080
>Sep  5 01:39:14 host-074 kernel: ffff880027435888 0000000000000082 ffff880027435818 ffffffff81041e98
>Sep  5 01:39:14 host-074 kernel: ffff880000000000 0000fea14de48a24 ffff880027435858 ffff88003a832b40
>Sep  5 01:39:14 host-074 kernel: 000000000b5176ec 00000001077e475d ffff880022c35ab8 ffff880027435fd8
>Sep  5 01:39:14 host-074 kernel: Call Trace:
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a22a>] __wait_on_bit_lock+0x5a/0xc0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c688c>] ? __block_prepare_write+0x11c/0x530
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a308>] out_of_line_wait_on_bit_lock+0x78/0x90
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c4e46>] __lock_buffer+0x36/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c5918>] __sync_dirty_buffer+0xc8/0xf0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c5953>] sync_dirty_buffer+0x13/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03afc7e>] journal_dirty_data+0x1de/0x280 [jbd]
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03d0d60>] ext3_journal_dirty_data+0x20/0x50 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03d0db5>] journal_dirty_data_fn+0x25/0x30 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03cfc92>] walk_page_buffers+0x82/0xc0 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03d0d90>] ? journal_dirty_data_fn+0x0/0x30 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffffa03d485b>] ext3_ordered_write_end+0xab/0x1c0 [ext3]
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81012c1e>] ? __copy_from_user_inatomic+0xe/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81124d4a>] generic_file_buffered_write+0x18a/0x2e0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8107c297>] ? current_fs_time+0x27/0x30
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81126740>] __generic_file_aio_write+0x260/0x490
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811269f8>] generic_file_aio_write+0x88/0x100
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81126970>] ? generic_file_aio_write+0x0/0x100
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8118da2b>] do_sync_readv_writev+0xfb/0x140
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811aad70>] ? iput+0x30/0x70
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8123a24b>] ? selinux_file_permission+0xfb/0x150
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8122d0a6>] ? security_file_permission+0x16/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8118ea86>] do_readv_writev+0xd6/0x1f0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81040f2c>] ? kvm_clock_read+0x1c/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81040f39>] ? kvm_clock_get_cycles+0x9/0x10
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8118ebe6>] vfs_writev+0x46/0x60
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8118ed11>] sys_writev+0x51/0xb0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff810e5aee>] ? __audit_syscall_exit+0x25e/0x290
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
>Sep  5 01:39:14 host-074 kernel: INFO: task sync:704 blocked for more than 120 seconds.
>Sep  5 01:39:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:39:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:39:14 host-074 kernel: sync          D 0000000000000000     0   704    703 0x00000080
>Sep  5 01:39:14 host-074 kernel: ffff880027555d38 0000000000000082 ffff880027555cd8 ffff880027555da8
>Sep  5 01:39:14 host-074 kernel: ffff880027555e20 0000000000000000 000000000000000e 0000000000000000
>Sep  5 01:39:14 host-074 kernel: ffff880027555d08 ffffffff81123df0 ffff8800275ce5f8 ffff880027555fd8
>Sep  5 01:39:14 host-074 kernel: Call Trace:
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81123df0>] ? find_get_pages_tag+0x40/0x130
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8152a0a5>] schedule_timeout+0x215/0x2e0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8105dea4>] ? check_preempt_wakeup+0x1a4/0x260
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8106d58b>] ? enqueue_task_fair+0xfb/0x100
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81529d23>] wait_for_common+0x123/0x180
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff81529e3d>] wait_for_completion+0x1d/0x20
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811ba035>] writeback_inodes_sb_nr+0x75/0xa0
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811ba0a3>] writeback_inodes_sb+0x43/0x50
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c080e>] __sync_filesystem+0x3e/0x90
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c0958>] sync_filesystems+0xf8/0x130
>Sep  5 01:39:14 host-074 kernel: [<ffffffff811c09e7>] sys_sync+0x17/0x40
>Sep  5 01:39:14 host-074 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
>Sep  5 01:41:14 host-074 kernel: INFO: task flush-253:6:2184 blocked for more than 120 seconds.
>Sep  5 01:41:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:41:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:41:14 host-074 kernel: flush-253:6   D 0000000000000000     0  2184      2 0x00000080
>Sep  5 01:41:14 host-074 kernel: ffff880020d8f840 0000000000000046 ffff880020d8f7d0 ffffffff81041e98
>Sep  5 01:41:14 host-074 kernel: ffff880000000000 0000fea2edc4482c ffff880020d8f810 ffff88003cfff670
>Sep  5 01:41:14 host-074 kernel: 000000000b517908 00000001077e5118 ffff88003b8bd098 ffff880020d8ffd8
>Sep  5 01:41:14 host-074 kernel: Call Trace:
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a22a>] __wait_on_bit_lock+0x5a/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a308>] out_of_line_wait_on_bit_lock+0x78/0x90
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4fd0>] ? end_buffer_async_write+0x0/0x190
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4e46>] __lock_buffer+0x36/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c6145>] __block_write_full_page+0x305/0x330
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4fd0>] ? end_buffer_async_write+0x0/0x190
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c6250>] block_write_full_page_endio+0xe0/0x120
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03cfd10>] ? buffer_unmapped+0x0/0x20 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c62a5>] block_write_full_page+0x15/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03d08bd>] ext3_ordered_writepage+0x1ed/0x240 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811380e7>] __writepage+0x17/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811393ad>] write_cache_pages+0x1fd/0x4c0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811380d0>] ? __writepage+0x0/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81139694>] generic_writepages+0x24/0x30
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811396d5>] do_writepages+0x35/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811bab3d>] writeback_single_inode+0xdd/0x290
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811baf3d>] writeback_sb_inodes+0xbd/0x170
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811bb09b>] writeback_inodes_wb+0xab/0x1b0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811bb493>] wb_writeback+0x2f3/0x410
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81088062>] ? del_timer_sync+0x22/0x30
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811bb66b>] wb_do_writeback+0xbb/0x240
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811bb853>] bdi_writeback_task+0x63/0x1b0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81148390>] ? bdi_start_fn+0x0/0x100
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81148416>] bdi_start_fn+0x86/0x100
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81148390>] ? bdi_start_fn+0x0/0x100
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
>Sep  5 01:41:14 host-074 kernel: INFO: task kjournald:2192 blocked for more than 120 seconds.
>Sep  5 01:41:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:41:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:41:14 host-074 kernel: kjournald     D 0000000000000000     0  2192      2 0x00000080
>Sep  5 01:41:14 host-074 kernel: ffff880008e15c50 0000000000000046 ffff880008e15be0 ffffffff81041e98
>Sep  5 01:41:14 host-074 kernel: ffff880000000000 0000fea3e7b879fe ffff880008e15c20 ffff88003cfff280
>Sep  5 01:41:14 host-074 kernel: 000000000b517980 ffffffffabf8928f ffff88003b257ab8 ffff880008e15fd8
>Sep  5 01:41:14 host-074 kernel: Call Trace:
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a45f>] __wait_on_bit+0x5f/0x90
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a508>] out_of_line_wait_on_bit+0x78/0x90
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c56>] __wait_on_buffer+0x26/0x30
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03b102f>] journal_commit_transaction+0x9ef/0x1300 [jbd]
>Sep  5 01:41:14 host-074 kernel: [<ffffffff810873dc>] ? lock_timer_base+0x3c/0x70
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03b6085>] kjournald+0xe5/0x240 [jbd]
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03b5fa0>] ? kjournald+0x0/0x240 [jbd]
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
>Sep  5 01:41:14 host-074 kernel: INFO: task xdoio:679 blocked for more than 120 seconds.
>Sep  5 01:41:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:41:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:41:14 host-074 kernel: xdoio         D 0000000000000000     0   679    678 0x00000080
>Sep  5 01:41:14 host-074 kernel: ffff880027435888 0000000000000082 ffff880027435818 ffffffff81041e98
>Sep  5 01:41:14 host-074 kernel: ffff880000000000 0000fea14de48a24 ffff880027435858 ffff88003a832b40
>Sep  5 01:41:14 host-074 kernel: 000000000b5176ec 00000001077e475d ffff880022c35ab8 ffff880027435fd8
>Sep  5 01:41:14 host-074 kernel: Call Trace:
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a22a>] __wait_on_bit_lock+0x5a/0xc0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c688c>] ? __block_prepare_write+0x11c/0x530
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a308>] out_of_line_wait_on_bit_lock+0x78/0x90
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c4e46>] __lock_buffer+0x36/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c5918>] __sync_dirty_buffer+0xc8/0xf0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c5953>] sync_dirty_buffer+0x13/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03afc7e>] journal_dirty_data+0x1de/0x280 [jbd]
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03d0d60>] ext3_journal_dirty_data+0x20/0x50 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03d0db5>] journal_dirty_data_fn+0x25/0x30 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03cfc92>] walk_page_buffers+0x82/0xc0 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03d0d90>] ? journal_dirty_data_fn+0x0/0x30 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffffa03d485b>] ext3_ordered_write_end+0xab/0x1c0 [ext3]
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81012c1e>] ? __copy_from_user_inatomic+0xe/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81124d4a>] generic_file_buffered_write+0x18a/0x2e0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8107c297>] ? current_fs_time+0x27/0x30
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81126740>] __generic_file_aio_write+0x260/0x490
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811269f8>] generic_file_aio_write+0x88/0x100
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81126970>] ? generic_file_aio_write+0x0/0x100
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8118da2b>] do_sync_readv_writev+0xfb/0x140
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811aad70>] ? iput+0x30/0x70
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8123a24b>] ? selinux_file_permission+0xfb/0x150
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8122d0a6>] ? security_file_permission+0x16/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8118ea86>] do_readv_writev+0xd6/0x1f0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81040f2c>] ? kvm_clock_read+0x1c/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81040f39>] ? kvm_clock_get_cycles+0x9/0x10
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8118ebe6>] vfs_writev+0x46/0x60
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8118ed11>] sys_writev+0x51/0xb0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff810e5aee>] ? __audit_syscall_exit+0x25e/0x290
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
>Sep  5 01:41:14 host-074 kernel: INFO: task sync:704 blocked for more than 120 seconds.
>Sep  5 01:41:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:41:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:41:14 host-074 kernel: sync          D 0000000000000000     0   704    703 0x00000080
>Sep  5 01:41:14 host-074 kernel: ffff880027555d38 0000000000000082 ffff880027555cd8 ffff880027555da8
>Sep  5 01:41:14 host-074 kernel: ffff880027555e20 0000000000000000 000000000000000e 0000000000000000
>Sep  5 01:41:14 host-074 kernel: ffff880027555d08 ffffffff81123df0 ffff8800275ce5f8 ffff880027555fd8
>Sep  5 01:41:14 host-074 kernel: Call Trace:
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81123df0>] ? find_get_pages_tag+0x40/0x130
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8152a0a5>] schedule_timeout+0x215/0x2e0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8105dea4>] ? check_preempt_wakeup+0x1a4/0x260
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8106d58b>] ? enqueue_task_fair+0xfb/0x100
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81529d23>] wait_for_common+0x123/0x180
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81064b90>] ? default_wake_function+0x0/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff81529e3d>] wait_for_completion+0x1d/0x20
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811ba035>] writeback_inodes_sb_nr+0x75/0xa0
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811ba0a3>] writeback_inodes_sb+0x43/0x50
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c080e>] __sync_filesystem+0x3e/0x90
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c0958>] sync_filesystems+0xf8/0x130
>Sep  5 01:41:14 host-074 kernel: [<ffffffff811c09e7>] sys_sync+0x17/0x40
>Sep  5 01:41:14 host-074 kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
>Sep  5 01:43:14 host-074 kernel: INFO: task flush-253:6:2184 blocked for more than 120 seconds.
>Sep  5 01:43:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:43:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:43:14 host-074 kernel: flush-253:6   D 0000000000000000     0  2184      2 0x00000080
>Sep  5 01:43:14 host-074 kernel: ffff880020d8f840 0000000000000046 ffff880020d8f7d0 ffffffff81041e98
>Sep  5 01:43:14 host-074 kernel: ffff880000000000 0000fea2edc4482c ffff880020d8f810 ffff88003cfff670
>Sep  5 01:43:14 host-074 kernel: 000000000b517908 00000001077e5118 ffff88003b8bd098 ffff880020d8ffd8
>Sep  5 01:43:14 host-074 kernel: Call Trace:
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8152a22a>] __wait_on_bit_lock+0x5a/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8152a308>] out_of_line_wait_on_bit_lock+0x78/0x90
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4fd0>] ? end_buffer_async_write+0x0/0x190
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4e46>] __lock_buffer+0x36/0x40
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c6145>] __block_write_full_page+0x305/0x330
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4fd0>] ? end_buffer_async_write+0x0/0x190
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c6250>] block_write_full_page_endio+0xe0/0x120
>Sep  5 01:43:14 host-074 kernel: [<ffffffffa03cfd10>] ? buffer_unmapped+0x0/0x20 [ext3]
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c62a5>] block_write_full_page+0x15/0x20
>Sep  5 01:43:14 host-074 kernel: [<ffffffffa03d08bd>] ext3_ordered_writepage+0x1ed/0x240 [ext3]
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811380e7>] __writepage+0x17/0x40
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811393ad>] write_cache_pages+0x1fd/0x4c0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811380d0>] ? __writepage+0x0/0x40
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81139694>] generic_writepages+0x24/0x30
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811396d5>] do_writepages+0x35/0x40
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811bab3d>] writeback_single_inode+0xdd/0x290
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811baf3d>] writeback_sb_inodes+0xbd/0x170
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811bb09b>] writeback_inodes_wb+0xab/0x1b0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811bb493>] wb_writeback+0x2f3/0x410
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81088062>] ? del_timer_sync+0x22/0x30
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811bb66b>] wb_do_writeback+0xbb/0x240
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811bb853>] bdi_writeback_task+0x63/0x1b0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109e987>] ? bit_waitqueue+0x17/0xd0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81148390>] ? bdi_start_fn+0x0/0x100
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81148416>] bdi_start_fn+0x86/0x100
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81148390>] ? bdi_start_fn+0x0/0x100
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
>Sep  5 01:43:14 host-074 kernel: INFO: task kjournald:2192 blocked for more than 120 seconds.
>Sep  5 01:43:14 host-074 kernel:      Not tainted 2.6.32-494.el6.x86_64 #1
>Sep  5 01:43:14 host-074 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>Sep  5 01:43:14 host-074 kernel: kjournald     D 0000000000000000     0  2192      2 0x00000080
>Sep  5 01:43:14 host-074 kernel: ffff880008e15c50 0000000000000046 ffff880008e15be0 ffffffff81041e98
>Sep  5 01:43:14 host-074 kernel: ffff880000000000 0000fea3e7b879fe ffff880008e15c20 ffff88003cfff280
>Sep  5 01:43:14 host-074 kernel: 000000000b517980 ffffffffabf8928f ffff88003b257ab8 ffff880008e15fd8
>Sep  5 01:43:14 host-074 kernel: Call Trace:
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81041e98>] ? pvclock_clocksource_read+0x58/0xd0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff810aaa21>] ? ktime_get_ts+0xb1/0xf0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81529993>] io_schedule+0x73/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4ca0>] sync_buffer+0x40/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8152a45f>] __wait_on_bit+0x5f/0x90
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4c60>] ? sync_buffer+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8152a508>] out_of_line_wait_on_bit+0x78/0x90
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109eb80>] ? wake_bit_function+0x0/0x50
>Sep  5 01:43:14 host-074 kernel: [<ffffffff811c4c56>] __wait_on_buffer+0x26/0x30
>Sep  5 01:43:14 host-074 kernel: [<ffffffffa03b102f>] journal_commit_transaction+0x9ef/0x1300 [jbd]
>Sep  5 01:43:14 host-074 kernel: [<ffffffff810873dc>] ? lock_timer_base+0x3c/0x70
>Sep  5 01:43:14 host-074 kernel: [<ffffffff81087fdb>] ? try_to_del_timer_sync+0x7b/0xe0
>Sep  5 01:43:14 host-074 kernel: [<ffffffffa03b6085>] kjournald+0xe5/0x240 [jbd]
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
>Sep  5 01:43:14 host-074 kernel: [<ffffffffa03b5fa0>] ? kjournald+0x0/0x240 [jbd]
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109e66e>] kthread+0x9e/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8109e5d0>] ? kthread+0x0/0xc0
>Sep  5 01:43:14 host-074 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
>Sep  5 01:49:46 host-074 dhclient[1115]: DHCPREQUEST on eth1 to 192.168.1.254 port 67 (xid=0x1736a52a)
>Sep  5 01:49:46 host-074 dhclient[1115]: DHCPACK from 192.168.1.254 (xid=0x1736a52a)
>Sep  5 01:49:48 host-074 dhclient[1115]: bound to 192.168.1.74 -- renewal in 1498 seconds.
>Sep  5 02:14:46 host-074 dhclient[1115]: DHCPREQUEST on eth1 to 192.168.1.254 port 67 (xid=0x1736a52a)
>Sep  5 02:14:46 host-074 dhclient[1115]: DHCPACK from 192.168.1.254 (xid=0x1736a52a)
>Sep  5 02:14:48 host-074 dhclient[1115]: bound to 192.168.1.74 -- renewal in 1196 seconds.
>Sep  5 02:34:44 host-074 dhclient[1115]: DHCPREQUEST on eth1 to 192.168.1.254 port 67 (xid=0x1736a52a)
>Sep  5 02:34:44 host-074 dhclient[1115]: DHCPACK from 192.168.1.254 (xid=0x1736a52a)
>Sep  5 02:34:46 host-074 dhclient[1115]: bound to 192.168.1.74 -- renewal in 1682 seconds.
>Sep  5 03:02:48 host-074 dhclient[1115]: DHCPREQUEST on eth1 to 192.168.1.254 port 67 (xid=0x1736a52a)
>Sep  5 03:02:48 host-074 dhclient[1115]: DHCPACK from 192.168.1.254 (xid=0x1736a52a)
>Sep  5 03:02:50 host-074 dhclient[1115]: bound to 192.168.1.74 -- renewal in 1211 seconds.
>Sep  5 03:23:01 host-074 dhclient[1115]: DHCPREQUEST on eth1 to 192.168.1.254 port 67 (xid=0x1736a52a)
>Sep  5 03:23:01 host-074 dhclient[1115]: DHCPACK from 192.168.1.254 (xid=0x1736a52a)
>Sep  5 03:23:03 host-074 dhclient[1115]: bound to 192.168.1.74 -- renewal in 1407 seconds.
>Sep  5 03:26:05 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: device-mapper: raid1: Mirror read failed from 253:4. Trying alternative device.
>Sep  5 03:26:05 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 lvm[3854]: Primary mirror device 253:4 read failed.
>Sep  5 03:26:06 host-074 lvm[3854]: Log device 253:2 has failed (D).
>Sep  5 03:26:06 host-074 lvm[3854]: Device failure in revolution_9-mirror_1.
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sda1: read failed after 0 of 512 at 16104947712: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sda1: read failed after 0 of 512 at 16105054208: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sda1: read failed after 0 of 512 at 0: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sda1: read failed after 0 of 512 at 4096: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sda1: read failed after 0 of 2048 at 0: Input/output error
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 16104947712: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 16105054208: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 0: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 512 at 4096: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdg1: read failed after 0 of 2048 at 0: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 16104947712: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 16105054208: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 0: Input/output error
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 512 at 4096: Input/output error
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 lvm[3854]: /dev/sdh1: read failed after 0 of 2048 at 0: Input/output error
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 lvm[3854]: Couldn't find device with uuid P2DTMe-vQrr-ArL8-D3s8-jhzd-BLOy-vtM6Vp.
>Sep  5 03:26:06 host-074 lvm[3854]: Couldn't find device with uuid iHLb4R-mzpv-1zFj-HMI1-62tV-6ZR0-sulMkq.
>Sep  5 03:26:06 host-074 lvm[3854]: Couldn't find device with uuid eFdxXo-mV8d-e0T9-mmAT-lFzA-dgEI-ewsmbH.
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 2:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 9:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: sd 8:0:0:1: rejecting I/O to offline device
>Sep  5 03:26:06 host-074 kernel: device-mapper: raid1: log postsuspend failed
>Sep  5 03:26:08 host-074 lvm[3854]: Mirror status: 2 of 3 images failed.
>Sep  5 03:26:08 host-074 lvm[3854]: Mirror log status: 1 of 1 images failed.
>Sep  5 03:26:08 host-074 lvm[3854]: Trying to up-convert to 3 images, 1 logs.
>Sep  5 03:26:09 host-074 lvm[3854]: Monitoring mirror device revolution_9-mirror_1 for events.
>Sep  5 03:26:09 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 0.0%
>Sep  5 03:26:24 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 2.7%
>Sep  5 03:26:39 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 4.8%
>Sep  5 03:26:54 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 6.0%
>Sep  5 03:27:09 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 9.4%
>Sep  5 03:27:25 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 11.8%
>Sep  5 03:27:40 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 13.1%
>Sep  5 03:27:55 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 15.6%
>Sep  5 03:28:10 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 18.2%
>Sep  5 03:28:25 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 19.4%
>Sep  5 03:28:40 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 21.4%
>Sep  5 03:28:55 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 23.4%
>Sep  5 03:29:10 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 25.4%
>Sep  5 03:29:25 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 28.5%
>Sep  5 03:29:40 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 31.8%
>Sep  5 03:29:56 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 33.5%
>Sep  5 03:30:11 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 36.7%
>Sep  5 03:30:26 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 38.5%
>Sep  5 03:30:41 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 39.8%
>Sep  5 03:30:56 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 42.8%
>Sep  5 03:31:11 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 44.6%
>Sep  5 03:31:26 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 46.4%
>Sep  5 03:31:41 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 49.0%
>Sep  5 03:31:56 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 49.8%
>Sep  5 03:32:11 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 51.8%
>Sep  5 03:32:27 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 54.3%
>Sep  5 03:32:42 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 55.7%
>Sep  5 03:32:57 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 56.8%
>Sep  5 03:33:12 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 59.4%
>Sep  5 03:33:27 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 61.3%
>Sep  5 03:33:42 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 62.6%
>Sep  5 03:33:57 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 65.2%
>Sep  5 03:34:12 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 66.7%
>Sep  5 03:34:27 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 69.1%
>Sep  5 03:34:42 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 71.2%
>Sep  5 03:34:57 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 72.3%
>Sep  5 03:35:13 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 74.7%
>Sep  5 03:35:28 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 76.7%
>Sep  5 03:35:43 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 78.8%
>Sep  5 03:35:58 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 80.5%
>Sep  5 03:36:13 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 82.2%
>Sep  5 03:36:28 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 84.1%
>Sep  5 03:36:43 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 86.2%
>Sep  5 03:36:59 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 89.3%
>Sep  5 03:37:14 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 91.3%
>Sep  5 03:37:29 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 96.5%
>Sep  5 03:37:44 host-074 lvm[3854]: revolution_9/mirror_1: Converted: 100.0%
>Sep  5 03:37:59 host-074 lvm[3854]: Repair of mirrored device revolution_9-mirror_1 finished successfully.
>Sep  5 03:37:59 host-074 lvm[3854]: revolution_9-mirror_1 is now in-sync.
>Sep  5 03:38:00 host-074 lvm[3854]: No longer monitoring mirror device revolution_9-mirror_1 for events.

Actions: View

Attachments on bug 1138452: 934870