Attachment 908622 Details for Bug 1107315 – Log of unmounting a multipath device with no paths, set for deferred remove

Log of unmounting a multipath device with no paths, set for deferred remove

hang_log (text/plain), 6.92 KB, created by Ben Marzinski on 2014-06-13 15:34:10 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ben Marzinski

Created: 2014-06-13 15:34:10 UTC

Size: 6.92 KB

patch

obsolete

>login: root
>Password: 
>Last login: Fri Jun 13 04:20:12 from octiron.msp.redhat.com
>
>This node was installed via Steel Toe
>
>The last tree installed was RHEL6.5-Server
>
>[root@ask-07 ~]# multipath -ll
>mpathd (3600d0230000000000e13955cc3757803) dm-7
>size=16G features='0' hwhandler='0' wp=rw
>[root@ask-07 ~]# dmsetup info mpathd
>Name:              mpathd
>State:             ACTIVE (DEFERRED REMOVE)
>Read Ahead:        256
>Tables present:    LIVE
>Open count:        1
>Event number:      1
>Major, minor:      253, 7
>Number of targets: 1
>UUID: mpath-3600d0230000000000e13955cc3757803
>
>[root@ask-07 ~]# df
>Filesystem                   1K-blocks     Used Available Use% Mounted on
>/dev/mapper/vg_ask07-lv_root  51475068 13620020  35233608  28% /
>tmpfs                          1993536        0   1993536   0% /dev/shm
>/dev/sda1                       487652    83560    378492  19% /boot
>/dev/mapper/vg_ask07-lv_home  20587768    44992  19490312   1% /home
>/dev/mapper/mpathd            16513960   176240  15498860   2% /mnt/test
>[root@ask-07 ~]# umount /mnt/test
>end_request: I/O error, dev dm-7, sector 0
>Buffer I/O error on device dm-7, logical block 0
>lost page write due to I/O error on dm-7
>EXT3-fs (dm-7): I/O error while writing superblock
>------------[ cut here ]------------
>WARNING: at kernel/workqueue.c:366 flush_cpu_workqueue+0x8d/0x90() (Not tainted)
>Hardware name: PowerEdge SC1435
>Modules linked in: ext3 jbd[root@ask-07 ~]# be2iscsi  iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_round_robin dm_multipath microcode dcdbas serio_raw tg3 ptp pps_core k8temp amd64_edac_mod edac_core edac_mce_amd sg i2c_piix4 shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom qla2xxx scsi_transport_fc scsi_tgt sata_svw pata_acpi ata_generic pata_serverworks radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: mperf]
>Pid: 20, comm: events/1 Not tainted 2.6.32-478.el6.x86_64 #1
>Call Trace:
> [<ffffffff81074027>] ? warn_slowpath_common+0x87/0xc0
> [<ffffffff8107407a>] ? warn_slowpath_null+0x1a/0x20
> [<ffffffff810975bd>] ? flush_cpu_workqueue+0x8d/0x90
> [<ffffffff8109755a>] ? flush_cpu_workqueue+0x2a/0x90
> [<ffffffff81098064>] ? flush_workqueue+0x54/0x80
> [<ffffffff810980a5>] ? flush_scheduled_work+0x15/0x20
> [<ffffffffa0239786>] ? flush_multipath_work+0xe6/0x120 [dm_multipath]
> [<ffffffff81063e30>] ? default_wake_function+0x0/0x20
> [<ffffffffa02397f2>] ? multipath_postsuspend+0x32/0x50 [dm_multipath]
> [<ffffffffa00043a2>] ? suspend_targets+0x42/0x80 [dm_mod]
> [<ffffffffa0004418>] ? dm_table_postsuspend_targets+0x18/0x20 [dm_mod]
> [<ffffffffa000389a>] ? __dm_destroy+0x22a/0x230 [dm_mod]
> [<ffffffffa00038d3>] ? dm_destroy+0x13/0x20 [dm_mod]
> [<ffffffffa0009a45>] ? dm_hash_remove_all+0x75/0x130 [dm_mod]
> [<ffffffffa0002bd0>] ? do_deferred_remove+0x0/0x10 [dm_mod]
> [<ffffffffa0009b9a>] ? dm_deferred_remove+0x1a/0x20 [dm_mod]
> [<ffffffffa0002bde>] ? do_deferred_remove+0xe/0x10 [dm_mod]
> [<ffffffff810971c0>] ? worker_thread+0x170/0x2a0
> [<ffffffff8109db90>] ? autoremove_wake_function+0x0/0x40
> [<ffffffff81097050>] ? worker_thread+0x0/0x2a0
> [<ffffffff8109d7e6>] ? kthread+0x96/0xa0
> [<ffffffff8100c20a>] ? child_rip+0xa/0x20
> [<ffffffff8109d750>] ? kthread+0x0/0xa0
> [<ffffffff8100c200>] ? child_rip+0x0/0x20
>---[ end trace 5a8c50c5e8ce4b8d ]---
>
>[root@ask-07 ~]# ps ax | grep D
>  PID TTY      STAT   TIME COMMAND
>   20 ?        D      0:03 [events/1]
> 4379 ttyS0    S+     0:00 grep D
>[root@ask-07 ~]# df
>Filesystem                   1K-blocks     Used Available Use% Mounted on
>/dev/mapper/vg_ask07-lv_root  51475068 13620012  35233616  28% /
>tmpfs                          1993536        0   1993536   0% /dev/shm
>/dev/sda1                       487652    83560    378492  19% /boot
>/dev/mapper/vg_ask07-lv_home  20587768    44992  19490312   1% /home
>[root@ask-07 ~]# cat /proc/20/stack
>[<ffffffff81097591>] flush_cpu_workqueue+0x61/0x90
>[<ffffffff81098064>] flush_workqueue+0x54/0x80
>[<ffffffff810980a5>] flush_scheduled_work+0x15/0x20
>[<ffffffffa0239786>] flush_multipath_work+0xe6/0x120 [dm_multipath]
>[<ffffffffa02397f2>] multipath_postsuspend+0x32/0x50 [dm_multipath]
>[<ffffffffa00043a2>] suspend_targets+0x42/0x80 [dm_mod]
>[<ffffffffa0004418>] dm_table_postsuspend_targets+0x18/0x20 [dm_mod]
>[<ffffffffa000389a>] __dm_destroy+0x22a/0x230 [dm_mod]
>[<ffffffffa00038d3>] dm_destroy+0x13/0x20 [dm_mod]
>[<ffffffffa0009a45>] dm_hash_remove_all+0x75/0x130 [dm_mod]
>[<ffffffffa0009b9a>] dm_deferred_remove+0x1a/0x20 [dm_mod]
>[<ffffffffa0002bde>] do_deferred_remove+0xe/0x10 [dm_mod]
>[<ffffffff810971c0>] worker_thread+0x170/0x2a0
>[<ffffffff8109d7e6>] kthread+0x96/0xa0
>[<ffffffff8100c20a>] child_rip+0xa/0x20
>[<ffffffffffffffff>] 0xffffffffffffffff
>[root@ask-07 ~]# ps ax | grep D
>  PID TTY      STAT   TIME COMMAND
>   20 ?        D      0:03 [events/1]
> 4383 ttyS0    S+     0:00 grep D
>[root@ask-07 ~]# INFO: task events/1:20 blocked for more than 120 seconds.
>      Tainted: G        W  ---------------    2.6.32-478.el6.x86_64 #1
>"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>events/1      D 0000000000000001     0    20      2 0x00000000
> ffff88007f3dbaa0 0000000000000046 0000000000000000 ffff88007f3b2aa0
> ffff88007f3b2ad8 0000000000000000 00009d219b961089 ffffffff8106b780
> ffff8800822168e8 000000010a47a51c ffff88007f3b3058 ffff88007f3dbfd8
>Call Trace:
> [<ffffffff8106b780>] ? pick_next_task_fair+0xd0/0x130
> [<ffffffff815262f5>] schedule_timeout+0x215/0x2e0
> [<ffffffff81525f73>] wait_for_common+0x123/0x180
> [<ffffffff81063e30>] ? default_wake_function+0x0/0x20
> [<ffffffff8152608d>] wait_for_completion+0x1d/0x20
> [<ffffffff81097591>] flush_cpu_workqueue+0x61/0x90
> [<ffffffff81097640>] ? wq_barrier_func+0x0/0x20
> [<ffffffff81098064>] flush_workqueue+0x54/0x80
> [<ffffffff810980a5>] flush_scheduled_work+0x15/0x20
> [<ffffffffa0239786>] flush_multipath_work+0xe6/0x120 [dm_multipath]
> [<ffffffff81063e30>] ? default_wake_function+0x0/0x20
> [<ffffffffa02397f2>] multipath_postsuspend+0x32/0x50 [dm_multipath]
> [<ffffffffa00043a2>] suspend_targets+0x42/0x80 [dm_mod]
> [<ffffffffa0004418>] dm_table_postsuspend_targets+0x18/0x20 [dm_mod]
> [<ffffffffa000389a>] __dm_destroy+0x22a/0x230 [dm_mod]
> [<ffffffffa00038d3>] dm_destroy+0x13/0x20 [dm_mod]
> [<ffffffffa0009a45>] dm_hash_remove_all+0x75/0x130 [dm_mod]
> [<ffffffffa0002bd0>] ? do_deferred_remove+0x0/0x10 [dm_mod]
> [<ffffffffa0009b9a>] dm_deferred_remove+0x1a/0x20 [dm_mod]
> [<ffffffffa0002bde>] do_deferred_remove+0xe/0x10 [dm_mod]
> [<ffffffff810971c0>] worker_thread+0x170/0x2a0
> [<ffffffff8109db90>] ? autoremove_wake_function+0x0/0x40
> [<ffffffff81097050>] ? worker_thread+0x0/0x2a0
> [<ffffffff8109d7e6>] kthread+0x96/0xa0
> [<ffffffff8100c20a>] child_rip+0xa/0x20
> [<ffffffff8109d750>] ? kthread+0x0/0xa0
> [<ffffffff8100c200>] ? child_rip+0x0/0x20

Actions: View

Attachments on bug 1107315: 908622