Bug 1414119 - running out of disk space in /etc/lvm/ on one cluster node may cause lvm ops running exclusively on another node to deadlock
Summary: running out of disk space in /etc/lvm/ on one cluster node may cause lvm ops ...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.9
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: rc
: ---
Assignee: LVM and device-mapper development team
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-17 19:57 UTC by Corey Marthaler
Modified: 2017-12-06 10:27 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-06 10:27:52 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Corey Marthaler 2017-01-17 19:57:35 UTC
Description of problem:
# In this cluster node host-122 ran out of disk space
[root@host-122 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M    132   2017-01-16 15:12:39  host-122
   2   M    144   2017-01-16 15:12:42  host-123
   3   M    144   2017-01-16 15:12:42  host-124

[root@host-122 ~]# cman_tool services
fence domain
member count  3
victim count  0
victim now    0
master nodeid 1
wait state    none
members       1 2 3 

dlm lockspaces
name          clvmd
id            0x4104eefa
flags         0x00000000 
change        member 3 joined 1 remove 0 failed 0 seq 1,1
members       1 2 3 

[root@host-122 ~]# lvs -a -o +devices
  LV                    VG            Attr       LSize   Pool Devices
  POOL                  snapper_thinp twi---tz-- 252.00m      POOL_tdata(0)
  [POOL_tdata]          snapper_thinp rwi---r--- 252.00m      POOL_tdata_rimage_0(0),POOL_tdata_rimage_1(0)
  [POOL_tdata_rimage_0] snapper_thinp Iwi---r--- 252.00m      /dev/sdb1(3)
  [POOL_tdata_rimage_1] snapper_thinp Iwi---r--- 252.00m      /dev/sda1(3)
  [POOL_tdata_rmeta_0]  snapper_thinp ewi---r---   4.00m      /dev/sdb1(2)
  [POOL_tdata_rmeta_1]  snapper_thinp ewi---r---   4.00m      /dev/sda1(2)
  [POOL_tmeta]          snapper_thinp ewi---r---   4.00m      POOL_tmeta_rimage_0(0),POOL_tmeta_rimage_1(0)
  [POOL_tmeta_rimage_0] snapper_thinp Iwi---r---   4.00m      /dev/sdb1(1)
  [POOL_tmeta_rimage_1] snapper_thinp Iwi---r---   4.00m      /dev/sda1(1)
  [POOL_tmeta_rmeta_0]  snapper_thinp ewi---r---   4.00m      /dev/sdb1(0)
  [POOL_tmeta_rmeta_1]  snapper_thinp ewi---r---   4.00m      /dev/sda1(0)
  contig_with           snapper_thinp -wc------- 400.00m      /dev/sdg1(0) 
  [lvol0_pmspare]       snapper_thinp ewi-------   4.00m      /dev/sdb1(66)
  origin                snapper_thinp Vwi---tz--   1.00g POOL
  other1                snapper_thinp Vwi---tz--   1.00g POOL
  other2                snapper_thinp Vwi---tz--   1.00g POOL
  other3                snapper_thinp Vwi---tz--   1.00g POOL
  other4                snapper_thinp Vwi---tz--   1.00g POOL
  other5                snapper_thinp Vwi---tz--   1.00g POOL
  lv_root               vg_host122    -wi-ao----   6.71g      /dev/vda2(0)
  lv_swap               vg_host122    -wi-ao---- 816.00m      /dev/vda2(1718)
  /etc/lvm/cache/.cache.tmp: write error failed: No space left on device

[root@host-122 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_host122-lv_root
                      6.5G  6.5G     0 100% /
tmpfs                 499M   26M  474M   6% /dev/shm
/dev/vda1             477M   35M  417M   8% /boot


# host-124 was doing exclusive activation lvm operations that deadlocked, presumably due to the inability to write in /etc/lvm/ on host-122?

 Removing snap volume snapper_thinp/contig_with
 lvremove -f /dev/snapper_thinp/contig_with
 couldn't remove volume contig_with
   Error locking on node host-123: Refusing activation of partial LV snapper_thinp/contig_with.  Use '--activationmode partial' to override.
    Error locking on node host-122: Refusing activation of partial LV snapper_thinp/contig_with.  Use '--activationmode partial' to override.
    Failed to activate contig_with.
    Releasing activation in critical section.
 

# HOST-122 
Jan 16 22:15:36 host-122 udevd-work[18877]: inotify_add_watch(6, /dev/dm-2, 10) failed: No such file or directory

# HOST-123 
Jan 16 22:15:36 host-123 udevd-work[16450]: inotify_add_watch(6, /dev/dm-2, 10) failed: No such file or directory

# HOST-124 
host-124 qarshd[22345]: Running cmdline: lvcreate --activate ey -n contig_without -C y -L 600M -s snapper_thinp/origin
host-124 qarshd[22348]: Running cmdline: lvcreate --activate ey -n contig_with -C y -L 400M -s snapper_thinp/origin
host-124 multipathd: uevent trigger error
host-124 multipathd: dm-20: remove map (uevent)
host-124 multipathd: dm-20: devmap not registered, can't remove
host-124 qarshd[22438]: Running cmdline: lvremove -f /dev/snapper_thinp/contig_with
host-124 multipathd: uevent trigger error
host-124 multipathd: dm-21: remove map (uevent)
host-124 multipathd: dm-21: devmap not registered, can't remove
host-124 qarshd[22482]: Running cmdline: lvremove -f /dev/snapper_thinp/contig_with
host-124 kernel: INFO: task lvremove:22494 blocked for more than 120 seconds.
host-124 kernel:      Not tainted 2.6.32-680.el6.x86_64 #1
host-124 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
host-124 kernel: lvremove      D 0000000000000000     0 22494  22482 0x00000080
host-124 kernel: ffff88002ff9fa18 0000000000000086 ffff880000000000 00068014e647e8d0
host-124 kernel: ffff880035a35680 ffff8800396a5d70 000000000275e218 ffffffffa782e3f6
host-124 kernel: 000000003aa490e1 ffffffff81aa8380 ffff880016663ad8 ffff88002ff9ffd8
host-124 kernel: Call Trace:
host-124 kernel: [<ffffffff8154b313>] io_schedule+0x73/0xc0
host-124 kernel: [<ffffffff811dae0d>] __blockdev_direct_IO_newtrunc+0xb7d/0x1270
host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90
host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90
host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20
host-124 kernel: [<ffffffff81287b0d>] ? get_disk+0x7d/0xf0
host-124 kernel: [<ffffffff811db577>] __blockdev_direct_IO+0x77/0xe0
host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20
host-124 kernel: [<ffffffff811d77f7>] blkdev_direct_IO+0x57/0x60
host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20
host-124 kernel: [<ffffffff8113055b>] generic_file_aio_read+0x6bb/0x700
host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90
host-124 kernel: [<ffffffff811d6ce1>] blkdev_aio_read+0x51/0x80
host-124 kernel: [<ffffffff81199d9a>] do_sync_read+0xfa/0x140
host-124 kernel: [<ffffffff810a6960>] ? autoremove_wake_function+0x0/0x40
host-124 kernel: [<ffffffff811d6b0c>] ? block_ioctl+0x3c/0x40
host-124 kernel: [<ffffffff811af852>] ? vfs_ioctl+0x22/0xa0
host-124 kernel: [<ffffffff8124822b>] ? selinux_file_permission+0xfb/0x150
host-124 kernel: [<ffffffff8123aea6>] ? security_file_permission+0x16/0x20
host-124 kernel: [<ffffffff8119a695>] vfs_read+0xb5/0x1a0
host-124 kernel: [<ffffffff8119b446>] ? fget_light_pos+0x16/0x50
host-124 kernel: [<ffffffff8119a9e1>] sys_read+0x51/0xb0
host-124 kernel: [<ffffffff810ee52e>] ? __audit_syscall_exit+0x25e/0x290
host-124 kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
host-124 kernel: INFO: task lvremove:22494 blocked for more than 120 seconds.
host-124 kernel:      Not tainted 2.6.32-680.el6.x86_64 #1
host-124 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
host-124 kernel: lvremove      D 0000000000000000     0 22494  22482 0x00000080
host-124 kernel: ffff88002ff9fa18 0000000000000086 ffff880000000000 00068014e647e8d0
host-124 kernel: ffff880035a35680 ffff8800396a5d70 000000000275e218 ffffffffa782e3f6
host-124 kernel: 000000003aa490e1 ffffffff81aa8380 ffff880016663ad8 ffff88002ff9ffd8
host-124 kernel: Call Trace:
host-124 kernel: [<ffffffff8154b313>] io_schedule+0x73/0xc0
host-124 kernel: [<ffffffff811dae0d>] __blockdev_direct_IO_newtrunc+0xb7d/0x1270
host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90
host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90
host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20
host-124 kernel: [<ffffffff81287b0d>] ? get_disk+0x7d/0xf0
host-124 kernel: [<ffffffff811db577>] __blockdev_direct_IO+0x77/0xe0
host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20
host-124 kernel: [<ffffffff811d77f7>] blkdev_direct_IO+0x57/0x60
host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20
host-124 kernel: [<ffffffff8113055b>] generic_file_aio_read+0x6bb/0x700
host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90
host-124 kernel: [<ffffffff811d6ce1>] blkdev_aio_read+0x51/0x80
host-124 kernel: [<ffffffff81199d9a>] do_sync_read+0xfa/0x140
host-124 kernel: [<ffffffff810a6960>] ? autoremove_wake_function+0x0/0x40
host-124 kernel: [<ffffffff811d6b0c>] ? block_ioctl+0x3c/0x40
host-124 kernel: [<ffffffff811af852>] ? vfs_ioctl+0x22/0xa0
host-124 kernel: [<ffffffff8124822b>] ? selinux_file_permission+0xfb/0x150
host-124 kernel: [<ffffffff8123aea6>] ? security_file_permission+0x16/0x20
host-124 kernel: [<ffffffff8119a695>] vfs_read+0xb5/0x1a0
host-124 kernel: [<ffffffff8119b446>] ? fget_light_pos+0x16/0x50
host-124 kernel: [<ffffffff8119a9e1>] sys_read+0x51/0xb0
host-124 kernel: [<ffffffff810ee52e>] ? __audit_syscall_exit+0x25e/0x290
host-124 kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b




Version-Release number of selected component (if applicable):
2.6.32-680.el6.x86_64

lvm2-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
lvm2-libs-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
lvm2-cluster-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
udev-147-2.73.el6_8.2    BUILT: Tue Aug 30 08:17:19 CDT 2016
device-mapper-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-libs-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-event-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-event-libs-1.02.117-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017
device-mapper-persistent-data-0.6.2-0.1.rc7.el6    BUILT: Tue Mar 22 08:58:09 CDT 2016
cmirror-2.02.143-12.el6    BUILT: Wed Jan 11 09:35:04 CST 2017

Comment 3 Zdenek Kabelac 2017-02-09 11:17:03 UTC
To make myself clear what is tested here -

So to make it clear - thin-pool (and all its thins) have to be active only on 1 node in cluster - unfortunately this is not 'enforced' properly by lvm2 ATM and depends on a user to not make a mistake  (there is not yet upstream fix for this).

So is this bug  about 'overfilling' thin-pool on 1 node.
Then kill thin-pool on this particular node (lvchange -an for all LVs)
And then trying thin-pool stack activation on different node ?

Comment 4 Corey Marthaler 2017-02-09 15:11:40 UTC
No. This has nothing to do with full thin pool testing. All "cluster thin lvm" testing was done exclusively on just one node (in this scenario host-124). Activation and all other operations happened on only this *one* node. 

This bug is about how while this testing was happening, the root file systems on the other two nodes in the cluster (host-122 and host-123) were filled to 100%. So lvm no longer had any space in /etc/lvm/* to do any backup, cache, or archive writing on those two nodes, and as such, caused the testing that was happening exclusively on host-124 to start failing. Host-124 did still have free space in /.

Comment 6 Jan Kurik 2017-12-06 10:27:52 UTC
Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available.

The official life cycle policy can be reviewed here:

http://redhat.com/rhel/lifecycle

This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL:

https://access.redhat.com/


Note You need to log in before you can comment on or make changes to this bug.