Bug 1414119
| Summary: | running out of disk space in /etc/lvm/ on one cluster node may cause lvm ops running exclusively on another node to deadlock | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 6 | Reporter: | Corey Marthaler <cmarthal> |
| Component: | lvm2 | Assignee: | LVM and device-mapper development team <lvm-team> |
| lvm2 sub component: | Clustering / clvmd (RHEL6) | QA Contact: | cluster-qe <cluster-qe> |
| Status: | CLOSED WONTFIX | Docs Contact: | |
| Severity: | low | ||
| Priority: | unspecified | CC: | agk, cmarthal, heinzm, jbrassow, msnitzer, prajnoha, prockai, zkabelac |
| Version: | 6.9 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-12-06 10:27:52 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
To make myself clear what is tested here - So to make it clear - thin-pool (and all its thins) have to be active only on 1 node in cluster - unfortunately this is not 'enforced' properly by lvm2 ATM and depends on a user to not make a mistake (there is not yet upstream fix for this). So is this bug about 'overfilling' thin-pool on 1 node. Then kill thin-pool on this particular node (lvchange -an for all LVs) And then trying thin-pool stack activation on different node ? No. This has nothing to do with full thin pool testing. All "cluster thin lvm" testing was done exclusively on just one node (in this scenario host-124). Activation and all other operations happened on only this *one* node. This bug is about how while this testing was happening, the root file systems on the other two nodes in the cluster (host-122 and host-123) were filled to 100%. So lvm no longer had any space in /etc/lvm/* to do any backup, cache, or archive writing on those two nodes, and as such, caused the testing that was happening exclusively on host-124 to start failing. Host-124 did still have free space in /. Red Hat Enterprise Linux 6 is in the Production 3 Phase. During the Production 3 Phase, Critical impact Security Advisories (RHSAs) and selected Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available. The official life cycle policy can be reviewed here: http://redhat.com/rhel/lifecycle This issue does not meet the inclusion criteria for the Production 3 Phase and will be marked as CLOSED/WONTFIX. If this remains a critical requirement, please contact Red Hat Customer Support to request a re-evaluation of the issue, citing a clear business justification. Note that a strong business justification will be required for re-evaluation. Red Hat Customer Support can be contacted via the Red Hat Customer Portal at the following URL: https://access.redhat.com/ |
Description of problem: # In this cluster node host-122 ran out of disk space [root@host-122 ~]# cman_tool nodes Node Sts Inc Joined Name 1 M 132 2017-01-16 15:12:39 host-122 2 M 144 2017-01-16 15:12:42 host-123 3 M 144 2017-01-16 15:12:42 host-124 [root@host-122 ~]# cman_tool services fence domain member count 3 victim count 0 victim now 0 master nodeid 1 wait state none members 1 2 3 dlm lockspaces name clvmd id 0x4104eefa flags 0x00000000 change member 3 joined 1 remove 0 failed 0 seq 1,1 members 1 2 3 [root@host-122 ~]# lvs -a -o +devices LV VG Attr LSize Pool Devices POOL snapper_thinp twi---tz-- 252.00m POOL_tdata(0) [POOL_tdata] snapper_thinp rwi---r--- 252.00m POOL_tdata_rimage_0(0),POOL_tdata_rimage_1(0) [POOL_tdata_rimage_0] snapper_thinp Iwi---r--- 252.00m /dev/sdb1(3) [POOL_tdata_rimage_1] snapper_thinp Iwi---r--- 252.00m /dev/sda1(3) [POOL_tdata_rmeta_0] snapper_thinp ewi---r--- 4.00m /dev/sdb1(2) [POOL_tdata_rmeta_1] snapper_thinp ewi---r--- 4.00m /dev/sda1(2) [POOL_tmeta] snapper_thinp ewi---r--- 4.00m POOL_tmeta_rimage_0(0),POOL_tmeta_rimage_1(0) [POOL_tmeta_rimage_0] snapper_thinp Iwi---r--- 4.00m /dev/sdb1(1) [POOL_tmeta_rimage_1] snapper_thinp Iwi---r--- 4.00m /dev/sda1(1) [POOL_tmeta_rmeta_0] snapper_thinp ewi---r--- 4.00m /dev/sdb1(0) [POOL_tmeta_rmeta_1] snapper_thinp ewi---r--- 4.00m /dev/sda1(0) contig_with snapper_thinp -wc------- 400.00m /dev/sdg1(0) [lvol0_pmspare] snapper_thinp ewi------- 4.00m /dev/sdb1(66) origin snapper_thinp Vwi---tz-- 1.00g POOL other1 snapper_thinp Vwi---tz-- 1.00g POOL other2 snapper_thinp Vwi---tz-- 1.00g POOL other3 snapper_thinp Vwi---tz-- 1.00g POOL other4 snapper_thinp Vwi---tz-- 1.00g POOL other5 snapper_thinp Vwi---tz-- 1.00g POOL lv_root vg_host122 -wi-ao---- 6.71g /dev/vda2(0) lv_swap vg_host122 -wi-ao---- 816.00m /dev/vda2(1718) /etc/lvm/cache/.cache.tmp: write error failed: No space left on device [root@host-122 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_host122-lv_root 6.5G 6.5G 0 100% / tmpfs 499M 26M 474M 6% /dev/shm /dev/vda1 477M 35M 417M 8% /boot # host-124 was doing exclusive activation lvm operations that deadlocked, presumably due to the inability to write in /etc/lvm/ on host-122? Removing snap volume snapper_thinp/contig_with lvremove -f /dev/snapper_thinp/contig_with couldn't remove volume contig_with Error locking on node host-123: Refusing activation of partial LV snapper_thinp/contig_with. Use '--activationmode partial' to override. Error locking on node host-122: Refusing activation of partial LV snapper_thinp/contig_with. Use '--activationmode partial' to override. Failed to activate contig_with. Releasing activation in critical section. # HOST-122 Jan 16 22:15:36 host-122 udevd-work[18877]: inotify_add_watch(6, /dev/dm-2, 10) failed: No such file or directory # HOST-123 Jan 16 22:15:36 host-123 udevd-work[16450]: inotify_add_watch(6, /dev/dm-2, 10) failed: No such file or directory # HOST-124 host-124 qarshd[22345]: Running cmdline: lvcreate --activate ey -n contig_without -C y -L 600M -s snapper_thinp/origin host-124 qarshd[22348]: Running cmdline: lvcreate --activate ey -n contig_with -C y -L 400M -s snapper_thinp/origin host-124 multipathd: uevent trigger error host-124 multipathd: dm-20: remove map (uevent) host-124 multipathd: dm-20: devmap not registered, can't remove host-124 qarshd[22438]: Running cmdline: lvremove -f /dev/snapper_thinp/contig_with host-124 multipathd: uevent trigger error host-124 multipathd: dm-21: remove map (uevent) host-124 multipathd: dm-21: devmap not registered, can't remove host-124 qarshd[22482]: Running cmdline: lvremove -f /dev/snapper_thinp/contig_with host-124 kernel: INFO: task lvremove:22494 blocked for more than 120 seconds. host-124 kernel: Not tainted 2.6.32-680.el6.x86_64 #1 host-124 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. host-124 kernel: lvremove D 0000000000000000 0 22494 22482 0x00000080 host-124 kernel: ffff88002ff9fa18 0000000000000086 ffff880000000000 00068014e647e8d0 host-124 kernel: ffff880035a35680 ffff8800396a5d70 000000000275e218 ffffffffa782e3f6 host-124 kernel: 000000003aa490e1 ffffffff81aa8380 ffff880016663ad8 ffff88002ff9ffd8 host-124 kernel: Call Trace: host-124 kernel: [<ffffffff8154b313>] io_schedule+0x73/0xc0 host-124 kernel: [<ffffffff811dae0d>] __blockdev_direct_IO_newtrunc+0xb7d/0x1270 host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90 host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90 host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20 host-124 kernel: [<ffffffff81287b0d>] ? get_disk+0x7d/0xf0 host-124 kernel: [<ffffffff811db577>] __blockdev_direct_IO+0x77/0xe0 host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20 host-124 kernel: [<ffffffff811d77f7>] blkdev_direct_IO+0x57/0x60 host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20 host-124 kernel: [<ffffffff8113055b>] generic_file_aio_read+0x6bb/0x700 host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90 host-124 kernel: [<ffffffff811d6ce1>] blkdev_aio_read+0x51/0x80 host-124 kernel: [<ffffffff81199d9a>] do_sync_read+0xfa/0x140 host-124 kernel: [<ffffffff810a6960>] ? autoremove_wake_function+0x0/0x40 host-124 kernel: [<ffffffff811d6b0c>] ? block_ioctl+0x3c/0x40 host-124 kernel: [<ffffffff811af852>] ? vfs_ioctl+0x22/0xa0 host-124 kernel: [<ffffffff8124822b>] ? selinux_file_permission+0xfb/0x150 host-124 kernel: [<ffffffff8123aea6>] ? security_file_permission+0x16/0x20 host-124 kernel: [<ffffffff8119a695>] vfs_read+0xb5/0x1a0 host-124 kernel: [<ffffffff8119b446>] ? fget_light_pos+0x16/0x50 host-124 kernel: [<ffffffff8119a9e1>] sys_read+0x51/0xb0 host-124 kernel: [<ffffffff810ee52e>] ? __audit_syscall_exit+0x25e/0x290 host-124 kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b host-124 kernel: INFO: task lvremove:22494 blocked for more than 120 seconds. host-124 kernel: Not tainted 2.6.32-680.el6.x86_64 #1 host-124 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. host-124 kernel: lvremove D 0000000000000000 0 22494 22482 0x00000080 host-124 kernel: ffff88002ff9fa18 0000000000000086 ffff880000000000 00068014e647e8d0 host-124 kernel: ffff880035a35680 ffff8800396a5d70 000000000275e218 ffffffffa782e3f6 host-124 kernel: 000000003aa490e1 ffffffff81aa8380 ffff880016663ad8 ffff88002ff9ffd8 host-124 kernel: Call Trace: host-124 kernel: [<ffffffff8154b313>] io_schedule+0x73/0xc0 host-124 kernel: [<ffffffff811dae0d>] __blockdev_direct_IO_newtrunc+0xb7d/0x1270 host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90 host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90 host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20 host-124 kernel: [<ffffffff81287b0d>] ? get_disk+0x7d/0xf0 host-124 kernel: [<ffffffff811db577>] __blockdev_direct_IO+0x77/0xe0 host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20 host-124 kernel: [<ffffffff811d77f7>] blkdev_direct_IO+0x57/0x60 host-124 kernel: [<ffffffff811d6770>] ? blkdev_get_block+0x0/0x20 host-124 kernel: [<ffffffff8113055b>] generic_file_aio_read+0x6bb/0x700 host-124 kernel: [<ffffffff812426e1>] ? avc_has_perm+0x71/0x90 host-124 kernel: [<ffffffff811d6ce1>] blkdev_aio_read+0x51/0x80 host-124 kernel: [<ffffffff81199d9a>] do_sync_read+0xfa/0x140 host-124 kernel: [<ffffffff810a6960>] ? autoremove_wake_function+0x0/0x40 host-124 kernel: [<ffffffff811d6b0c>] ? block_ioctl+0x3c/0x40 host-124 kernel: [<ffffffff811af852>] ? vfs_ioctl+0x22/0xa0 host-124 kernel: [<ffffffff8124822b>] ? selinux_file_permission+0xfb/0x150 host-124 kernel: [<ffffffff8123aea6>] ? security_file_permission+0x16/0x20 host-124 kernel: [<ffffffff8119a695>] vfs_read+0xb5/0x1a0 host-124 kernel: [<ffffffff8119b446>] ? fget_light_pos+0x16/0x50 host-124 kernel: [<ffffffff8119a9e1>] sys_read+0x51/0xb0 host-124 kernel: [<ffffffff810ee52e>] ? __audit_syscall_exit+0x25e/0x290 host-124 kernel: [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b Version-Release number of selected component (if applicable): 2.6.32-680.el6.x86_64 lvm2-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 lvm2-libs-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 lvm2-cluster-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 udev-147-2.73.el6_8.2 BUILT: Tue Aug 30 08:17:19 CDT 2016 device-mapper-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-libs-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-event-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-event-libs-1.02.117-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017 device-mapper-persistent-data-0.6.2-0.1.rc7.el6 BUILT: Tue Mar 22 08:58:09 CDT 2016 cmirror-2.02.143-12.el6 BUILT: Wed Jan 11 09:35:04 CST 2017