1382141 – exclusive activation of cached thin pool device is not maintained when attempting to merge thin snaps

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1382141 - exclusive activation of cached thin pool device is not maintained when attempting to merge thin snaps

Summary: exclusive activation of cached thin pool device is not maintained when attemp...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	lvm2
Sub Component:
Version:	7.3
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Zdenek Kabelac
QA Contact:	cluster-qe@redhat.com
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-10-05 21:09 UTC by Corey Marthaler
Modified:	2021-09-03 12:54 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-12-15 07:46:54 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
verbose lvconvert attempt (112.50 KB, text/plain) 2016-10-05 22:29 UTC, Corey Marthaler	no flags	Details
View All

Description Corey Marthaler 2016-10-05 21:09:17 UTC

Description of problem:
./snapper_thinp -r /usr/tests/sts-rhel7.3 -l /usr/tests/sts-rhel7.3/ -R ../../../resource-STSRHTS555.xml  -E host-081 -e invalidated_thin_snap_merge -c

creating lvm devices...
host-081: pvcreate /dev/sdg1 /dev/sdh1 /dev/sde1 /dev/sdf1 /dev/sdc1 /dev/sdd1 /dev/sda1
host-081: vgcreate  snapper_thinp /dev/sdg1 /dev/sdh1 /dev/sde1 /dev/sdf1 /dev/sdc1 /dev/sdd1 /dev/sda1

============================================================
Iteration 1 of 1 started at Wed Oct  5 14:56:26 CDT 2016
============================================================
SCENARIO - [invalidated_thin_snap_merge]
Create "invalidated" (full) thin snapshots and then verify that merge attempts will not cause problem
Making pool volume
Converting *cached* volume to thin pool data device
lvcreate --activate ey  --zero y -L 4M -n meta snapper_thinp /dev/sda1
lvcreate --activate ey  --zero y -L 500M -n POOL snapper_thinp /dev/sda1
lvcreate --activate ey --zero y -L 400M -n cpool snapper_thinp /dev/sdc1
lvcreate --activate ey --zero y -L 8M -n cpool_meta snapper_thinp /dev/sdc1
Create cache pool volume by combining the cache data and cache metadata (fast) volumes
lvconvert --yes --type cache-pool --poolmetadata snapper_thinp/cpool_meta snapper_thinp/cpool
  WARNING: Converting logical volume snapper_thinp/cpool and snapper_thinp/cpool_meta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Create cached volume by combining the cache pool (fast) and origin (slow) volumes
lvconvert --yes --type cache --cachepool snapper_thinp/cpool snapper_thinp/POOL
Create thin pool volume by combining the cached thin data and meta volumes
lvconvert --zero y --thinpool snapper_thinp/POOL --poolmetadata meta --yes
  WARNING: Converting logical volume snapper_thinp/POOL and snapper_thinp/meta to thin pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)

Sanity checking pool device (POOL) metadata
examining superblock
examining devices tree
examining mapping tree
checking space map counts


Making origin volume
lvcreate --activate ey --virtualsize 100M -T snapper_thinp/POOL -n origin
lvcreate --activate ey --virtualsize 100M -T snapper_thinp/POOL -n other1
lvcreate --activate ey -V 100M -T snapper_thinp/POOL -n other2
lvcreate --activate ey -V 100M -T snapper_thinp/POOL -n other3
lvcreate --activate ey --virtualsize 100M -T snapper_thinp/POOL -n other4
lvcreate --activate ey -V 100M -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (600.00 MiB) exceeds the size of thin pool snapper_thinp/POOL (500.00 MiB)!

lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n invalid1
Filling snapshot /dev/snapper_thinp/invalid1
dd if=/dev/zero of=/dev/snapper_thinp/invalid1 bs=1M count=101
dd: error writing ‘/dev/snapper_thinp/invalid1’: No space left on device
101+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 5.09591 s, 20.6 MB/s
Attempt to merge back an invalidated snapshot volume
lvconvert --merge /dev/snapper_thinp/invalid1 --yes

lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n invalid2
Filling snapshot /dev/snapper_thinp/invalid2
dd if=/dev/zero of=/dev/snapper_thinp/invalid2 bs=1M count=101
dd: error writing ‘/dev/snapper_thinp/invalid2’: No space left on device
101+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 5.97496 s, 17.5 MB/s
Attempt to merge back an invalidated snapshot volume
lvconvert --merge /dev/snapper_thinp/invalid2 --yes


# host-082

[432011.901213] device-mapper: space map common: unable to decrement a reference count below 0
[432011.903167] device-mapper: cache: 253:6: metadata operation 'dm_cache_set_dirty' failed: error = -22
[432011.905228] device-mapper: cache: 253:6: aborting current metadata transaction
[432011.907740] ------------[ cut here ]------------
[432011.908881] WARNING: at drivers/md/dm-bufio.c:1500 dm_bufio_client_destroy+0x1e0/0x1f0 [dm_bufio]()
[432011.910906] Modules linked in: dm_cache_smq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_userspace gfs2 ip6table_filter ip6_tables binfmt_misc dlm sd_mod crc_t10dif crct10dif_generic crct10dif_common sg iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter i2c_piix4 ppdev virtio_balloon pcspkr i6300esb i2c_core parport_pc parport dm_multipath nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi virtio_net virtio_blk ata_piix libata serio_raw virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
[432011.924697] CPU: 0 PID: 3157 Comm: clvmd Tainted: G        W      ------------   3.10.0-511.el7.x86_64 #1
[432011.926857] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[432011.928171]  0000000000000000 0000000067106829 ffff88003b5e7af8 ffffffff81685e8c
[432011.929995]  ffff88003b5e7b30 ffffffff81085820 ffff88001742f400 ffff88001742f428
[432011.931837]  0000000000000000 ffff8800150ca818 ffff88001742f428 ffff88003b5e7b40
[432011.933677] Call Trace:
[432011.934297]  [<ffffffff81685e8c>] dump_stack+0x19/0x1b
[432011.935487]  [<ffffffff81085820>] warn_slowpath_common+0x70/0xb0
[432011.936890]  [<ffffffff8108596a>] warn_slowpath_null+0x1a/0x20
[432011.938238]  [<ffffffffa035caa0>] dm_bufio_client_destroy+0x1e0/0x1f0 [dm_bufio]
[432011.939923]  [<ffffffffa03ec2a5>] dm_block_manager_destroy+0x15/0x20 [dm_persistent_data]
[432011.941781]  [<ffffffffa044c9f8>] __destroy_persistent_data_objects+0x28/0x30 [dm_cache]
[432011.943620]  [<ffffffffa044e8c5>] dm_cache_metadata_abort+0x25/0x60 [dm_cache]
[432011.945263]  [<ffffffffa0447c9d>] metadata_operation_failed+0x8d/0x110 [dm_cache]
[432011.946995]  [<ffffffffa044bab6>] cache_postsuspend+0x296/0x4b0 [dm_cache]
[432011.948570]  [<ffffffff8131b67b>] ? kobject_uevent_env+0x1ab/0x620
[432011.949987]  [<ffffffff810b1600>] ? wake_up_atomic_t+0x30/0x30
[432011.951354]  [<ffffffffa00068fa>] dm_table_postsuspend_targets+0x4a/0x60 [dm_mod]
[432011.953056]  [<ffffffffa0002b23>] __dm_destroy+0x2e3/0x320 [dm_mod]
[432011.954484]  [<ffffffffa0003833>] dm_destroy+0x13/0x20 [dm_mod]
[432011.955859]  [<ffffffffa00093de>] dev_remove+0x11e/0x180 [dm_mod]
[432011.957254]  [<ffffffffa00092c0>] ? dev_suspend+0x250/0x250 [dm_mod]
[432011.958718]  [<ffffffffa0009a35>] ctl_ioctl+0x1e5/0x500 [dm_mod]
[432011.960135]  [<ffffffffa0009d63>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
[432011.961584]  [<ffffffff81211eb5>] do_vfs_ioctl+0x2d5/0x4b0
[432011.962916]  [<ffffffff812aea3e>] ? file_has_perm+0xae/0xc0
[432011.964201]  [<ffffffff81292e01>] ? wake_up_sem_queue_do+0x11/0x60
[432011.965628]  [<ffffffff81212131>] SyS_ioctl+0xa1/0xc0
[432011.966807]  [<ffffffff81696489>] system_call_fastpath+0x16/0x1b
[432011.968171] ---[ end trace 7f98a93d71d141b8 ]---
[432011.969256] device-mapper: bufio: leaked buffer 7, hold count 1, list 0
[432011.970814] ------------[ cut here ]------------
[432011.971770] kernel BUG at drivers/md/dm-bufio.c:1516!
[432011.971770] invalid opcode: 0000 [#1] SMP 
[432011.971770] Modules linked in: dm_cache_smq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_userspace gfs2 ip6table_filter ip6_tables binfmt_misc dlm sd_mod crc_t10dif crct10dif_generic crct10dif_common sg iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter i2c_piix4 ppdev virtio_balloon pcspkr i6300esb i2c_core parport_pc parport dm_multipath nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi virtio_net virtio_blk ata_piix libata serio_raw virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
[432011.971770] CPU: 0 PID: 3157 Comm: clvmd Tainted: G        W      ------------   3.10.0-511.el7.x86_64 #1
[432011.971770] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[432011.971770] task: ffff88003bac3ec0 ti: ffff88003b5e4000 task.ti: ffff88003b5e4000
[432011.971770] RIP: 0010:[<ffffffffa035ca73>]  [<ffffffffa035ca73>] dm_bufio_client_destroy+0x1b3/0x1f0 [dm_bufio]
[432011.971770] RSP: 0018:ffff88003b5e7b50  EFLAGS: 00010287
[432011.971770] RAX: 0000000000000001 RBX: ffff88001742f400 RCX: 0000000000000000
[432011.971770] RDX: 0000000000000000 RSI: ffff88003fc0f838 RDI: ffff88003fc0f838
[432011.971770] RBP: ffff88003b5e7b78 R08: 0000000000000096 R09: 0000000000000492
[432011.971770] R10: 3120746e756f6320 R11: 30207473696c202c R12: ffff88001742f448
[432011.971770] R13: 0000000000000002 R14: ffff88001742f438 R15: ffff88001742f428
[432011.971770] FS:  00007f92c2c6c700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[432011.971770] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[432011.971770] CR2: 00007febc0be02c8 CR3: 000000003b4fa000 CR4: 00000000000006f0
[432011.971770] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[432011.971770] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[432011.971770] Stack:
[432011.971770]  ffff88000ca3c950 ffff88003c82d1a0 00000000ffffffea ffff880012fbb85c
[432011.971770]  00000000ffffffea ffff88003b5e7b90 ffffffffa03ec2a5 ffff88002b739c00
[432011.971770]  ffff88003b5e7ba8 ffffffffa044c9f8 ffff88002b739c00 ffff88003b5e7bc8
[432011.971770] Call Trace:
[432011.971770]  [<ffffffffa03ec2a5>] dm_block_manager_destroy+0x15/0x20 [dm_persistent_data]
[432011.971770]  [<ffffffffa044c9f8>] __destroy_persistent_data_objects+0x28/0x30 [dm_cache]
[432011.971770]  [<ffffffffa044e8c5>] dm_cache_metadata_abort+0x25/0x60 [dm_cache]
[432011.971770]  [<ffffffffa0447c9d>] metadata_operation_failed+0x8d/0x110 [dm_cache]
[432011.971770]  [<ffffffffa044bab6>] cache_postsuspend+0x296/0x4b0 [dm_cache]
[432011.971770]  [<ffffffff8131b67b>] ? kobject_uevent_env+0x1ab/0x620
[432011.971770]  [<ffffffff810b1600>] ? wake_up_atomic_t+0x30/0x30
[432011.971770]  [<ffffffffa00068fa>] dm_table_postsuspend_targets+0x4a/0x60 [dm_mod]
[432011.971770]  [<ffffffffa0002b23>] __dm_destroy+0x2e3/0x320 [dm_mod]
[432011.971770]  [<ffffffffa0003833>] dm_destroy+0x13/0x20 [dm_mod]
[432011.971770]  [<ffffffffa00093de>] dev_remove+0x11e/0x180 [dm_mod]
[432011.971770]  [<ffffffffa00092c0>] ? dev_suspend+0x250/0x250 [dm_mod]
[432011.971770]  [<ffffffffa0009a35>] ctl_ioctl+0x1e5/0x500 [dm_mod]
[432011.971770]  [<ffffffffa0009d63>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
[432011.971770]  [<ffffffff81211eb5>] do_vfs_ioctl+0x2d5/0x4b0
[432011.971770]  [<ffffffff812aea3e>] ? file_has_perm+0xae/0xc0
[432011.971770]  [<ffffffff81292e01>] ? wake_up_sem_queue_do+0x11/0x60
[432011.971770]  [<ffffffff81212131>] SyS_ioctl+0xa1/0xc0
[432011.971770]  [<ffffffff81696489>] system_call_fastpath+0x16/0x1b
[432011.971770] Code: 18 49 39 c4 4c 8d 70 e8 75 da b8 01 00 00 00 e9 d0 fe ff ff 0f 0b 31 f6 48 c7 c7 a0 e4 35 a0 31 c0 e8 68 29 32 e1 e9 7b ff ff ff <0f> 0b be 01 00 00 00 48 c7 c7 a0 e4 35 a0 31 c0 e8 4e 29 32 e1 
[432011.971770] RIP  [<ffffffffa035ca73>] dm_bufio_client_destroy+0x1b3/0x1f0 [dm_bufio]
[432011.971770]  RSP <ffff88003b5e7b50>
[432012.057146] ---[ end trace 7f98a93d71d141b9 ]---
[432012.058353] Kernel panic - not syncing: Fatal exception





# host-083

[432015.228371] device-mapper: space map common: unable to decrement a reference count below 0
[432015.230325] device-mapper: cache: 253:6: metadata operation 'dm_cache_set_dirty' failed: error = -22
[432015.232339] device-mapper: cache: 253:6: aborting current metadata transaction
[432015.234879] ------------[ cut here ]------------
[432015.235980] WARNING: at drivers/md/dm-bufio.c:1500 dm_bufio_client_destroy+0x1e0/0x1f0 [dm_bufio]()
[432015.238695] Modules linked in: dm_cache_smq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_userspace gfs2 dlm sd_mod crc_t10dif crct10dif_generic crct10dif_common sg iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter i2c_piix4 ppdev pcspkr i6300esb virtio_balloon i2c_core parport_pc parport dm_multipath nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi virtio_blk virtio_net ata_piix serio_raw libata virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
[432015.253728] CPU: 0 PID: 3151 Comm: clvmd Not tainted 3.10.0-511.el7.x86_64 #1
[432015.255942] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[432015.257371]  0000000000000000 0000000011a565ce ffff88002d26baf8 ffffffff81685e8c
[432015.260464]  ffff88002d26bb30 ffffffff81085820 ffff880014ab9a00 ffff880014ab9a28
[432015.262837]  0000000000000000 ffff88003a9d9818 ffff880014ab9a28 ffff88002d26bb40
[432015.265222] Call Trace:
[432015.266224]  [<ffffffff81685e8c>] dump_stack+0x19/0x1b
[432015.268214]  [<ffffffff81085820>] warn_slowpath_common+0x70/0xb0
[432015.269651]  [<ffffffff8108596a>] warn_slowpath_null+0x1a/0x20
[432015.271532]  [<ffffffffa0343aa0>] dm_bufio_client_destroy+0x1e0/0x1f0 [dm_bufio]
[432015.273207]  [<ffffffffa03552a5>] dm_block_manager_destroy+0x15/0x20 [dm_persistent_data]
[432015.275003]  [<ffffffffa039e9f8>] __destroy_persistent_data_objects+0x28/0x30 [dm_cache]
[432015.276835]  [<ffffffffa03a08c5>] dm_cache_metadata_abort+0x25/0x60 [dm_cache]
[432015.279478]  [<ffffffffa0399c9d>] metadata_operation_failed+0x8d/0x110 [dm_cache]
[432015.281910]  [<ffffffffa039dab6>] cache_postsuspend+0x296/0x4b0 [dm_cache]
[432015.283472]  [<ffffffff8131b67b>] ? kobject_uevent_env+0x1ab/0x620
[432015.285355]  [<ffffffff810b1600>] ? wake_up_atomic_t+0x30/0x30
[432015.286689]  [<ffffffffa00068fa>] dm_table_postsuspend_targets+0x4a/0x60 [dm_mod]
[432015.288358]  [<ffffffffa0002b23>] __dm_destroy+0x2e3/0x320 [dm_mod]
[432015.289773]  [<ffffffffa0003833>] dm_destroy+0x13/0x20 [dm_mod]
[432015.291123]  [<ffffffffa00093de>] dev_remove+0x11e/0x180 [dm_mod]
[432015.293458]  [<ffffffffa00092c0>] ? dev_suspend+0x250/0x250 [dm_mod]
[432015.295164]  [<ffffffffa0009a35>] ctl_ioctl+0x1e5/0x500 [dm_mod]
[432015.296523]  [<ffffffffa0009d63>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
[432015.297890]  [<ffffffff81211eb5>] do_vfs_ioctl+0x2d5/0x4b0
[432015.299789]  [<ffffffff812aea3e>] ? file_has_perm+0xae/0xc0
[432015.301117]  [<ffffffff81292e01>] ? wake_up_sem_queue_do+0x11/0x60
[432015.302498]  [<ffffffff81212131>] SyS_ioctl+0xa1/0xc0
[432015.303647]  [<ffffffff81696489>] system_call_fastpath+0x16/0x1b
[432015.304996] ---[ end trace 6dba8a4b2fdcc54c ]---
[432015.306763] device-mapper: bufio: leaked buffer 7, hold count 1, list 0
[432015.309293] ------------[ cut here ]------------
[432015.310243] kernel BUG at drivers/md/dm-bufio.c:1516!
[432015.310243] invalid opcode: 0000 [#1] SMP 
[432015.310243] Modules linked in: dm_cache_smq dm_cache dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio dm_log_userspace gfs2 dlm sd_mod crc_t10dif crct10dif_generic crct10dif_common sg iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi iptable_filter i2c_piix4 ppdev pcspkr i6300esb virtio_balloon i2c_core parport_pc parport dm_multipath nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi virtio_blk virtio_net ata_piix serio_raw libata virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
[432015.310243] CPU: 0 PID: 3151 Comm: clvmd Tainted: G        W      ------------   3.10.0-511.el7.x86_64 #1
[432015.310243] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
[432015.310243] task: ffff88003a16edd0 ti: ffff88002d268000 task.ti: ffff88002d268000
[432015.310243] RIP: 0010:[<ffffffffa0343a73>]  [<ffffffffa0343a73>] dm_bufio_client_destroy+0x1b3/0x1f0 [dm_bufio]
[432015.310243] RSP: 0018:ffff88002d26bb50  EFLAGS: 00010206
[432015.310243] RAX: 0000000000000001 RBX: ffff880014ab9a00 RCX: 0000000000000000
[432015.310243] RDX: 0000000000000000 RSI: ffff88003fc0f838 RDI: ffff88003fc0f838
[432015.310243] RBP: ffff88002d26bb78 R08: 0000000000000096 R09: 00000000000004a3
[432015.310243] R10: 3120746e756f6320 R11: 30207473696c202c R12: ffff880014ab9a48
[432015.310243] R13: 0000000000000002 R14: ffff880014ab9a38 R15: ffff880014ab9a28
[432015.310243] FS:  00007f60e23e0700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
[432015.310243] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[432015.310243] CR2: 00007f55c0001b50 CR3: 000000002d244000 CR4: 00000000000006f0
[432015.310243] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[432015.310243] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[432015.310243] Stack:
[432015.310243]  ffff88001ae23f80 ffff88001d8e99a0 00000000ffffffea ffff88003c86385c
[432015.310243]  00000000ffffffea ffff88002d26bb90 ffffffffa03552a5 ffff880035803800
[432015.310243]  ffff88002d26bba8 ffffffffa039e9f8 ffff880035803800 ffff88002d26bbc8
[432015.310243] Call Trace:
[432015.310243]  [<ffffffffa03552a5>] dm_block_manager_destroy+0x15/0x20 [dm_persistent_data]
[432015.310243]  [<ffffffffa039e9f8>] __destroy_persistent_data_objects+0x28/0x30 [dm_cache]
[432015.310243]  [<ffffffffa03a08c5>] dm_cache_metadata_abort+0x25/0x60 [dm_cache]
[432015.310243]  [<ffffffffa0399c9d>] metadata_operation_failed+0x8d/0x110 [dm_cache]
[432015.310243]  [<ffffffffa039dab6>] cache_postsuspend+0x296/0x4b0 [dm_cache]
[432015.310243]  [<ffffffff8131b67b>] ? kobject_uevent_env+0x1ab/0x620
[432015.310243]  [<ffffffff810b1600>] ? wake_up_atomic_t+0x30/0x30
[432015.310243]  [<ffffffffa00068fa>] dm_table_postsuspend_targets+0x4a/0x60 [dm_mod]
[432015.310243]  [<ffffffffa0002b23>] __dm_destroy+0x2e3/0x320 [dm_mod]
[432015.310243]  [<ffffffffa0003833>] dm_destroy+0x13/0x20 [dm_mod]
[432015.310243]  [<ffffffffa00093de>] dev_remove+0x11e/0x180 [dm_mod]
[432015.310243]  [<ffffffffa00092c0>] ? dev_suspend+0x250/0x250 [dm_mod]
[432015.310243]  [<ffffffffa0009a35>] ctl_ioctl+0x1e5/0x500 [dm_mod]
[432015.310243]  [<ffffffffa0009d63>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
[432015.310243]  [<ffffffff81211eb5>] do_vfs_ioctl+0x2d5/0x4b0
[432015.310243]  [<ffffffff812aea3e>] ? file_has_perm+0xae/0xc0
[432015.310243]  [<ffffffff81292e01>] ? wake_up_sem_queue_do+0x11/0x60
[432015.310243]  [<ffffffff81212131>] SyS_ioctl+0xa1/0xc0
[432015.310243]  [<ffffffff81696489>] system_call_fastpath+0x16/0x1b
[432015.310243] Code: 18 49 39 c4 4c 8d 70 e8 75 da b8 01 00 00 00 e9 d0 fe ff ff 0f 0b 31 f6 48 c7 c7 a0 54 34 a0 31 c0 e8 68 b9 33 e1 e9 7b ff ff ff <0f> 0b be 01 00 00 00 48 c7 c7 a0 54 34 a0 31 c0 e8 4e b9 33 e1 
[432015.310243] RIP  [<ffffffffa0343a73>] dm_bufio_client_destroy+0x1b3/0x1f0 [dm_bufio]
[432015.310243]  RSP <ffff88002d26bb50>
[432015.400346] ---[ end trace 6dba8a4b2fdcc54d ]---
[432015.401541] Kernel panic - not syncing: Fatal exception

Version-Release number of selected component (if applicable):
3.10.0-511.el7.x86_64

lvm2-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
lvm2-libs-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
lvm2-cluster-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-libs-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-event-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-event-libs-1.02.135-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016
device-mapper-persistent-data-0.6.3-1.el7    BUILT: Fri Jul 22 05:29:13 CDT 2016
cmirror-2.02.166-1.el7    BUILT: Wed Sep 28 02:26:52 CDT 2016

Comment 1 Corey Marthaler 2016-10-05 21:22:39 UTC

Another result (82 and 83 are the non exclusively active cluster nodes):  

[...]
lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n invalid2
Filling snapshot /dev/snapper_thinp/invalid2
dd if=/dev/zero of=/dev/snapper_thinp/invalid2 bs=1M count=101
dd: error writing ‘/dev/snapper_thinp/invalid2’: No space left on device
101+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 6.44011 s, 16.3 MB/s
Attempt to merge back an invalidated snapshot volume
lvconvert --merge /dev/snapper_thinp/invalid2 --yes
  Error locking on node 3: Check of pool snapper_thinp/cpool failed (status:1). Manual repair required!
  Error locking on node 2: Check of pool snapper_thinp/cpool failed (status:1). Manual repair required!
  Failed to reactivate origin snapper_thinp/origin.
couldn't merge invalidated snap



Oct  5 15:44:04 host-082 kernel: device-mapper: space map common: bitmap check failed: blocknr 34359738877 != wanted 5
Oct  5 15:44:04 host-082 kernel: device-mapper: block manager: sm_bitmap validator check failed for block 5
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: metadata operation 'dm_cache_set_dirty' failed: error = -15
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: aborting current metadata transaction
Oct  5 15:44:04 host-082 kernel: device-mapper: space map common: index_check failed: csum 828282714 != wanted 828193943
Oct  5 15:44:04 host-082 kernel: device-mapper: block manager: index validator check failed for block 19
Oct  5 15:44:04 host-082 kernel: device-mapper: transaction manager: couldn't open metadata space map
Oct  5 15:44:04 host-082 kernel: device-mapper: cache metadata: tm_open_with_sm failed
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: failed to abort metadata transaction
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: unable to read needs_check flag, setting failure mode
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: switching cache to fail mode
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: unable to read needs_check flag, setting failure mode
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: could not write dirty bitset
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: could not write discard bitset
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: could not write hints
Oct  5 15:44:04 host-082 kernel: device-mapper: cache: 253:6: could not write cache metadata




Oct  5 15:44:04 host-083 kernel: device-mapper: space map common: bitmap check failed: blocknr 34359738877 != wanted 5
Oct  5 15:44:04 host-083 kernel: device-mapper: block manager: sm_bitmap validator check failed for block 5
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: metadata operation 'dm_cache_set_dirty' failed: error = -15
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: aborting current metadata transaction
Oct  5 15:44:04 host-083 kernel: device-mapper: space map common: index_check failed: csum 828282714 != wanted 828193943
Oct  5 15:44:04 host-083 kernel: device-mapper: block manager: index validator check failed for block 19
Oct  5 15:44:04 host-083 kernel: device-mapper: transaction manager: couldn't open metadata space map
Oct  5 15:44:04 host-083 kernel: device-mapper: cache metadata: tm_open_with_sm failed
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: failed to abort metadata transaction
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: unable to read needs_check flag, setting failure mode
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: switching cache to fail mode
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: unable to read needs_check flag, setting failure mode
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: could not write dirty bitset
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: could not write discard bitset
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: could not write hints
Oct  5 15:44:04 host-083 kernel: device-mapper: cache: 253:6: could not write cache metadata

Comment 2 Corey Marthaler 2016-10-05 22:15:38 UTC

Simpler set of cmds to reproduce:

[root@host-081 ~]# pcs status
Cluster name: STSRHTS555
Stack: corosync
Current DC: host-083 (version 1.1.15-11.el7-e174ec8) - partition with quorum
Last updated: Wed Oct  5 17:08:07 2016          Last change: Wed Oct  5 15:50:05 2016 by root via cibadmin on host-081

3 nodes and 9 resources configured

Online: [ host-081 host-082 host-083 ]

Full list of resources:

 fence-host-081 (stonith:fence_xvm):    Started host-081
 fence-host-082 (stonith:fence_xvm):    Started host-082
 fence-host-083 (stonith:fence_xvm):    Started host-083
 Clone Set: dlm-clone [dlm]
     Started: [ host-081 host-082 host-083 ]
 Clone Set: clvmd-clone [clvmd]
     Started: [ host-081 host-082 host-083 ]

Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled



[root@host-081 ~]# pvcreate /dev/sdf1 /dev/sda1 /dev/sdh1 /dev/sdg1 /dev/sdc1 /dev/sdb1 /dev/sdd1
  Physical volume "/dev/sdf1" successfully created.
  Physical volume "/dev/sda1" successfully created.
  Physical volume "/dev/sdh1" successfully created.
  Physical volume "/dev/sdg1" successfully created.
  Physical volume "/dev/sdc1" successfully created.
  Physical volume "/dev/sdb1" successfully created.
  Physical volume "/dev/sdd1" successfully created.
[root@host-081 ~]# vgcreate  snapper_thinp /dev/sdf1 /dev/sda1 /dev/sdh1 /dev/sdg1 /dev/sdc1 /dev/sdb1 /dev/sdd1
  Clustered volume group "snapper_thinp" successfully created

[root@host-081 ~]# lvcreate --activate ey --profile thin-performance --zero y -L 4M -n meta snapper_thinp /dev/sda1
  Logical volume "meta" created.
[root@host-081 ~]# lvcreate --activate ey --profile thin-performance --zero y -L 500M -n POOL snapper_thinp /dev/sda1
  Logical volume "POOL" created.
[root@host-081 ~]# lvcreate --activate ey --zero y -L 400M -n cpool snapper_thinp /dev/sdb1
  Logical volume "cpool" created.
[root@host-081 ~]# lvcreate --activate ey --zero y -L 8M -n cpool_meta snapper_thinp /dev/sdb1
  Logical volume "cpool_meta" created.
[root@host-081 ~]# lvconvert --yes --type cache-pool --poolmetadata snapper_thinp/cpool_meta snapper_thinp/cpool
  WARNING: Converting logical volume snapper_thinp/cpool and snapper_thinp/cpool_meta to cache pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted snapper_thinp/cpool to cache pool.
[root@host-081 ~]# lvconvert --yes --type cache --cachepool snapper_thinp/cpool snapper_thinp/POOL
  Logical volume snapper_thinp/POOL is now cached.
[root@host-081 ~]# lvconvert --zero y --thinpool snapper_thinp/POOL --poolmetadata meta --yes
  WARNING: Converting logical volume snapper_thinp/POOL and snapper_thinp/meta to thin pool's data and metadata volumes with metadata wiping.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted snapper_thinp/POOL to thin pool.

[root@host-081 ~]# lvcreate --activate ey --virtualsize 100M -T snapper_thinp/POOL -n origin
  Using default stripesize 64.00 KiB.
  Logical volume "origin" created.
[root@host-081 ~]# lvcreate --activate ey -k n -s /dev/snapper_thinp/origin -n invalid1
  Using default stripesize 64.00 KiB.
  Logical volume "invalid1" created.

# EXCLUSIVE ACTIVE
[root@host-081 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool    Origin             Data%  Meta%  Cpy%Sync Devices            
  POOL               snapper_thinp twi-aotz-- 500.00m                            0.00   0.98            POOL_tdata(0)      
  [POOL_tdata]       snapper_thinp Cwi-aoC--- 500.00m [cpool] [POOL_tdata_corig] 0.03   1.37   0.00     POOL_tdata_corig(0)
  [POOL_tdata_corig] snapper_thinp owi-aoC--- 500.00m                                                   /dev/sda1(1)       
  [POOL_tmeta]       snapper_thinp ewi-ao----   4.00m                                                   /dev/sda1(0)       
  [cpool]            snapper_thinp Cwi---C--- 400.00m                            0.03   1.37   0.00     cpool_cdata(0)     
  [cpool_cdata]      snapper_thinp Cwi-ao---- 400.00m                                                   /dev/sdb1(0)       
  [cpool_cmeta]      snapper_thinp ewi-ao----   8.00m                                                   /dev/sdb1(100)     
  invalid1           snapper_thinp Vwi-a-tz-- 100.00m POOL    origin             0.00
  [lvol0_pmspare]    snapper_thinp ewi-------   8.00m                                                   /dev/sdf1(0)       
  origin             snapper_thinp Vwi-a-tz-- 100.00m POOL                       0.00

# INACTIVE
[root@host-082 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool    Origin             Data%  Meta%  Cpy%Sync Devices            
  POOL               snapper_thinp twi---tz-- 500.00m                                                   POOL_tdata(0)      
  [POOL_tdata]       snapper_thinp Cwi---C--- 500.00m [cpool] [POOL_tdata_corig]                        POOL_tdata_corig(0)
  [POOL_tdata_corig] snapper_thinp owi---C--- 500.00m                                                   /dev/sdf1(1)       
  [POOL_tmeta]       snapper_thinp ewi-------   4.00m                                                   /dev/sdf1(0)       
  [cpool]            snapper_thinp Cwi---C--- 400.00m                                                   cpool_cdata(0)     
  [cpool_cdata]      snapper_thinp Cwi------- 400.00m                                                   /dev/sdh1(0)       
  [cpool_cmeta]      snapper_thinp ewi-------   8.00m                                                   /dev/sdh1(100)     
  invalid1           snapper_thinp Vwi---tz-- 100.00m POOL    origin
  [lvol0_pmspare]    snapper_thinp ewi-------   8.00m                                                   /dev/sda1(0)       
  origin             snapper_thinp Vwi---tz-- 100.00m POOL

# INACTIVE
[root@host-083 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool    Origin             Data%  Meta%  Cpy%Sync Devices            
  POOL               snapper_thinp twi---tz-- 500.00m                                                   POOL_tdata(0)      
  [POOL_tdata]       snapper_thinp Cwi---C--- 500.00m [cpool] [POOL_tdata_corig]                        POOL_tdata_corig(0)
  [POOL_tdata_corig] snapper_thinp owi---C--- 500.00m                                                   /dev/sda1(1)       
  [POOL_tmeta]       snapper_thinp ewi-------   4.00m                                                   /dev/sda1(0)       
  [cpool]            snapper_thinp Cwi---C--- 400.00m                                                   cpool_cdata(0)     
  [cpool_cdata]      snapper_thinp Cwi------- 400.00m                                                   /dev/sdb1(0)       
  [cpool_cmeta]      snapper_thinp ewi-------   8.00m                                                   /dev/sdb1(100)     
  invalid1           snapper_thinp Vwi---tz-- 100.00m POOL    origin
  [lvol0_pmspare]    snapper_thinp ewi-------   8.00m                                                   /dev/sdh1(0)       
  origin             snapper_thinp Vwi---tz-- 100.00m POOL

# After this all nodes will report the volumes as active
[root@host-081 ~]# lvconvert --merge /dev/snapper_thinp/invalid1 --yes
  Merging of thin snapshot snapper_thinp/origin will occur on next activation of snapper_thinp/invalid1.

# ACTIVE
[root@host-081 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool    Origin             Data%  Meta%  Cpy%Sync Devices
  POOL               snapper_thinp twi-aotz-- 500.00m                            0.00   0.98            POOL_tdata(0)
  [POOL_tdata]       snapper_thinp Cwi-aoC--- 500.00m [cpool] [POOL_tdata_corig] 0.03   1.37   0.00     POOL_tdata_corig(0)
  [POOL_tdata_corig] snapper_thinp owi-aoC--- 500.00m                                                   /dev/sda1(1)
  [POOL_tmeta]       snapper_thinp ewi-ao----   4.00m                                                   /dev/sda1(0)
  [cpool]            snapper_thinp Cwi---C--- 400.00m                            0.03   1.37   0.00     cpool_cdata(0)
  [cpool_cdata]      snapper_thinp Cwi-ao---- 400.00m                                                   /dev/sdb1(0)
  [cpool_cmeta]      snapper_thinp ewi-ao----   8.00m                                                   /dev/sdb1(100)
  [lvol0_pmspare]    snapper_thinp ewi-------   8.00m                                                   /dev/sdf1(0)
  origin             snapper_thinp Vwi-a-tz-- 100.00m POOL                       0.00

# ACTIVE
[root@host-082 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool    Origin             Data%  Meta%  Cpy%Sync Devices
  POOL               snapper_thinp twi---tz-- 500.00m                            0.00   0.98            POOL_tdata(0)
  [POOL_tdata]       snapper_thinp Cwi-aoC--- 500.00m [cpool] [POOL_tdata_corig] 0.03   1.37   100.00   POOL_tdata_corig(0)
  [POOL_tdata_corig] snapper_thinp owi-aoC--- 500.00m                                                   /dev/sdf1(1)
  [POOL_tmeta]       snapper_thinp ewi-ao----   4.00m                                                   /dev/sdf1(0)
  [cpool]            snapper_thinp Cwi---C--- 400.00m                            0.03   1.37   100.00   cpool_cdata(0)
  [cpool_cdata]      snapper_thinp Cwi-ao---- 400.00m                                                   /dev/sdh1(0)
  [cpool_cmeta]      snapper_thinp ewi-ao----   8.00m                                                   /dev/sdh1(100)
  [lvol0_pmspare]    snapper_thinp ewi-------   8.00m                                                   /dev/sda1(0)
  origin             snapper_thinp Vwi-a-tz-- 100.00m POOL                       0.00

# ACTIVE
[root@host-083 ~]# lvs -a -o +devices
  LV                 VG            Attr       LSize   Pool    Origin             Data%  Meta%  Cpy%Sync Devices
  POOL               snapper_thinp twi---tz-- 500.00m                            0.00   0.98            POOL_tdata(0)
  [POOL_tdata]       snapper_thinp Cwi-aoC--- 500.00m [cpool] [POOL_tdata_corig] 0.03   1.37   100.00   POOL_tdata_corig(0)
  [POOL_tdata_corig] snapper_thinp owi-aoC--- 500.00m                                                   /dev/sda1(1)
  [POOL_tmeta]       snapper_thinp ewi-ao----   4.00m                                                   /dev/sda1(0)
  [cpool]            snapper_thinp Cwi---C--- 400.00m                            0.03   1.37   100.00   cpool_cdata(0)
  [cpool_cdata]      snapper_thinp Cwi-ao---- 400.00m                                                   /dev/sdb1(0)
  [cpool_cmeta]      snapper_thinp ewi-ao----   8.00m                                                   /dev/sdb1(100)
  [lvol0_pmspare]    snapper_thinp ewi-------   8.00m                                                   /dev/sdh1(0)
  origin             snapper_thinp Vwi-a-tz-- 100.00m POOL                       0.00

Comment 3 Corey Marthaler 2016-10-05 22:29:20 UTC

Created attachment 1207702 [details]
verbose lvconvert attempt

Comment 6 RHEL Program Management 2020-12-15 07:46:54 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Note You need to log in before you can comment on or make changes to this bug.