RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 886694 - Failure to delete thin volumes with lvmetad running
Summary: Failure to delete thin volumes with lvmetad running
Keywords:
Status: CLOSED DUPLICATE of bug 889454
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.4
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Zdenek Kabelac
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-12-12 21:58 UTC by Corey Marthaler
Modified: 2013-01-15 15:07 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-01-15 15:07:38 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Corey Marthaler 2012-12-12 21:58:07 UTC 
Description of problem:
[cmarthal@silver bin]$ ./snapper_thinp -l /home/msp/cmarthal/work/sts/sts-root -r /usr/tests/sts-rhel6.4 -o hayes-01 -e poolmetadatasizes -i 10 

1 disk(s) to be used:
        hayes-01=/dev/etherd/e1.1

on hayes-01...
dicing /dev/etherd/e1.1 into 2... 
re-reading disks on hayes-01...
Zeroing out the new partitions.../dev/etherd/e1.1p1.../dev/etherd/e1.1p2...
creating lvm devices...
Create 2 PV(s) for snapper_thinp on hayes-01
Create VG snapper_thinp on hayes-01
============================================================
Iteration 1 of 10 started at Wed Dec 12 14:55:06 CST 2012
============================================================
SCENARIO - [poolmetadatasizes]
Create pool volumes of varying pool metadata sizes (--poolmetadatasize)

Creating [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_4.00m --poolmetadatasize 4.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_4.00m -n origin_4.00m
lvcreate -s /dev/snapper_thinp/origin_4.00m -n snap_4.00m

Creating [16.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_16.00m --poolmetadatasize 16.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_16.00m -n origin_16.00m
lvcreate -s /dev/snapper_thinp/origin_16.00m -n snap_16.00m

Creating [32.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_32.00m --poolmetadatasize 32.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_32.00m -n origin_32.00m
lvcreate -s /dev/snapper_thinp/origin_32.00m -n snap_32.00m

Creating [64.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_64.00m --poolmetadatasize 64.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_64.00m -n origin_64.00m
lvcreate -s /dev/snapper_thinp/origin_64.00m -n snap_64.00m

Creating [128.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_128.00m --poolmetadatasize 128.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_128.00m -n origin_128.00m
lvcreate -s /dev/snapper_thinp/origin_128.00m -n snap_128.00m

Creating [256.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_256.00m --poolmetadatasize 256.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_256.00m -n origin_256.00m
lvcreate -s /dev/snapper_thinp/origin_256.00m -n snap_256.00m

Creating [512.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_512.00m --poolmetadatasize 512.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_512.00m -n origin_512.00m
lvcreate -s /dev/snapper_thinp/origin_512.00m -n snap_512.00m

Removing [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
Removing volume snapper_thinp/snap_4.00m
  device-mapper: message ioctl on  failed: Device or resource busy
  Unable to deactivate open snapper_thinp-POOL_4.00m_tdata (253:4)
  Unable to deactivate open snapper_thinp-POOL_4.00m_tmeta (253:3)
  Failed to deactivate snapper_thinp-POOL_4.00m-tpool
  Failed to resume POOL_4.00m.
  Failed to update thin pool POOL_4.00m.
couldn't remove thin origin volume



Dec 12 14:56:30 hayes-01 qarshd[3277]: Running cmdline: lvremove /dev/snapper_thinp/origin_4.00m
Dec 12 14:56:33 hayes-01 kernel: device-mapper: thin: Data device (dm-4) discard unsupported: Disabling discard passdown.
Dec 12 14:56:33 hayes-01 lvm[2654]: Monitoring thin snapper_thinp-POOL_4.00m-tpool.
Dec 12 14:56:33 hayes-01 kernel: device-mapper: thin: Data device (dm-9) discard unsupported: Disabling discard passdown.
Dec 12 14:56:33 hayes-01 lvm[2654]: Monitoring thin snapper_thinp-POOL_16.00m-tpool.
Dec 12 14:56:33 hayes-01 udevd-work[3284]: setfilecon /dev/dm-13 failed: No such file or directory
Dec 12 14:56:34 hayes-01 kernel: device-mapper: thin: Deletion of thin device 1 failed.
Dec 12 14:56:34 hayes-01 xinetd[1633]: EXIT: qarsh status=0 pid=3277 duration=4(sec)
Dec 12 14:56:34 hayes-01 kernel: device-mapper: thin: Data device (dm-15) discard unsupported: Disabling discard passdown.
Dec 12 14:56:34 hayes-01 kernel: device-mapper: table: 253:17: thin: Error opening pool device
Dec 12 14:56:34 hayes-01 kernel: device-mapper: ioctl: error adding target to table
Dec 12 14:56:34 hayes-01 kernel: device-mapper: thin: Data device (dm-15) discard unsupported: Disabling discard passdown.
Dec 12 14:56:34 hayes-01 lvm[2654]: Monitoring thin snapper_thinp-POOL_32.00m-tpool.
Dec 12 14:56:35 hayes-01 udevd-work[3282]: setfilecon /dev/dm-22 failed: No such file or directory
Dec 12 14:56:35 hayes-01 kernel: device-mapper: thin: Data device (dm-19) discard unsupported: Disabling discard passdown.
Dec 12 14:56:35 hayes-01 kernel: device-mapper: table: 253:22: thin: Error opening pool device
Dec 12 14:56:35 hayes-01 kernel: device-mapper: ioctl: error adding target to table
Dec 12 14:56:35 hayes-01 kernel: device-mapper: table: 253:23: thin-pool: metadata device already in use by a pool
Dec 12 14:56:35 hayes-01 kernel: device-mapper: ioctl: error adding target to table
Dec 12 14:56:37 hayes-01 kernel: device-mapper: thin: Data device (dm-19) discard unsupported: Disabling discard passdown.
Dec 12 14:56:37 hayes-01 lvm[2654]: Monitoring thin snapper_thinp-POOL_64.00m-tpool.
Dec 12 14:56:38 hayes-01 kernel: device-mapper: thin: Data device (dm-25) discard unsupported: Disabling discard passdown.
Dec 12 14:56:38 hayes-01 lvm[2654]: Monitoring thin snapper_thinp-POOL_128.00m-tpool.
Dec 12 14:56:40 hayes-01 kernel: device-mapper: thin: Data device (dm-31) discard unsupported: Disabling discard passdown.
Dec 12 14:56:40 hayes-01 kernel: device-mapper: table: 253:33: thin: Error opening pool device
Dec 12 14:56:40 hayes-01 kernel: device-mapper: ioctl: error adding target to table
Dec 12 14:56:43 hayes-01 kernel: device-mapper: thin: Data device (dm-31) discard unsupported: Disabling discard passdown.
Dec 12 14:56:43 hayes-01 lvm[2654]: Monitoring thin snapper_thinp-POOL_256.00m-tpool.
Dec 12 14:56:45 hayes-01 udevd-work[3284]: setfilecon /dev/dm-37 failed: No such file or directory
Dec 12 14:56:45 hayes-01 kernel: device-mapper: thin: Data device (dm-35) discard unsupported: Disabling discard passdown.
Dec 12 14:56:45 hayes-01 kernel: device-mapper: table: 253:37: thin: Error opening pool device
Dec 12 14:56:45 hayes-01 kernel: device-mapper: ioctl: error adding target to table
Dec 12 14:56:46 hayes-01 kernel: device-mapper: thin: Data device (dm-35) discard unsupported: Disabling discard passdown.
Dec 12 14:56:46 hayes-01 kernel: device-mapper: table: 253:39: thin: Error opening pool device
Dec 12 14:56:46 hayes-01 kernel: device-mapper: ioctl: error adding target to table



Version-Release number of selected component (if applicable):
2.6.32-343.el6.x86_64

lvm2-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
lvm2-libs-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
lvm2-cluster-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
device-mapper-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-libs-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-event-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
device-mapper-event-libs-1.02.77-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
cmirror-2.02.98-5.el6    BUILT: Wed Dec 12 09:18:07 CST 2012
Comment 3 Corey Marthaler 2012-12-13 00:23:07 UTC 
This is reproducible.

============================================================
Iteration 1 of 10 started at Wed Dec 12 18:18:17 CST 2012
============================================================
SCENARIO - [poolmetadatasizes]
Create pool volumes of varying pool metadata sizes (--poolmetadatasize)

Creating [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_4.00m --poolmetadatasize 4.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_4.00m -n origin_4.00m
lvcreate -s /dev/snapper_thinp/origin_4.00m -n snap_4.00m

Creating [16.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_16.00m --poolmetadatasize 16.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_16.00m -n origin_16.00m
lvcreate -s /dev/snapper_thinp/origin_16.00m -n snap_16.00m

Creating [32.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_32.00m --poolmetadatasize 32.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_32.00m -n origin_32.00m
lvcreate -s /dev/snapper_thinp/origin_32.00m -n snap_32.00m

Creating [64.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_64.00m --poolmetadatasize 64.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_64.00m -n origin_64.00m
lvcreate -s /dev/snapper_thinp/origin_64.00m -n snap_64.00m

Creating [128.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_128.00m --poolmetadatasize 128.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_128.00m -n origin_128.00m
lvcreate -s /dev/snapper_thinp/origin_128.00m -n snap_128.00m

Creating [256.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_256.00m --poolmetadatasize 256.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_256.00m -n origin_256.00m
lvcreate -s /dev/snapper_thinp/origin_256.00m -n snap_256.00m

Creating [512.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_512.00m --poolmetadatasize 512.00m -L 500M snapper_thinp
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_512.00m -n origin_512.00m
lvcreate -s /dev/snapper_thinp/origin_512.00m -n snap_512.00m

Removing [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
Removing volume snapper_thinp/snap_4.00m
Do you really want to remove active logical volume origin_4.00m? [y/n]: ^C




Dec 12 18:18:41 taft-01 qarshd[6387]: Running cmdline: lvremove /dev/snapper_thinp/origin_4.00m
device-mapper: thin: Data device (dm-4) discard unsupported: Disabling discard passdown.
Dec 12 18:18:41 taft-01 kernel: device-mapper: thin: Data device (dm-4) discard unsupported: Disabling discard passdown.
Dec 12 18:18:41 taft-01 lvm[3452]: Monitoring thin snapper_thinp-POOL_4.00m-tpool.
device-mapper: thin: Data device (dm-9) discard unsupported: Disabling discard passdown.
Dec 12 18:18:41 taft-01 kernel: device-mapper: thin: Data device (dm-9) discard unsupported: Disabling discard passdown.
Dec 12 18:18:41 taft-01 lvm[3452]: Monitoring thin snapper_thinp-POOL_16.00m-tpool.
device-mapper: thin: Data device (dm-15) discard unsupported: Disabling discard passdown.
Dec 12 18:18:41 taft-01 kernel: device-mapper: thin: Data device (dm-15) discard unsupported: Disabling discard passdown.
Dec 12 18:18:41 taft-01 lvm[3452]: Monitoring thin snapper_thinp-POOL_32.00m-tpool.
device-mapper: thin: Data device (dm-21) discard unsupported: Disabling discard passdown.
Dec 12 18:18:42 taft-01 kernel: device-mapper: thin: Data device (dm-21) discard unsupported: Disabling discard passdown.
Dec 12 18:18:42 taft-01 lvm[3452]: Monitoring thin snapper_thinp-POOL_64.00m-tpool.
device-mapper: thin: Data device (dm-27) discard unsupported: Disabling discard passdown.
Dec 12 18:18:42 taft-01 kernel: device-mapper: tdevice-mapper: table: 253:29: linear: dm-linear: Device lookup failed
hin: Data devicedevice-mapper: ioctl: error adding target to table
 (dm-27) discard unsupported: Disabling discard passdown.
Dec 12 18:18:42 taft-01 kernel: device-mapper: table: 253:29: linear: dm-linear: Device lookup failed
Dec 12 18:18:42 taft-01 kernel: device-mapper: ioctl: error adding target to table
device-mapper: thin: Data device (dm-27) discard unsupported: Disabling discard passdown.
Dec 12 18:18:42 device-mapper: table: 253:30: thin: Error opening pool device
taft-01 kernel: device-mapper: ioctl: error adding target to table
device-mapper: thin: Data device (dm-27) discard unsupported: Disabling discard passdown.
Dec 12 18:18:42 taft-01 kernel: device-mapper: table: 253:30: thin: Error opening pool device
Comment 4 Zdenek Kabelac 2012-12-13 13:07:15 UTC 
> lvcreate -s /dev/snapper_thinp/origin_512.00m -n snap_512.00m
> 
> Removing [4.00m] MDA thinpool and corresponding thin virtual volume and thin
> snapshot
> Removing volume snapper_thinp/snap_4.00m
>   device-mapper: message ioctl on  failed: Device or resource busy
>   Unable to deactivate open snapper_thinp-POOL_4.00m_tdata (253:4)
>   Unable to deactivate open snapper_thinp-POOL_4.00m_tmeta (253:3)
>   Failed to deactivate snapper_thinp-POOL_4.00m-tpool
>   Failed to resume POOL_4.00m.
>   Failed to update thin pool POOL_4.00m.
> couldn't remove thin origin volume
> 
> 
> 
> Dec 12 14:56:30 hayes-01 qarshd[3277]: Running cmdline: lvremove
> /dev/snapper_thinp/origin_4.00m
> Dec 12 14:56:33 hayes-01 kernel: device-mapper: thin: Data device (dm-4)
> discard unsupported: Disabling discard passdown.
> Dec 12 14:56:33 hayes-01 lvm[2654]: Monitoring thin


Hmm I'm being a bit confused on the order of commands here,
since I'm not able to get close to your machine's behavior.

How did the remove of  snapper_thinp/snap_4.00m  triggered deactivation
of  snapper_thinp-POOL_4.00m_tdata ??

There is some 'chance'  your  snap_4.00m is being kept open by another process
(possibly overloaded udev) for a very very long time (>10s) - thus avoiding even retry operation for removal of LV.

But it still should not follow with any further operation.


We need to see 

dmsetup table
dmsetup ls --tree -o ascii
dmsetup status

before failing command
and possibly right after failure.

> Dec 12 14:56:33 hayes-01 udevd-work[3284]: setfilecon /dev/dm-13 failed: No
> such file or directory

Interesting - why udevd runs something over removed device ???


> Dec 12 14:56:34 hayes-01 kernel: device-mapper: table: 253:17: thin: Error
> opening pool device

Scary

Why we would active thin pool with missing device?
Something weird must be going on here.

Is this with lvmetad  or without?
Comment 5 Corey Marthaler 2012-12-14 00:13:00 UTC 
It's with lvmetad, see the subject. Check out bug 886695 for more information on test cases that failed during removal. I think these two bugs are related.
Comment 6 Marian Csontos 2013-01-03 08:48:25 UTC 
(In reply to comment #3)

Corey, I am pretty sure this one was caused by Bug 889454 - looking at the test case, VG is deactivated, LV *snap* is removed, triggering udev action reactivating other LVs in the VG causing lvremove *origin* to fail:

> Removing [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot

Here snap_4.00m is successfully removed:

> Removing volume snapper_thinp/snap_4.00m

And here removing origin_4.00m fails as it's active:

> Do you really want to remove active logical volume origin_4.00m? [y/n]: ^C


As for the original problem:

I have run lvm_single_thin_snapper (with lvmetad and with 889454 fixed) few dozen times and have not seen the issue - so either it's gone or the reproducibility is fairly low.
Comment 7 Peter Rajnoha 2013-01-09 10:12:21 UTC 
Corey, the latest build lvm2-2.02.98-7.el6 (which also resolves bug #889454) has the fix for autoactivation that intefered in this case too. Please, try it if you this is still reproducible. If not, we can close it as a dup of bug #889454.
Comment 8 Corey Marthaler 2013-01-09 23:32:08 UTC 
This test case is no longer causing any issues with the latest rpms.

2.6.32-353.el6.x86_64

lvm2-2.02.98-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
lvm2-libs-2.02.98-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
lvm2-cluster-2.02.98-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
udev-147-2.43.el6    BUILT: Thu Oct 11 05:59:38 CDT 2012
device-mapper-1.02.77-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
device-mapper-libs-1.02.77-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
device-mapper-event-1.02.77-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
device-mapper-event-libs-1.02.77-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
cmirror-2.02.98-7.el6    BUILT: Wed Jan  9 03:34:27 CST 2013
Comment 9 Peter Rajnoha 2013-01-10 09:37:21 UTC 
Just a note: the only change in latest rpms (not taking into account one lvm1 vs. lvmetad fix) are the fixes for autoactivation to not fire when it should not and addition of proper udev synchronization with "pvscan --cache -aay" that is called from within udev rules to autoactivate volumes.
Comment 10 Peter Rajnoha 2013-01-15 15:07:38 UTC 
Closing this bug as it can't be reproduced anymore with latest rpms which
include fixes for autoactivation (bug #889454).

If still seen, please, reopen this bug report.

*** This bug has been marked as a duplicate of bug 889454 ***
Note You need to log in before you can comment on or make changes to this bug.