Bug 864203 - thinp volume removal can be painfully slow at times
thinp volume removal can be painfully slow at times
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2 (Show other bugs)
6.3
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Zdenek Kabelac
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-08 16:44 EDT by Corey Marthaler
Modified: 2012-12-14 18:29 EST (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-14 18:29:29 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
strace -ttttt lvremove -f snapper_thinp/snap_4.00m > /tmp/thin_snap_remove 2>&1 (450.59 KB, text/plain)
2012-10-09 17:06 EDT, Corey Marthaler
no flags Details
strace -ttttt lvremove -f snapper_thinp/origin_4.00m > /tmp/thin_origin_remove 2>&1 (445.26 KB, text/plain)
2012-10-09 17:07 EDT, Corey Marthaler
no flags Details
strace -ttttt lvremove -f snapper_thinp/POOL_4.00m > /tmp/thin_pool_remove 2>&1 (431.24 KB, text/plain)
2012-10-09 17:08 EDT, Corey Marthaler
no flags Details

  None (edit)
Description Corey Marthaler 2012-10-08 16:44:38 EDT
Description of problem:

./snapper_thinp -o taft-04 -l /home/msp/cmarthal/work/sts/sts-root -r /usr/tests/sts-rhel6.3 -e poolmetadatasizes -i 1
[...]
SCENARIO - [poolmetadatasizes]
Create pool volumes of varying pool metadata sizes (--poolmetadatasize)

Creating [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_4.00m --poolmetadatasize 4.00m -L 500M snapper_thinp
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_4.00m -n origin_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
lvcreate -s /dev/snapper_thinp/origin_4.00m -n snap_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).

## Should the removal of these volumes each take half a minute?

[root@taft-04 ~]# lvs -a -o +devices
  LV                  Attr     LSize   Pool       Origin      Data% Devices
  POOL_4.00m          twi-a-tz 500.00m                         0.00 POOL_4.00m_tdata(0)
  [POOL_4.00m_tdata]  Twi-aot- 500.00m                              /dev/sdb1(0)
  [POOL_4.00m_tmeta]  ewi-aot-   4.00m                              /dev/sdg1(0)
  origin_4.00m        Vwi-a-tz 500.00m POOL_4.00m              0.00
  snap_4.00m          Vwi-a-tz 500.00m POOL_4.00m origin_4.00m 0.00

[root@taft-04 ~]# time lvremove -f snapper_thinp/snap_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
  Logical volume "snap_4.00m" successfully removed

real  0m27.979s
user  0m27.630s
sys   0m0.227s

[root@taft-04 ~]# lvs -a -o +devices
  LV                  Attr     LSize   Pool       Origin Data% Devices
  POOL_4.00m          twi-a-tz 500.00m                    0.00 POOL_4.00m_tdata(0)
  [POOL_4.00m_tdata]  Twi-aot- 500.00m                         /dev/sdb1(0)
  [POOL_4.00m_tmeta]  ewi-aot-   4.00m                         /dev/sdg1(0)
  origin_4.00m        Vwi-a-tz 500.00m POOL_4.00m         0.00

[root@taft-04 ~]# time lvremove -f snapper_thinp/origin_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
  Logical volume "origin_4.00m" successfully removed

real  0m28.099s
user  0m27.730s
sys   0m0.256s

[root@taft-04 ~]# lvs -a -o +devices
  LV                  Attr     LSize   Pool Origin Data% Devices
  POOL_4.00m          twi-a-tz 500.00m              0.00 POOL_4.00m_tdata(0)
  [POOL_4.00m_tdata]  Twi-aot- 500.00m                   /dev/sdb1(0)
  [POOL_4.00m_tmeta]  ewi-aot-   4.00m                   /dev/sdg1(0)
[root@taft-04 ~]# time lvremove -f snapper_thinp/POOL_4.00m
  Logical volume "POOL_4.00m" successfully removed

real  0m31.154s
user  0m30.748s
sys   0m0.250s


Version-Release number of selected component (if applicable):
2.6.32-279.el6.x86_64

lvm2-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
lvm2-libs-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
lvm2-cluster-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
udev-147-2.41.el6    BUILT: Thu Mar  1 13:01:08 CST 2012
device-mapper-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
device-mapper-libs-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
device-mapper-event-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
device-mapper-event-libs-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
cmirror-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
Comment 1 Mike Snitzer 2012-10-08 22:16:07 EDT
(In reply to comment #0)
> Description of problem:
> 
> ./snapper_thinp -o taft-04 -l /home/msp/cmarthal/work/sts/sts-root -r
> /usr/tests/sts-rhel6.3 -e poolmetadatasizes -i 1
> [...]
> SCENARIO - [poolmetadatasizes]
> Create pool volumes of varying pool metadata sizes (--poolmetadatasize)
> 
> Creating [4.00m] MDA thinpool and corresponding thin virtual volume and thin
> snapshot
> lvcreate --thinpool POOL_4.00m --poolmetadatasize 4.00m -L 500M snapper_thinp
>   WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
> lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_4.00m -n
> origin_4.00m
>   WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
> lvcreate -s /dev/snapper_thinp/origin_4.00m -n snap_4.00m
>   WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
> 
> ## Should the removal of these volumes each take half a minute?

I'll defer to others on why the removal is taking so long.

But the "WARNING: Thin pool target does not support discards (needs kernel >= 3.4)." needs fixing for RHEL6.  lvm2 needs to be updated to account for the fact that discards are now properly supported in RHEL6.4 for sure.
Comment 2 Zdenek Kabelac 2012-10-09 09:06:04 EDT
I believe most problems with --discards option have been already resolved for the next release 2.02.98 (and noted also in Bug 864380) - I've never seen such 'slowdown' factor on my system.

Removal of volumes should be reaching the speed of removal of 'normal' LVs.

Could you please attach  'strace -tttt' of such lengthy operation so we could more easily see what is blocking the command progress - I expect some weird interaction with device access/mda update.
Comment 3 Corey Marthaler 2012-10-09 17:06:52 EDT
Created attachment 624265 [details]
strace -ttttt lvremove -f snapper_thinp/snap_4.00m > /tmp/thin_snap_remove 2>&1

strace -ttttt lvremove -f snapper_thinp/snap_4.00m > /tmp/thin_snap_remove 2>&1
Comment 4 Corey Marthaler 2012-10-09 17:07:36 EDT
Created attachment 624266 [details]
strace -ttttt lvremove -f snapper_thinp/origin_4.00m > /tmp/thin_origin_remove 2>&1
Comment 5 Corey Marthaler 2012-10-09 17:08:07 EDT
Created attachment 624267 [details]
strace -ttttt lvremove -f snapper_thinp/POOL_4.00m > /tmp/thin_pool_remove 2>&1
Comment 6 Zdenek Kabelac 2012-10-10 04:29:37 EDT
Hmm - from attached traces it seems like machine spends that 'extra' 30s time in system call 'brk'.

Full -vvvv trace of lvm command would be helpful to see what is the memory size being locked by the command in memory.

Is that machine used for testing somehow memory limited and uses a lot of swap on external storage ?

Only thin manipulation commands are taking this amount of time ?
Comment 7 Marian Csontos 2012-11-09 05:38:32 EST
This reminds me of another already solved lvmetad issue (no BZ, commit b07df88) which should be fixed in lvm2-2.02.98 (all 6.4 builds) caused by string concatenation used when building the metadata.
Comment 10 Corey Marthaler 2012-12-14 18:29:29 EST
Closing this bug WORKSFORME. The lvmetad bugs can be dealt with separately. If the slow down comes back we can revisit this bug, but for now it works fine (with lvmetad turned off)

Note You need to log in before you can comment on or make changes to this bug.