Bug 864203

Summary:

thinp volume removal can be painfully slow at times

Product:

Red Hat Enterprise Linux 6

Reporter:

Corey Marthaler <cmarthal>

Component:

lvm2

Assignee:

Zdenek Kabelac <zkabelac>

Status:

CLOSED WORKSFORME

QA Contact:

Cluster QE <mspqa-list>

Severity:

medium

Docs Contact:

Priority:

medium

Version:

6.3

CC:

agk, ddumas, dwysocha, heinzm, jbrassow, mcsontos, msnitzer, prajnoha, prockai, thornber, zkabelac

Target Milestone:

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2012-12-14 23:29:29 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
strace -ttttt lvremove -f snapper_thinp/snap_4.00m > /tmp/thin_snap_remove 2>&1	none
strace -ttttt lvremove -f snapper_thinp/origin_4.00m > /tmp/thin_origin_remove 2>&1	none
strace -ttttt lvremove -f snapper_thinp/POOL_4.00m > /tmp/thin_pool_remove 2>&1	none

Description Corey Marthaler 2012-10-08 20:44:38 UTC

Description of problem:

./snapper_thinp -o taft-04 -l /home/msp/cmarthal/work/sts/sts-root -r /usr/tests/sts-rhel6.3 -e poolmetadatasizes -i 1
[...]
SCENARIO - [poolmetadatasizes]
Create pool volumes of varying pool metadata sizes (--poolmetadatasize)

Creating [4.00m] MDA thinpool and corresponding thin virtual volume and thin snapshot
lvcreate --thinpool POOL_4.00m --poolmetadatasize 4.00m -L 500M snapper_thinp
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_4.00m -n origin_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
lvcreate -s /dev/snapper_thinp/origin_4.00m -n snap_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).

## Should the removal of these volumes each take half a minute?

[root@taft-04 ~]# lvs -a -o +devices
  LV                  Attr     LSize   Pool       Origin      Data% Devices
  POOL_4.00m          twi-a-tz 500.00m                         0.00 POOL_4.00m_tdata(0)
  [POOL_4.00m_tdata]  Twi-aot- 500.00m                              /dev/sdb1(0)
  [POOL_4.00m_tmeta]  ewi-aot-   4.00m                              /dev/sdg1(0)
  origin_4.00m        Vwi-a-tz 500.00m POOL_4.00m              0.00
  snap_4.00m          Vwi-a-tz 500.00m POOL_4.00m origin_4.00m 0.00

[root@taft-04 ~]# time lvremove -f snapper_thinp/snap_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
  Logical volume "snap_4.00m" successfully removed

real  0m27.979s
user  0m27.630s
sys   0m0.227s

[root@taft-04 ~]# lvs -a -o +devices
  LV                  Attr     LSize   Pool       Origin Data% Devices
  POOL_4.00m          twi-a-tz 500.00m                    0.00 POOL_4.00m_tdata(0)
  [POOL_4.00m_tdata]  Twi-aot- 500.00m                         /dev/sdb1(0)
  [POOL_4.00m_tmeta]  ewi-aot-   4.00m                         /dev/sdg1(0)
  origin_4.00m        Vwi-a-tz 500.00m POOL_4.00m         0.00

[root@taft-04 ~]# time lvremove -f snapper_thinp/origin_4.00m
  WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
  Logical volume "origin_4.00m" successfully removed

real  0m28.099s
user  0m27.730s
sys   0m0.256s

[root@taft-04 ~]# lvs -a -o +devices
  LV                  Attr     LSize   Pool Origin Data% Devices
  POOL_4.00m          twi-a-tz 500.00m              0.00 POOL_4.00m_tdata(0)
  [POOL_4.00m_tdata]  Twi-aot- 500.00m                   /dev/sdb1(0)
  [POOL_4.00m_tmeta]  ewi-aot-   4.00m                   /dev/sdg1(0)
[root@taft-04 ~]# time lvremove -f snapper_thinp/POOL_4.00m
  Logical volume "POOL_4.00m" successfully removed

real  0m31.154s
user  0m30.748s
sys   0m0.250s


Version-Release number of selected component (if applicable):
2.6.32-279.el6.x86_64

lvm2-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
lvm2-libs-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
lvm2-cluster-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
udev-147-2.41.el6    BUILT: Thu Mar  1 13:01:08 CST 2012
device-mapper-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
device-mapper-libs-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
device-mapper-event-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
device-mapper-event-libs-1.02.76-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012
cmirror-2.02.97-3.el6    BUILT: Tue Sep 11 05:06:56 CDT 2012

Comment 1 Mike Snitzer 2012-10-09 02:16:07 UTC

(In reply to comment #0)
> Description of problem:
> 
> ./snapper_thinp -o taft-04 -l /home/msp/cmarthal/work/sts/sts-root -r
> /usr/tests/sts-rhel6.3 -e poolmetadatasizes -i 1
> [...]
> SCENARIO - [poolmetadatasizes]
> Create pool volumes of varying pool metadata sizes (--poolmetadatasize)
> 
> Creating [4.00m] MDA thinpool and corresponding thin virtual volume and thin
> snapshot
> lvcreate --thinpool POOL_4.00m --poolmetadatasize 4.00m -L 500M snapper_thinp
>   WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
> lvcreate --virtualsize 500M --thinpool snapper_thinp/POOL_4.00m -n
> origin_4.00m
>   WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
> lvcreate -s /dev/snapper_thinp/origin_4.00m -n snap_4.00m
>   WARNING: Thin pool target does not support discards (needs kernel >= 3.4).
> 
> ## Should the removal of these volumes each take half a minute?

I'll defer to others on why the removal is taking so long.

But the "WARNING: Thin pool target does not support discards (needs kernel >= 3.4)." needs fixing for RHEL6.  lvm2 needs to be updated to account for the fact that discards are now properly supported in RHEL6.4 for sure.

Comment 2 Zdenek Kabelac 2012-10-09 13:06:04 UTC

I believe most problems with --discards option have been already resolved for the next release 2.02.98 (and noted also in Bug 864380) - I've never seen such 'slowdown' factor on my system.

Removal of volumes should be reaching the speed of removal of 'normal' LVs.

Could you please attach  'strace -tttt' of such lengthy operation so we could more easily see what is blocking the command progress - I expect some weird interaction with device access/mda update.

Comment 3 Corey Marthaler 2012-10-09 21:06:52 UTC

Created attachment 624265 [details]
strace -ttttt lvremove -f snapper_thinp/snap_4.00m > /tmp/thin_snap_remove 2>&1

strace -ttttt lvremove -f snapper_thinp/snap_4.00m > /tmp/thin_snap_remove 2>&1

Comment 4 Corey Marthaler 2012-10-09 21:07:36 UTC

Created attachment 624266 [details]
strace -ttttt lvremove -f snapper_thinp/origin_4.00m > /tmp/thin_origin_remove 2>&1

Comment 5 Corey Marthaler 2012-10-09 21:08:07 UTC

Created attachment 624267 [details]
strace -ttttt lvremove -f snapper_thinp/POOL_4.00m > /tmp/thin_pool_remove 2>&1

Comment 6 Zdenek Kabelac 2012-10-10 08:29:37 UTC

Hmm - from attached traces it seems like machine spends that 'extra' 30s time in system call 'brk'.

Full -vvvv trace of lvm command would be helpful to see what is the memory size being locked by the command in memory.

Is that machine used for testing somehow memory limited and uses a lot of swap on external storage ?

Only thin manipulation commands are taking this amount of time ?

Comment 7 Marian Csontos 2012-11-09 10:38:32 UTC

This reminds me of another already solved lvmetad issue (no BZ, commit b07df88) which should be fixed in lvm2-2.02.98 (all 6.4 builds) caused by string concatenation used when building the metadata.

Comment 10 Corey Marthaler 2012-12-14 23:29:29 UTC

Closing this bug WORKSFORME. The lvmetad bugs can be dealt with separately. If the slow down comes back we can revisit this bug, but for now it works fine (with lvmetad turned off)