Bug 1276413 - Thin target report 'data' threshold reached while it should have been 'metadata'
Thin target report 'data' threshold reached while it should have been 'metadata'
Status: NEW
Product: Fedora
Classification: Fedora
Component: kernel (Show other bugs)
rawhide
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: Joe Thornber
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-29 11:55 EDT by Zdenek Kabelac
Modified: 2015-10-29 12:11 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Zdenek Kabelac 2015-10-29 11:55:22 EDT
Description of problem:

lvm2 2.02.133 has started to use low_water_mark and not its test suite is hitting 'interesting' kernel reported message.

Test:  shell/lvextend-thin-metadata-dmeventd.sh

Basically pre-creates this layout:

-- table:
vg-pool: 0 409600 linear 253:5 0
vg-pool-tpool: 0 409600 thin-pool 253:3 253:4 128 286720 0
vg-pool_tdata: 0 409600 linear 253:0 6144
vg-pool_tmeta: 0 4096 linear 253:2 2048
vg-LV2: 0 1024000 thin 253:5 3

-- status:
vg-pool: 0 409600 linear
vg-pool-tpool: 0 409600 thin-pool 3 358/512 29/3200 - rw discard_passdown queue_if_no_space -
vg-pool_tdata: 0 409600 linear
vg-pool_tmeta: 0 4096 linear
vg-LV2: 0 1024000 thin 0 -

-- lvs:
LV              VG   tr       LSize   Pool Origin Data%  Meta%
LV1             vg -wi-a-----   3.00m
LV2             vg Vwi-a-tz-k 500.00m pool thin   0.01
[lvol0_pmspare] vg ewi-------   2.00m
pool            vg twi-aotz-- 200.00m             0.94   46.74
[pool_tdata]    vg Twi-ao---- 200.00m
[pool_tmeta]    vg ewi-ao----   3.00m
thin            vg Vwi---tz-- 500.00m pool
thin2           vg Vwi---tz--   2.00m pool
--

now the test writes to LV thin2:
echo 2 >"/dev/mapper/vg-LV2"

invoking provisioning of metadata to go over 70%
(which should be trapped by dmeventd)

As a kind of 'surprise'  dmeventd is immediately awaken with this message:

device-mapper: thin: 253:5: reached low water mark for data device: sending event.

Obviously there could not have been 'data' reaching over threshold
(which is calculated to 70% for give data size)

As it seem kernel has 'hardcoded' value 75% for metadata reaching over threshold - creating this message afterwards:

device-mapper: thin: 253:5: reached low water mark for metadata device: sending event.

the question is from where do we get 'data' low water mark message in this case?

Version-Release number of selected component (if applicable):
4.3-rc6 kernel 
lvm2 git 2.02.133

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Zdenek Kabelac 2015-10-29 12:00:07 EDT
State before:


vg-LV1: 0 6144 linear 253:0 415744
pv3: 0 524288 linear 7:4 1050624
pv2: 0 524288 linear 7:4 526336
pv1: 0 524288 linear 7:4 2048
vg-pool-tpool: 0 409600 thin-pool 253:3 253:4 128 286720 0
vg-pool_tdata: 0 409600 linear 253:0 6144
vg-pool_tmeta: 0 4096 linear 253:2 2048
vg-pool: 0 409600 linear 253:5 0

vg-LV1: 0 6144 linear
pv3: 0 524288 linear
pv2: 0 524288 linear
pv1: 0 524288 linear
vg-pool-tpool: 0 409600 thin-pool 2 358/512 29/3200 - rw discard_passdown queue_if_no_space -
vg-pool_tdata: 0 409600 linear
vg-pool_tmeta: 0 4096 linear
vg-pool: 0 409600 linear


LV              VG  Attr       LSize   Pool Data%  Meta%
LV1             vg -wi-a-----   3.00m
[lvol0_pmspare] vg ewi-------   2.00m
pool            vg twi-aotz-- 200.00m       0.91   69.92
[pool_tdata]    vg Twi-ao---- 200.00m
[pool_tmeta]    vg ewi-ao----   2.00m
thin            vg Vwi---tz-- 500.00m pool
thin2           vg Vwi---tz--   2.00m pool
Comment 2 Mike Snitzer 2015-10-29 12:11:16 EDT
(In reply to Zdenek Kabelac from comment #0)
> Description of problem:
> 
> lvm2 2.02.133 has started to use low_water_mark and not its test suite is
> hitting 'interesting' kernel reported message.
> 
> Test:  shell/lvextend-thin-metadata-dmeventd.sh
> 
> Basically pre-creates this layout:
> 
> -- table:
> vg-pool: 0 409600 linear 253:5 0
> vg-pool-tpool: 0 409600 thin-pool 253:3 253:4 128 286720 0
> vg-pool_tdata: 0 409600 linear 253:0 6144
> vg-pool_tmeta: 0 4096 linear 253:2 2048
> vg-LV2: 0 1024000 thin 253:5 3
> 
> -- status:
> vg-pool: 0 409600 linear
> vg-pool-tpool: 0 409600 thin-pool 3 358/512 29/3200 - rw discard_passdown
> queue_if_no_space -
> vg-pool_tdata: 0 409600 linear
> vg-pool_tmeta: 0 4096 linear
> vg-LV2: 0 1024000 thin 0 -
> 
> -- lvs:
> LV              VG   tr       LSize   Pool Origin Data%  Meta%
> LV1             vg -wi-a-----   3.00m
> LV2             vg Vwi-a-tz-k 500.00m pool thin   0.01
> [lvol0_pmspare] vg ewi-------   2.00m
> pool            vg twi-aotz-- 200.00m             0.94   46.74
> [pool_tdata]    vg Twi-ao---- 200.00m
> [pool_tmeta]    vg ewi-ao----   3.00m
> thin            vg Vwi---tz-- 500.00m pool
> thin2           vg Vwi---tz--   2.00m pool
> --
> 
> now the test writes to LV thin2:
> echo 2 >"/dev/mapper/vg-LV2"
> 
> invoking provisioning of metadata to go over 70%
> (which should be trapped by dmeventd)

Going over 70% metadata shouldn't cause any low watermark event.  The default is min(4MB of metadata blocks, 75%)

Below you're clearly triggering metadata low watermark once you hit 75%.

> As a kind of 'surprise'  dmeventd is immediately awaken with this message:
> 
> device-mapper: thin: 253:5: reached low water mark for data device: sending
> event.
> 
> Obviously there could not have been 'data' reaching over threshold
> (which is calculated to 70% for give data size)
> 
> As it seem kernel has 'hardcoded' value 75% for metadata reaching over
> threshold - creating this message afterwards:
> 
> device-mapper: thin: 253:5: reached low water mark for metadata device:
> sending event.
> 
> the question is from where do we get 'data' low water mark message in this
> case?

Good question.  As you know: I had a look at the code and I'm not seeing why.

dm-thin.c:alloc_data_block calls check_low_water_mark.

At 70% metadata you haven't even triggered the metadata_low_callback.

Only thing I can think is somehow dm_pool_get_free_block_count() is being influenced by metadata getting low.. but that is just a conspiracy theory.. cannot see anywhere in code that would cause this.

Assigning to Joe since he might have more immediate understanding as to why this data low water mark is tripping...

Note You need to log in before you can comment on or make changes to this bug.