Created attachment 788388 [details]
testlog.txt

please refer to testlog.txt for details.

Hmm I've reproduced this with upstream kernel 3.10 as well.

Interestingly sometimes helps to deactivate & activate device and run fstrim again after mounting. I'd suspect kernel issue here.

Created attachment 788500 [details]
blkparse  of  dm-3

Traces from the moment discard/trim  does not work even though it should.
(FS on device has been just mounted  and  executed fstrim reported this:

# fstrim -v /mnt
/mnt: 91.3 MiB (95748096 bytes) trimmed




# dmsetup info -c
Name             Maj Min Stat Open Targ Event  UUID                                                                      
tvg-lv1          253   3 L--w    1    1      0 LVM-jLl3GpQlQ9bkcw199FQftRK5Y8ZALARdQfGaAfRb4Mbr8tPh2E92zYYFakgy6wCH      
tvg-pool-tpool   253   2 L--w    1    1      0 LVM-jLl3GpQlQ9bkcw199FQftRK5Y8ZALARderPCxP9HM7FNLn1uyqH4ZLAGbIMXzNDJ-tpool
tvg-pool_tdata   253   1 L--w    1    1      0 LVM-jLl3GpQlQ9bkcw199FQftRK5Y8ZALARd6j3CKjTUj3avruy5xN1oY0mg6m2CbYyE      
tvg-pool_tmeta   253   0 L--w    1    1      0 LVM-jLl3GpQlQ9bkcw199FQftRK5Y8ZALARduuOL1keIbQLYHGsKgaviC1w0dAj6h71S      


# dmsetup table
tvg-lv1: 0 204800 thin 253:2 1
tvg-pool-tpool: 0 106496 thin-pool 253:0 253:1 128 0 0 
tvg-pool_tdata: 0 106496 linear 8:16 10240
tvg-pool_tmeta: 0 8192 linear 8:16 116736

# dmsetup status
tvg-lv1: 0 204800 thin 57216 147583
tvg-pool-tpool: 0 106496 thin-pool 1 13/1024 447/832 - rw discard_passdown
tvg-pool_tdata: 0 106496 linear 
tvg-pool_tmeta: 0 8192 linear

Created attachment 788502 [details]
Traces for underlaying virtual iscsi debug device

Trace taken at the same time for scsi /dev/sdb device which is used as a PV for my tests  (and keeps data & metadata device)

lvs for devices after fstrim:

Non working case:

  LV    VG   Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  lv1   tvg  Vwi-aotz-- 100,00m pool         27,94                          
  pool  tvg  twi---tz--  52,00m              53,73                          


Working case:

  lv1   tvg  Vwi-aotz-- 100,00m pool          8,00
  pool  tvg  twi---tz--  52,00m              15,38

discards are flowing through the thin device (trace from comment#5), to the thin-pool, and down to the underlying device (trace from comment#6).

So if no free space is accumulating in the thin device (and backing thin-pool) then this may be a bio-prison issue, or other accounting bug.  Could be the entire block isn't considered discarded so no blocks get released.

I'm wondering if this is an alignment issue, e.g.: the fs _always_ has something in a thinp block.  As such thinp cannot discard the block.

The thin-pool blocksize is 64K.  Which FS is being used?

Does behavior change if you switch from using ext4 to xfs (or vice versa)?

(In reply to Mike Snitzer from comment #9)
> I'm wondering if this is an alignment issue, e.g.: the fs _always_ has
> something in a thinp block.  As such thinp cannot discard the block.
> 
> The thin-pool blocksize is 64K.  Which FS is being used?
> 
> Does behavior change if you switch from using ext4 to xfs (or vice versa)?

ext4 was used and the same behavior when using xfs.

(In reply to Xiaowei Li from comment #10)
> (In reply to Mike Snitzer from comment #9)
> > I'm wondering if this is an alignment issue, e.g.: the fs _always_ has
> > something in a thinp block.  As such thinp cannot discard the block.
> > 
> > The thin-pool blocksize is 64K.  Which FS is being used?
> > 
> > Does behavior change if you switch from using ext4 to xfs (or vice versa)?
> 
> ext4 was used and the same behavior when using xfs.

OK, but my point stands: ext4 and xfs could still be using portions of the block for filesystem metadata.  Thinp will only pass down discards to the underlying storage if the entire thinp block is no longer used at all.

There could be a thinp bug is lurking here.

But it'd be interesting to see if more care was taken to inform the filesystem about the underlying device's geomtry; does the filesystem allow for cleaner seperation of filesystem data and metadata areas?

(Cc'ing Eric, Dave and Carlos.)

ext4 might be a little "better" about constraining some metadata to certain areas, just as a function of its fixed inode table space, vs. xfs's dynamically allocated inodes.  Things would have to be fairly carefully sized accordingly; I have to remind myself how stripe units affect block group sizes.


There are also tools to look at all actual free space in the fs.

For ext4, 'dumpe2fs $DEVICE' will show (pretty verbosely) every free block.

for xfs, 'xfs_db -c "freesp -d" $DEVICE' will show free ranges as well.

-Eric

I really don't think there is a bug in the dm-thin kernel code (or in lvm2).  I think the FS is still using the thinp blocks, so thinp cannot discard them.

Please collect:
1) a thin_dump before issuing the fstrim.
2) a thin_dump after issuing the fstrim.

would also be useful to collect a blktrace of:
1) the thinp device to verify that the discards are being processed properly
2) the thin-pool's data volume to collect/see discards get passed down

comment#14 (fstrim doesn't work with XFS) vs comment#17 (fstrim works with btrfs) is really a question for the XFS developers.

Could it be that XFS needs an online discard flag set via mount, even though fstrim is being used for issuing async trims, whereas btrfs doesn't?

esandeen can you help and/or enlist the help of either Dave or Carlos?

no, xfs shouldn't need -o discard for fstrim to function.  Not sure what's going on but we'll look.

This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.