Description of problem:

Unable to remove LVM caching device in writeback mode

It started off saying
  0 blocks must still be flushed.
This slowly increased until it now has been saying:
  8 blocks must still be flushed.
  8 blocks must still be flushed.
Left for 12 hours and now it says:
  143745 blocks must still be flushed.
  143745 blocks must still be flushed.
  143745 blocks must still be flushed.
  143745 blocks must still be flushed.
  143746 blocks must still be flushed.
  143746 blocks must still be flushed.

Version-Release number of selected component (if applicable):

kernel-3.10.0-229.7.2.el7.jump2.x86_64
lvm2-2.02.115-3.el7.x86_64

How reproducible:
Notes from the case of SEG reproduction

  LV                          VG   Attr       LSize   Pool        Origin                      Data%  Meta%  Move Log Cpy%Sync Convert LV Tags Devices
  [ssd_cache]                 vg02 Cwi---C---   2.91t                                         13.96  4.44            69.02                    ssd_cache_cdata(0)                 
  [ssd_cache_cdata]           vg02 Cwi-ao----   2.91t                                                                                         /dev/sdb(100)                      
  [ssd_cache_cmeta]           vg02 ewi-ao---- 400.00m                                                                                         /dev/sdb(0)                        
  sybase_user_data_lv         vg02 Cwi-aoC---  20.00t [ssd_cache] [sybase_user_data_lv_corig] 13.96  4.44            69.02                    sybase_user_data_lv_corig(0)       
  [sybase_user_data_lv_corig] vg02 owi-aoC---  20.00t                                                                                         /dev/sdd(0),/dev/sde(0),/dev/sdg(0)
                                       ^^^

It seems the attempt was to remove the LV while it was still mounted as noted from the Attr above, most likely this is what was resulting in the cache incoherence.  When the fs is unmounted fsync() is called prior.  Secondly, we see the thresholds being changed at the LV layer: 

lvchange --cachesettings 'write_promote_adjustment=0 discard_promote_adjustment=0 \
        migration_threshold=2048000 random_threshold=512 sequential_threshold=1000000 \
        read_promote_adjustment=0' vg02/sybase_user_data_lv

However the table was not updated (again judging from sosreport) ie: 

vg02-sybase_user_data_lv: 0 42949681152 cache 253:3 253:2 253:4 4096 1 writeback cleaner 0

With the above, I again attempted to reproduce the issue leaving the fs mounted.  Of the many attempts, I managed to reproduce once: 

# lvcreate -L 1G -n ssd_meta testvg /dev/sdc1
  Logical volume "ssd_meta" created.

# lvcreate -L 10G -n ssd_cache testvg /dev/sdc1
  Logical volume "ssd_cache" created.

# lvconvert -c 2M --yes --type cache-pool --cachemode writeback --poolmetadata testvg/ssd_meta testvg/ssd_cache
  WARNING: Converting logical volume testvg/ssd_cache and testvg/ssd_meta to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
  Converted testvg/ssd_cache to cache pool.

# lvcreate --type cache --cachepool testvg/ssd_cache -L 5G -n cached_vol testvg

- As noted in #21: 

# lvchange --cachesettings 'write_promote_adjustment=0 discard_promote_adjustment=0 migration_threshold=2048000 random_threshold=512 sequential_threshold=1000000 read_promote_adjustment=0' testvg/cached_vol

- Reload the table to apply changes: 

# dmsetup reload --table '0 10485760 cache 253:15 253:14 253:16 4096 1 writeback cleaner 0 read_promote_adjustment 0 sequential_threshold 1000000 random_threshold 512 migration_threshold 2048000 discard_promote_adjustment 0 write_promote_adjustment 0' testvg-cached_vol

# dmsetup suspend testvg-cached_vol
# dmsetup resume testvg-cached_vol

# dmsetup table testvg-cached_vol
0 10485760 cache 253:15 253:14 253:16 4096 1 writeback cleaner 0 read_promote_adjustment 0 sequential_threshold 1000000 random_threshold 512 migration_threshold 2048000 discard_promote_adjustment 0 write_promote_adjustment 0

# mkfs.ext4 /dev/testvg/cached_vol
# mount /dev/testvg/cached_vol /cached_vol/
# dd if=/dev/zero of=/cached_vol/testfile bs=4M count=500
...
2147483648 bytes (2.1 GB) copied, 1.26503 s, 1.7 GB/s  

Only a few dirty blocks: 

# dmsetup status testvg-cached_vol
0 10485760 cache 8 41/262144 4096 3/5120 289 101 694 41209 0 0 3 1 writeback 2 migration_threshold 2048 cleaner 0

# lvremove testvg/cached_vol
Do you really want to remove active logical volume cached_vol? [y/n]: y
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  [...]

# lvconvert --splitcache testvg/cached_vol
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  3 blocks must still be flushed.
  [...]