For performance reasons the dirty bitset is not updated on disk whilst the target is running.  This would cause
latency, since we'd have to hold every bio that is about to dirty a block until the bitset had been updated.

Instead, we write out the complete dirty bitset on clean shutdown.  This means if there is a crash we
have to assume all blocks are dirty.  We do not know that the last activation was in writethrough mode, indeed there
are scenarios where writethrough mode could be enabled and there be dirty blocks.

So seeing dirty blocks after a crash is not a bug.

However if the dirty blocks are all written back, and then a _clean_ reboot is performed you should see no dirty
blocks after reboot.

The following tests check a couple of properties of a writetrhough cache:

https://github.com/jthornber/device-mapper-test-suite/blob/master/lib/dmtest/tests/cache/writethrough_tests.rb

i) check that a cache is still clean after applying an io workload
ii) after a crash, reboot, clean, reboot cycle it is still clean.

I've run these against upstream kernels and the RHEL 7.6 kernel and both pass.

At this point I need more info:

- is the cache clean at the point you reboot?
- is the cache taken down cleanly?  eg, fs unmounted, lv deactivated?
- after the reboot how many blocks are dirty?  How long does it take to subsequently clean it?
- do dirty blocks appear if you just deactivate then activate the cache lv with out rebooting?

I ran into a similar issue on a Fedora 30 desktop. My partitioning was something like:

/        LVM cached writeback
/boot    Simple GPT partition
/home    LVM cached writeback

On every single boot the entire cache for / got marked dirty and took hours to clean, but /home was not affected at all.

I suspect Fedora is not cleanly shutting down LVM when root is cached.

(In reply to Lucas Vinicius Hartmann from comment #12)
> I ran into a similar issue on a Fedora 30 desktop. My partitioning was
> something like:
> 
> /        LVM cached writeback
> /boot    Simple GPT partition
> /home    LVM cached writeback
> 
> On every single boot the entire cache for / got marked dirty and took hours
> to clean, but /home was not affected at all.
> 
> I suspect Fedora is not cleanly shutting down LVM when root is cached.

Thanks Lucas.  I'm able to reproduce what you're seeing.  I installed F30 using automatic LVM partitioning, then added root to cache using https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/logical_volume_manager_administration/lvm_cache_volume_creation.  

# vgextend fedora_localhost-live /dev/vdb
# lvcreate -n lv_cache -L 2G fedora_localhost-live /dev/vdb
# lvcreate -L 30m lv_cache_meta fedora_localhost-live /dev/vdb
# lvcreate -L 30m -n lv_cache_meta fedora_localhost-live /dev/vdb
# lvconvert --type cache-pool --poolmetadata fedora_localhost-live/lv_cache_meta fedora_localhost-live/lv_cache
# lvconvert --type cache --cachepool fedora_localhost-live/lv_cache fedora_localhost-live/root 

Ended up with the below config.

[root@localhost-live ~]# lvs -a
  LV               VG                    Attr       LSize   Pool       Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  [lv_cache]       fedora_localhost-live Cwi---C---   2.00g                         71.77  1.37            0.00            
  [lv_cache_cdata] fedora_localhost-live Cwi-ao----   2.00g                                                                
  [lv_cache_cmeta] fedora_localhost-live ewi-ao----  32.00m                                                                
  [lvol0_pmspare]  fedora_localhost-live ewi-------  32.00m                                                                
  root             fedora_localhost-live Cwi-aoC--- <12.50g [lv_cache] [root_corig] 71.77  1.37            0.00            
  [root_corig]     fedora_localhost-live owi-aoC--- <12.50g                                                                
  swap             fedora_localhost-live -wi-ao----   1.50g          

After the config was done, I rebooted and when the system came back up dirty blocks was gradually growing.

fedora_localhost--live-root: 0 26206208 cache 8 109/8192 128 23511/32768 30852 52940 4413 22171 0 22193 0 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 
fedora_localhost--live-root: 0 26206208 cache 8 109/8192 128 23513/32768 30857 52940 4431 22182 0 22195 0 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 
fedora_localhost--live-root: 0 26206208 cache 8 109/8192 128 23513/32768 30886 52944 4515 22182 0 22195 0 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 
fedora_localhost--live-root: 0 26206208 cache 8 109/8192 128 23517/32768 31201 53005 4535 22183 0 22199 0 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 
fedora_localhost--live-root: 0 26206208 cache 8 109/8192 128 23519/32768 31328 53007 4553 22187 0 22201 0 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 

Your symptoms and the symptoms of the original issue are the same, but I'm unsure if the root cause is the same.

Created attachment 1589665 [details]
Used and dirty blocks over time.

Space is void while the computer is powered down, except for the glitch on 6/11 09:00 which was a network issue.
Notice full-conversion of all used block to dirty on every power cycle.
All reboots were supposedly clean shutdowns.

Scenario: ext4 / filesystem, on lvm-cached-lvs, backed by a pair of lvm-pvs, each PV separately encrypted with luks. No separate /root mountpoint.

I happened to be playing with InfluxDB and Grafana at the time, which is how I stumbled on the issue.

(In reply to John Pittman from comment #13)

Just occurred to me, did you enable noatime mount option for the cached partition?

Maybe during boot there is a piece of software searching/scanning/reading a lot of files? Without using noatime reading files will cause access times to be updated/written back to the disk. This could potentially lead to an increase in dirty blocks at boot, followed by normal operation then.

Careful, though, as noatime may have adverse effects on some applications.

(In reply to Lucas Vinicius Hartmann from comment #16)
> (In reply to John Pittman from comment #13)
> 
> Just occurred to me, did you enable noatime mount option for the cached
> partition?
> 
> Maybe during boot there is a piece of software searching/scanning/reading a
> lot of files? Without using noatime reading files will cause access times to
> be updated/written back to the disk. This could potentially lead to an
> increase in dirty blocks at boot, followed by normal operation then.
> 
> Careful, though, as noatime may have adverse effects on some applications.

Thanks, I tried adding noatime, and rebooted.  When the system came back up there were maybe 60 dirty blocks and the value was slowly incrementing.  However, I tried about 5 more reboots with noatime, then maybe 10 without, and have not been able to reproduce again.  Note my cache volume is writethrough and yours is writeback.  Will look further later.

(In reply to Lucas Vinicius Hartmann from comment #16)
> (In reply to John Pittman from comment #13)
> 
> Just occurred to me, did you enable noatime mount option for the cached
> partition?
> 
> Maybe during boot there is a piece of software searching/scanning/reading a
> lot of files? Without using noatime reading files will cause access times to
> be updated/written back to the disk. This could potentially lead to an
> increase in dirty blocks at boot, followed by normal operation then.
> 
> Careful, though, as noatime may have adverse effects on some applications.

Hi Lucas, would you mind please splitting off your issue into a separate Fedora 30 bug?  We will continue to track your issue there.  You can note in that bug where you split it off from here.

As a note, I was mistaken that I reproduced your issue.  It was late in the day and I think my brain was seeing what it wanted. :)

fedora_localhost--live-root: 0 26206208 cache 8 109/8192 128 23511/32768 30852 52940 4413 22171 0 22193 [0] 2 metadata2 writethrough 2 migration_threshold 2048 smq 0 rw - 

Above, I've put the dirty blocks in brackets []; you can see they are 0.

I have a reproduction environment, but I have not yet been able to reproduce as the customer has.

LV               VG  Attr       LSize  Pool       Origin     Data%  Meta%  Move Log Cpy%Sync Convert Devices          
lv               vg2 Cwi-aoC--- <3.42g [lv_cache] [lv_corig] 99.95  2.13            0.00             lv_corig(0)      
[lv_cache]       vg2 Cwi---C---  2.00g                       99.95  2.13            0.00             lv_cache_cdata(0)
[lv_cache_cdata] vg2 Cwi-ao----  2.00g                                                               /dev/md0p2(0)    
[lv_cache_cmeta] vg2 ewi-ao---- 20.00m                                                               /dev/md0p2(512)  
[lv_corig]       vg2 owi-aoC--- <3.42g                                                               /dev/rbd0(0)     
[lvol0_pmspare]  vg2 ewi------- 20.00m                                                               /dev/md0p2(517)  

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vg2-lv 253:145  0  3.4G  0 lvm  /mnt/ceph-block-device2

In one terminal, I run:

# fio --bs=128k --ioengine=libaio --rw=randrw --name=blah --filename=/mnt/ceph-block-device2/file --size=3g --direct=0

And in another:

# while true ; do fio --bs=32k --ioengine=libaio --rw=randrw --name=blah2 --filename=/mnt/ceph-block-device2/file2 --size=40m --direct=1 ; done

I let them run for a while, then ctrl-c, quickly check that there are no dirty blocks, then reboot.  This process has not reproduced the issue.  I've also tried just killing the fio threads, unmounting the filesystem, then vgchange -an/-ay; this did not reproduce either.

Please note however, I did once issue a reboot while the fio threads were still running.  This caused all sorts of hung tasks and failures.  When the system came back up, there were many dirty blocks which eventually drained to 0.  So I was able to reproduce dirty blocks in that way.

If there are flaws in my procedure, or anything you guys want me to try, please let me know.  Otherwise, I will report back if I or the customer finds anything else.

Someone asking similar questions posed here at the below link:

https://www.redhat.com/archives/dm-devel/2019-July/msg00114.html

As a note, in my reproduction environment, I added the NFS setup.  Even when issuing I/O to the nfs mount from the nfs client system, then rebooting the nfs server while the I/O is still going, I have been unable to reproduce.

Still the only reproduction I've been able to do is when I was issuing I/O directly to the cache LV, then rebooting while the fio threads were still issuing I/O.  All sorts of hung tasks and failures as the system goes down; dirty blocks when the system comes back up.

Closing since no one has produced this issue if there's a clean shutdown.