Description of problem:

It is possible for the metadata space of a Stratis pool to become full under certain conditions.

Version-Release number of selected component (if applicable):
stratisd-3.4.0-1.el9
(Upstream issue, originally opened August 10, 2022: https://github.com/stratis-storage/project/issues/495 )


How reproducible:
This issue was originally difficult to reproduce, but the procedure below should be able to reproduce 100% of the time.
It seems to be more reproducible with a smaller block device, and a larger data file written to the initial filesystem.


Steps to Reproduce:
1. Create a Stratis pool on a device partition of 8 GiB in size, as "spool1".
# stratis pool create spool1 <device>

2. Create a filesystem on the pool with the following command:
# stratis fs create spool1 sfs1 --size 2GiB

3. Create a mount directory, "/mnt/sfs1", and mount the filesystem:
# mount /dev/stratis/spool1/sfs1 /mnt/sfs1

4. Using the fio program, write a data file to the mountpoint /mnt/sfs1.

---(contents of fio config file: 800M_randwrite_sfs1.ini)---
[globals]

bs=4096
rw=randwrite
name=800M_randwrite_sfs1
numjobs=1
size=800M
group_reporting
direct=1
iodepth=1

[thread1]
filename=/mnt/sfs1/testfile
---(end of fio config)---

# fio 800M_randwrite_sfs1.ini


5. After the file has been written, start creating multiple snapshots of the filesystem sfs1:

# stratis fs snapshot spool1 sfs1 sfs1snap1

6. Repeat the "fs snapshot" command from "sfs1" to new snapshot names.

Additionally, use the "dmsetup status" command to monitor the thin pool metadata usage:
# dmsetup status --target=thin-pool



Actual results:

The thin pool of the Stratis pool reports "No free metadata blocks"

(Example, with an 8 GiB block device holding a pool with a 2 GiB initial filesystem:)

(18 snapshots: 123/165)

(19 snapshots: 128/165)
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.

(20 snapshots: 133/165)
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.

(21 snapshots: 138/165)
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.

(22 snapshots: 143/165)
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.
kernel: device-mapper: thin: 253:4: reached low water mark for metadata device: sending event.

(23 snapshots: 148/165)

(24 snapshots: 153/165)

(25 snapshots: 158/165)

(26 snapshots: 163/165)

(27 snapshots: 165/165; thin-pool in "ro" mode)
kernel: device-mapper: thin: No free metadata blocks
kernel: device-mapper: thin: 253:4: switching pool to read-only mode



Expected results:

The thin pool of the Stratis pool does not run out of metadata space.