The 'usable space' when VDO is created is actually slightly greater than the usable space at later times, because VDO only allocates its block map as it receives writes to never-before-used logical addresses. If you fill the logical space, I believe the calculation is such that you'll have between 0 and 5 usable blocks left over.

In this case the logical space is completely filled (writing the same amount of data as there is space in logical blocks), if I try to write more, I get "out of space" in dd. So at this point I should expect 0 - 5 usable blocks left, but I am seeing 480 1-K blocks left, which is much more.

I am even able to increase the logical size to the point where I get 0 "available" 1-K blocks and still no I/O errors:
# vdo create --name vdo --device /dev/mapper/test-small
Creating VDO vdo
Starting VDO vdo
Starting compression on VDO vdo
VDO instance 1 volume is ready at /dev/mapper/vdo
# vdo status | grep ogical
    Logical size: 2091956K
    Logical threads: 1
        logical blocks: 522989
        logical blocks used: 0
### new logical size: 2091956K + 480K = 2092436K
# vdo growLogical --name vdo --vdoLogicalSize 2092436K
# vdostats
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo        5242880   3150444   2092436  60%          100%
# dd if=/dev/urandom of=/dev/mapper/vdo bs=4K count=523109 status=progress
2055581696 bytes (2.1 GB) copied, 11.000075 s, 187 MB/s
523109+0 records in
523109+0 records out
2142654464 bytes (2.1 GB) copied, 30.0255 s, 71.4 MB/s
# vdo status | grep ogical
    Logical size: 2092436K
    Logical threads: 1
        logical blocks: 523109
        logical blocks used: 523109
# vdostats
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo        5242880   5242880         0 100%            0%

So this means these 480 1-K blocks are "usable" by user, but not accessible by default, as default vdo has its logical size smaller than the "usable" physical space.

I was actually incorrect.

Each of our block map blocks stores 812 logical addresses' mappings. We only deal in whole block map blocks. So if we added 812 more logical addresses, we'd need (up to 5 more block map blocks) + (812 more physical blocks), which we don't have. Thus the number of logical addresses picked.

"We only deal in whole block map blocks."... (I think, for the purposes of picking a default logical size.)

I believe the problem isn't leaf map page granularity, or at least, not that simply. We can specify an exact number of logical blocks, even if it's not a multiple of 812 (522989 isn't a multiple of 812). The issue is that the calculation is a recurrence, and one that might oscillate. We chose to do the simplest first-order approximation of reserving enough logical blocks to address every possible data block. That reserves too many, since the reserved blocks themselves will never need to be mapped. We thought at the time the difference between the simplest approximation and a more complicated one wasn't significant enough (as above, fewer than one block per 10000) to justify the complexity.

The problem wasn't the approximation, but that for BZ 1519330 we used a function that measures the amount to space needed in memory, which includes 2 4K blocks for each of 60 tree roots which we don't allocate from the pool of data blocks. 2 * 60 * 4K = 480K.

I used 6G LV as backing device to test this:

# vdo create --name vdo --device /dev/mapper/vg-lv
Creating VDO vdo
Starting VDO vdo
Starting compression on VDO vdo
VDO instance 7 volume is ready at /dev/mapper/vdo

# dd if=/dev/urandom of=/dev/mapper/vdo bs=4K count=523108 status=progress
2130739200 bytes (2.1 GB) copied, 35.001183 s, 60.9 MB/s
523108+0 records in
523108+0 records out
2142650368 bytes (2.1 GB) copied, 69.3447 s, 30.9 MB/s
# vdostats
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo        6291456   6291452         4  99%            0%



It seems there is still 1 4K block left. Using new VDO:
# vdo growLogical --name vdo --vdoLogicalSize 2092432K
vdo: ERROR - Can't grow a VDO volume by less than 4096 bytes

# vdo growLogical --name vdo --vdoLogicalSize 2092436K

# dd if=/dev/urandom of=/dev/mapper/vdo bs=4K count=523109 status=progress
2103742464 bytes (2.1 GB) copied, 35.001183 s, 60.1 MB/s
523109+0 records in
523109+0 records out
2142654464 bytes (2.1 GB) copied, 70.7104 s, 30.3 MB/s

# vdostats
Device               1K-blocks      Used Available Use% Space saving%
/dev/mapper/vdo        6291456   6291456         0 100%            0%


So I am still able to grow the VDO and write to it random data without any errors. It is much smaller, just 1 block now. It is possible to change the default logical size to contain this block too? Also what is the reason behind this one, according to #c9 there was exactly 480K missed out for good reason, so where did this last block come from?
Thanks

The 480K was a bug that got fixed, which is what I tried to say in #c9. The explanation of why it can't be exact is in https://bugzilla.redhat.com/show_bug.cgi?id=1645690#c3

(In reply to sclafani from comment #12)
> The 480K was a bug that got fixed, which is what I tried to say in #c9. The
> explanation of why it can't be exact is in
> https://bugzilla.redhat.com/show_bug.cgi?id=1645690#c3

Thanks, setting to verified.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2233