I think I see the issue: when max_bytes is less than 4M (1 << CEPH_BLOCK_SHIFT), ceph_quota_update_statfs will end up returning false, and we fall back to using the info in ceph_statfs structure instead of the quota info. Is it sane to allow subvolumes that are that small?

One thing you could do is change the f_bsize/f_frsize to be smaller in the cases where the maxsize is less than 4M, and recalculate f_blocks/f_used based on that blocksize.

We need to consider how to handle this situation.

Currently we report a 4M blocksize in statfs, but that doesn't work for subvols that are smaller than that, so we'll need to use a smaller blocksize in those cases (at least). We can either just switch to a smaller blocksize when the quota is very small, or we could just use a smaller blocksize everywhere.

Which would be preferred? What smaller blocksize should we use? 1M? 1K? What's the smallest quota you can set?

Patrick, thoughts?

(In reply to Jeff Layton from comment #4)
> We need to consider how to handle this situation.
> 
> Currently we report a 4M blocksize in statfs, but that doesn't work for
> subvols that are smaller than that, so we'll need to use a smaller blocksize
> in those cases (at least). We can either just switch to a smaller blocksize
> when the quota is very small, or we could just use a smaller blocksize
> everywhere.
> 
> Which would be preferred? What smaller blocksize should we use? 1M? 1K?
> What's the smallest quota you can set?
> 
> Patrick, thoughts?

Changing the block size could have unintended consequences I think. I do agree we should (with 20/20 hindsight) not allow such small quotas but that ship has sailed (although we could change it!). Is it possible to keep the 4MB blocksize but just have a very binary use/free: either you've used the full block or the whole block is free?

(In reply to Patrick Donnelly from comment #5)
> 
> Changing the block size could have unintended consequences I think. I do
> agree we should (with 20/20 hindsight) not allow such small quotas but that
> ship has sailed (although we could change it!). Is it possible to keep the
> 4MB blocksize but just have a very binary use/free: either you've used the
> full block or the whole block is free?

It could have unintended consequences, but not much actually depends on the blocksize (particularly not with a netfs like ceph). We probably could change it and just see how it goes.

We could just go with the binary used/free like you suggest and leave the blocksize alone. It might make for weird accounting if you have a bunch of tiny quotas, but I guess it'd still be technically "correct".

I think I didn't get the benefit of going with binary use/free for subvolumes less than blocksize (4M)?

The quota calculation and validation happens at bytes level and it works correctly. The issue is
solely related to 'statfs' report. If we decide to go by binary use/free of full block, the
'total=used=free=0' for subvolumes less than block size (i.e. 4M). But the 'df' tool doesn't
report mounts with 0 size. Is that the expectation?

(In reply to Kotresh HR from comment #7)
> Is that the expectation?

FWIW, this problem is probably one of those where coding the fix won't be hard at all, but figuring out what we should code up is.

Good point about df not reporting those filesystems. That will certainly be the result and that's probably not what we want. Also, a binary use/free like that is probably not what users want either. If they're declaring quotas that small, then they probably want to be able to see that reflected in "df" or similar applications.

Let's step back and think about what applications really want here. statfs comes from a simpler time when all UNIX had to deal with was block-based filesystems. All netfs's fake this up to some degree. In practice most applications don't care about the blocksize at all -- they just use it to multiply the other values from statfs(2) to get values in bytes.

There is no reason we have to report the same f_frsize everywhere. We could keep a 4M blocksize for "reasonably sized" volumes and report a smaller blocksize for inodes with tiny quotas.

Just looking around, NFS uses these values:

    #define NFS_MAX_FILE_IO_SIZE    (1048576U)
    #define NFS_DEF_FILE_IO_SIZE    (4096U)
    #define NFS_MIN_FILE_IO_SIZE    (1024U)


Maybe what we should do is switch to reporting a 4k blocksize in ceph_quota_update_statfs when ci->i_max_bytes is below some threshold. That threshold could be 4M, or we could make it break at a larger value (64M? 1G?).

One caveat -- avoid using a blocksize that is so small that it causes other "count" values to overflow a 32-bit value. That can result in old 32-bit programs getting an EOVERFLOW from glibc when the values don't fit in a legacy statfs struct.

(In reply to Kotresh HR from comment #7)
> I think I didn't get the benefit of going with binary use/free for
> subvolumes less than blocksize (4M)?
> 
> The quota calculation and validation happens at bytes level and it works
> correctly. The issue is
> solely related to 'statfs' report. If we decide to go by binary use/free of
> full block, the
> 'total=used=free=0' for subvolumes less than block size (i.e. 4M).

No, it would be total=free=4MB, used=0 for subvolumes with a quota <= 4MB that is not fully exhausted. If the quota is full, then total=used=4MB, free=0.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: kernel security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1988