Description of problem: There is no canonical utility for estimating bitmap space usage. Expected results: There would be some utility or well-defined space usage of bitmaps in qcow2 files that can be used by RHV to predict image growth. At the moment, bitmap space usage looks like this: ``` def bitmap_estimate(nbitmaps, disk_size, cluster_size, granularity): if nbitmaps <= 0: return 0 # Bitmap Directory # Assume name = 1023 bytes bd_entry = 24 + 1024 bd_size = nbitmaps * bd_entry bd_clusters = ceil(bd_size / cluster_size) # Bitmap Data data_size = ceil(ceil(disk_size / granularity) / 8) data_clusters = ceil(data_size / cluster_size) # Bitmap Table bt_size = data_clusters * 8 bt_clusters = ceil(bt_size / cluster_size) # Total clusters = ((data_clusters + bt_clusters) * nbitmaps) clusters += bd_clusters + 1 return clusters * cluster_size ``` This function assumes all bitmaps have the same granularity, but that isn't necessarily true in practice. Use the smallest cluster size you intend to use as a worse case scenario. It also assumes all bitmap names are the longest they can possibly be, but you probably won't see any actual space savings in the real world if you have less than ~ 50 bitmaps on a 64KiB cluster qcow2. Distilled into very fuzzy math: The bitmaps feature itself needs: - Space for the bitmap feature metadata itself (Not more than 1, ever) - Space for the bitmap directory (between 32 or 1028 bytes per bitmap, rounded up to cluster size) Each bitmap needs: - about (disk_size / granularity / 8) bytes; rounded up to cluster size. - about (disk_size / granularity / cluster_size^2) clusters for the bitmap table Adding a good measurement/estimate function might be difficult; the existing qemu-img measure function does not support specifying future "advanced" feature usage, so an API has to be designed. Perhaps any of the estimates above are suitable as a workaround for now.
I think what we need is something like: qemu-img measure -f qcow2 -O qcow2 --bitmaps src.qcow2 This will include the bitmaps in src.qcow2 when reporting the required size, so: lvcreate --size require_size --name lv-name vg-name qemu-img create -f qcow2 /dev/vg-name/lv-name 10g qemu-img convert -n -f qcow2 -O qcow2 --bitmaps src.qcow2 /dev/vg-name/lv-name Will copy the image and the bitmaps successfully to the target image. The current way to do this is to run: qemu-img info --output json src.qcow2 Then collect all the bitmaps under "bitmaps" and assuming that all bitmaps conver the entire image, and calculate the size of the bitmap based on granularity, and some extra space for the metadata overhead mentioned in comment 0. Related discussion in qemu-block: https://lists.nongnu.org/archive/html/qemu-block/2020-04/msg00745.html
Related to bug 1779893.
Quick-and-dirty upstream patch proposed - you don't even have to add a --bitmaps parameter. If the output format is qcow2 and the input format has persistent bitmaps, then the 'qemu-img measure' output will automatically includes a new "bitmaps: ..." line covering the additional size of metadata that will ultimately need to be provided when using 'qemu-img convert --bitmaps'. https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg02681.html
*** Bug 1780359 has been marked as a duplicate of this bug. ***
Hi Aihua, Could you kindly grant qa_ack ? Thanks!
Placing this back into 8.2.1 where it should be for RHEL AV. The most recent patches are still upstream as part of the series: https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg03464.html with the fix being part of patch 4/6: https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg03466.html The expectation is that after a successful review, the patches will be backported, so not removing the devel_ack even though the patches are not downstream yet.
Upstream pull request https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg07419.html
Test on qemu-kvm-4.2.0-24.module+el8.2.1+6959+9b840e7c, details as bellow: 1. Start guest with data disk and cluster_size 512M #qemu-img create -f qcow2 /home/data.qcow2 -o cluster_size=512 2G -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=6 \ -blockdev node-name=file_data1,driver=file,aio=native,filename=/home/data.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_data1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_data1 \ -device virtio-blk-pci,id=data1,drive=drive_data1,write-cache=on,bus=pcie-root-port-4,addr=0x0,iothread=iothread1 \ 2. Add 3 persistent bitmaps with different granularity on data disks {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name':'bitmap0','persistent':true,'granularity':512}} {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name':'bitmap1','persistent':true}} {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name':'bitmap2','persistent':true,'granularity':2147483648}} 3. DD 1G file on data disk (guest)#mkfs.ext4 /dev/vdb #mount /dev/vdb /mnt #cd /mnt #dd if=/dev/urandom of=test bs=1M count=1000 4. Shutdown vm (qemu)system_powerdown 5. Measure bitmap space utilization. #qemu-img measure -f qcow2 -O qcow2 /home/data.qcow2 required size: 1051131904 fully allocated size: 2148073472 bitmaps size: 917504 Hi, Eric Can you help to check if the verification steps are correct? If yes, I will test with maximum bitmaps Thanks, Aliang
(In reply to aihua liang from comment #16) > Test on qemu-kvm-4.2.0-24.module+el8.2.1+6959+9b840e7c, details as bellow: > > 1. Start guest with data disk and cluster_size 512M > #qemu-img create -f qcow2 /home/data.qcow2 -o cluster_size=512 2G Okay, so we have image size 2G, and: > 2. Add 3 persistent bitmaps with different granularity on data disks > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > 'bitmap0','persistent':true,'granularity':512}} bitmap with granularity 512 bytes. In this image, if the bitmap is fully utilized, it will occupy 2G/512 = 4Mbits = 512kbytes = 1024 clusters, plus an additional 2 clusters (1024/512) for the table of clusters used in the bitmap. > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > 'bitmap1','persistent':true}} bitmap with default granularity, which is chosen as 4k (granularity defaults to max(4k, cluster size). If the bitmap is fully utilized, it will occupy 2G/4k = 512kbits = 64kbytes = 128 clusters, plus 1 additional cluster for the table of clusters > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > 'bitmap2','persistent':true,'granularity':2147483648}} bitmap with 2G granularity (wow - the qcow2 spec doesn't forbid a granularity that large?): a single bit is clear while the image is untouched, and set on any write. The bitmap occupies 1 cluster (1 bit used, 511 7/8 bits unused), plus 1 additional cluster for the table. The three bitmaps together still fit within 512 bytes for the overall bitmap table. So the image can use up to (1024+2)+(128+1)+(1+1)+1 = 1158 clusters = 592896 bytes if the bitmap is fully utilized (the image starts out with less space used, because the bitmaps use less space when they are completely empty or completely set). > > 3. DD 1G file on data disk > (guest)#mkfs.ext4 /dev/vdb > #mount /dev/vdb /mnt > #cd /mnt > #dd if=/dev/urandom of=test bs=1M count=1000 Here, you are touching half of each bitmap (well, other than bitmap2 where the lone bit is now set for the entire image). > > 4. Shutdown vm > (qemu)system_powerdown > > 5. Measure bitmap space utilization. > #qemu-img measure -f qcow2 -O qcow2 /home/data.qcow2 > required size: 1051131904 > fully allocated size: 2148073472 > bitmaps size: 917504 But here, you are measuring what would happen if we want to create a NEW qcow2 image that copies the bitmaps out of data.qcow2 into the new image; and since the new image did NOT specify -o cluster_size, the new image will default to 64k clusters rather than your explicit 512-byte cluster in the source. So we have to recalculate the numbers: copying bitmap0 would require 2G/512 = 4Mbits = 512kbytes, ceil(512k/64k) = 8 clusters, plus ceil(8/64k) = 1 cluster for the map copying bitmap1 would require 2G/4k = 512kbits = 64kbytes, ceil(64k/64k) = 1 cluster, plus ceil(1/64k) = 1 cluster for the map copying bitmap2 would require 2G/2G = 1 bit, ceil(1/64k) = 1 cluster, plus ceil(1/64k) = 1 cluster for the map and we still fit all three bitmaps in one cluster for the bitmap table the image requires (8+1)+(1+1)+(1+1)+1 = 14 clusters = 917504 bytes. The output is correct. > > > Hi, Eric > Can you help to check if the verification steps are correct? If yes, I will > test with maximum bitmaps Yes, your steps look correct, although you may want to play with -o cluster_size in the measure command.
Hi, Eric When measure with -o cluster_size=512, the bitmap size is 600576 not 592896 as expected. #qemu-img measure -f qcow2 -O qcow2 /home/data.qcow2 -o cluster_size=512 required size: 1092783104 fully allocated size: 2190253056 bitmaps size: 600576 BR， Aliang
Can't verify 65535 bitmaps for bz#1846832, so will verify this scenario after bz#1846832 fixed.
Hi, Eric Can you help to check comment18, the bitmaps size is not as expected when set dst cluster_size=512. BR, Aliang
(In reply to aihua liang from comment #20) > Hi, Eric > > Can you help to check comment18, the bitmaps size is not as expected when > set dst cluster_size=512. > > BR, > Aliang The code is correct. I found the error in my hand calculation: (In reply to Eric Blake from comment #17) > > 2. Add 3 persistent bitmaps with different granularity on data disks > > > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > > 'bitmap0','persistent':true,'granularity':512}} > > bitmap with granularity 512 bytes. In this image, if the bitmap is fully > utilized, it will occupy 2G/512 = 4Mbits = 512kbytes = 1024 clusters, plus > an additional 2 clusters (1024/512) for the table of clusters used in the > bitmap. Actually, it is 8 bytes per cluster for the table of clusters, not 1 byte per cluster. So this is 1024 clusters plus an additional (1024*8/512) = 16 clusters for the table of clusters. > > > > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > > 'bitmap1','persistent':true}} > > bitmap with default granularity, which is chosen as 4k (granularity defaults > to max(4k, cluster size). If the bitmap is fully utilized, it will occupy > 2G/4k = 512kbits = 64kbytes = 128 clusters, plus 1 additional cluster for > the table of clusters and this is (128*8/512) = 2 clusters for the table. > > > > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > > 'bitmap2','persistent':true,'granularity':2147483648}} > > bitmap with 2G granularity (wow - the qcow2 spec doesn't forbid a > granularity that large?): a single bit is clear while the image is > untouched, and set on any write. The bitmap occupies 1 cluster (1 bit used, > 511 7/8 bits unused), plus 1 additional cluster for the table. This one is correct at 1 cluster for the table. > > The three bitmaps together still fit within 512 bytes for the overall bitmap > table. So the image can use up to (1024+2)+(128+1)+(1+1)+1 = 1158 clusters > = 592896 bytes if the bitmap is fully utilized (the image starts out with > less space used, because the bitmaps use less space when they are completely > empty or completely set). The image needs (1024+16)+(128+2)+(1+1)+1 = 1173 clusters or 600576 bytes. Serves me right for trying things by hand instead of reading the code, and thankfully the code is correct.
(In reply to Eric Blake from comment #21) > (In reply to aihua liang from comment #20) > > Hi, Eric > > > > Can you help to check comment18, the bitmaps size is not as expected when > > set dst cluster_size=512. > > > > BR, > > Aliang > > The code is correct. I found the error in my hand calculation: > > (In reply to Eric Blake from comment #17) > > > > 2. Add 3 persistent bitmaps with different granularity on data disks > > > > > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > > > 'bitmap0','persistent':true,'granularity':512}} > > > > bitmap with granularity 512 bytes. In this image, if the bitmap is fully > > utilized, it will occupy 2G/512 = 4Mbits = 512kbytes = 1024 clusters, plus > > an additional 2 clusters (1024/512) for the table of clusters used in the > > bitmap. > > Actually, it is 8 bytes per cluster for the table of clusters, not 1 byte > per cluster. So this is 1024 clusters plus an additional (1024*8/512) = 16 > clusters for the table of clusters. > > > > > > > > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > > > 'bitmap1','persistent':true}} > > > > bitmap with default granularity, which is chosen as 4k (granularity defaults > > to max(4k, cluster size). If the bitmap is fully utilized, it will occupy > > 2G/4k = 512kbits = 64kbytes = 128 clusters, plus 1 additional cluster for > > the table of clusters > > and this is (128*8/512) = 2 clusters for the table. > > > > > > > > > {'execute':'block-dirty-bitmap-add','arguments':{'node':'drive_data1','name': > > > 'bitmap2','persistent':true,'granularity':2147483648}} > > > > bitmap with 2G granularity (wow - the qcow2 spec doesn't forbid a > > granularity that large?): a single bit is clear while the image is > > untouched, and set on any write. The bitmap occupies 1 cluster (1 bit used, > > 511 7/8 bits unused), plus 1 additional cluster for the table. > > This one is correct at 1 cluster for the table. > > > > > The three bitmaps together still fit within 512 bytes for the overall bitmap > > table. So the image can use up to (1024+2)+(128+1)+(1+1)+1 = 1158 clusters > > = 592896 bytes if the bitmap is fully utilized (the image starts out with > > less space used, because the bitmaps use less space when they are completely > > empty or completely set). > > The image needs (1024+16)+(128+2)+(1+1)+1 = 1173 clusters or 600576 bytes. > > Serves me right for trying things by hand instead of reading the code, and > thankfully the code is correct. Got it, thanks Eric.
Test on qemu-kvm-4.2.0-24.module+el8.2.1+6959+9b840e7c, the problem has been resolved, set bug's status to "Verified".
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3172