Description of problem: [incremental backup] backup in push mode will actually write ZEROes to target image Version-Release number of selected component (if applicable): libvirt-6.0.0-9.module+el8.2.0+5957+7ae8988e.x86_64 How reproducible: 100% Steps to Reproduce: 1. Create a virtual_size=5G qcow2 file [root@hp-dl320eg8-05 push]# qemu-img create -f qcow2 /var/lib/libvirt/images/vdb.qcow2 5G Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 size=5368709120 cluster_size=65536 lazy_refcounts=off refcount_bits=16 2. Use it in a vm as vdb, and generate 100M data on it [root@hp-dl320eg8-05 push]# virsh domblklist vm1 Target Source -------------------------------------------------- vda /var/lib/libvirt/images/base_os.active vdb /var/lib/libvirt/images/vdb.qcow2 [root@hp-dl320eg8-05 push]# virsh start vm1 Domain vm1 started In vm: # mkfs.ext4 /dev/vdb # mount /dev/vdb /mnt # dd if=/dev/urandom of=/mnt/100M bs=1M count=100; sync 3. Clear the libvirtd log to capture fresh debug log [root@hp-dl320eg8-05 push]# echo "" > /var/log/libvirtd-debug.log 4. Start the backup job [root@hp-dl320eg8-05 push]# cat backup_push_full.xml <domainbackup mode='push'> <disks> <disk name='vdb' type='file'> <target file='/home/images/vdb.full.backup'/> <driver type='qcow2'/> </disk> <disk name='vda' backup='no'/> </disks> </domainbackup> [root@hp-dl320eg8-05 push]# cat checkpoint_push_full.xml <domaincheckpoint> <name>check_full</name> </domaincheckpoint> [root@hp-dl320eg8-05 push]# virsh backup-begin vm1 backup_push_full.xml checkpoint_push_full.xml Backup started 5. Wait for job end, it took around 48 seconds, and all ZEROes actually write to host disk [root@hp-dl320eg8-05 push]# virsh domjobinfo vm1 --completed Job type: Completed Operation: Backup Time elapsed: 48601 ms File processed: 5.000 GiB File remaining: 0.000 B File total: 5.000 GiB [root@hp-dl320eg8-05 push]# qemu-img info /home/images/vdb.full.backup image: /home/images/vdb.full.backup file format: qcow2 virtual size: 5 GiB (5368709120 bytes) disk size: 5 GiB <=============== 5G data wrote to disk cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false Actual result: As step 5, all ZEROes wrote to disk, this is quite slow and take lots of host disk space unnecessary when image has a large virtual size. Expected: File holes should not cause actually writing ZEROes. Additional info: With blockcopy function, we can see only valid data transferred to dest file, so the push backup function could be improved with holes. [root@hp-dl320eg8-05 push]# virsh blockcopy vm1 vdb /tmp/vdb.blockcopy --transient-job Block Copy started [root@hp-dl320eg8-05 push]# virsh blockjob vm1 vdb Block Copy: [100 %] [root@hp-dl320eg8-05 push]# qemu-img info /tmp/vdb.blockcopy -U image: /tmp/vdb.blockcopy file format: qcow2 virtual size: 5 GiB (5368709120 bytes) disk size: 105 MiB <============== 105M data wrote to disk cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false
Created attachment 1671061 [details] libvirtd-debug.log
Moving to qemu for further investigation. Libvirt attaches the image: 2020-03-18 13:21:27.174+0000: 50759: info : qemuMonitorSend:996 : QEMU_MONITOR_SEND_MSG: mon=0x7f80944057c0 msg={"execute":"blockdev-add","arguments":{"driver":"file","filename":"/home/images/vdb.full.backup","node-name":"libvirt-4-storage","auto-read-only":true,"discard":"unmap"},"id":"libvirt-370"}^M 2020-03-18 13:21:27.317+0000: 50759: info : qemuMonitorSend:996 : QEMU_MONITOR_SEND_MSG: mon=0x7f80944057c0 msg={"execute":"blockdev-add","arguments":{"node-name":"libvirt-4-format","read-only":false,"driver":"qcow2","file":"libvirt-4-storage","backing":"libvirt-1-format"},"id":"libvirt-374"} And starts the job. {"type":"blockdev-backup","data":{"device":"libvirt-1-format","job-id":"backup-vdb-libvirt-1-format","target":"libvirt-4-format","sync":"full","auto-finalize":true,"auto-dismiss":false}} so what's written into the file is under control of qemu. We indeed don't want to enable zero detection in this case as it implies a performance penalty.
qemu was told to do a full backup, so it MUST ensure that the destination image contains the same contents as the source (even though the source is 100M of non-zero and 4.9G of zero bytes). I am working on code for upstream qemu 5.1 that will let qemu be smarter about realizing when a destination already starts life as all zeroes (which is the case for a just-created qcow2 or sparse file), at which point we can optimize qemu to quit writing explicit zeroes when it knows the destination already reads as zeroes. https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08075.html But even if qcow2 itself learns how to record when an image still reads as all zeroes, there may be cases where we need to inform qemu to behave as if the image reads as all zeroes even when qemu can't prove that (we recently added 'qemu-img convert --target-is-zero' for 5.0) - we would need additional patches to qemu to support this same sort of knob for the destination of a blockdev-backup operation, and I don't know of any patches started on that front, yet. Looks like I get to expand the scope of my work on fast zero detection when I get back to v2 of my qcow2 patches mentioned above...
The same issue with that in https://bugzilla.redhat.com/show_bug.cgi?id=1802401, and can't hit it now. Hi, Sunyi Please help to check if it still exist in your latest test version. If not, will close it as dup to 1802401. Thanks, Aliang
Tested with: [root@dell-per740xd-10 ~]# rpm -qa | egrep "^libvirt-6|^qemu-kvm-5" qemu-kvm-5.1.0-6.module+el8.3.0+8041+42ff16b8.x86_64 libvirt-6.6.0-4.module+el8.3.0+7883+3d717aa8.x86_64 Result: PASS Steps: 1. prepare a 5G virtual size qcow2 file vdb.qcow2 [root@dell-per740xd-10 ~]# qemu-img create -f qcow2 /var/lib/libvirt/images/vdb.qcow2 5G Formatting '/var/lib/libvirt/images/vdb.qcow2', fmt=qcow2 cluster_size=65536 compression_type=zlib size=5368709120 lazy_refcounts=off refcount_bits=16 [root@dell-per740xd-10 ~]# virsh edit vm1 Domain vm1 XML configuration edited. 2. use it in vm [root@dell-per740xd-10 ~]# virsh start vm1 Domain vm1 started [root@dell-per740xd-10 ~]# virsh domblklist vm1 Target Source -------------------------------------------------------- vda /var/lib/libvirt/images/jeos-27-x86_64.qcow2 vdb /var/lib/libvirt/images/vdb.qcow2 3. create 100M data in vm's vdb, and get 10 lines of its hex data [root@dell-per740xd-10 ~]# virsh console vm1 Connected to domain vm1 Escape character is ^] (Ctrl + ]) Red Hat Enterprise Linux 8.3 Beta (Ootpa) Kernel 4.18.0-235.el8.x86_64 on an x86_64 localhost login: [root@localhost ~]# mkfs.ext4 /dev/vdb mke2fs 1.45.6 (20-Mar-2020) Discarding device blocks: done Creating filesystem with 1310720 4k blocks and 327680 inodes Filesystem UUID: c1019c90-591b-48e1-bfe1-417223bb9ca0 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done [root@localhost ~]# mount /dev/vdb /mnt/ [ 56.128699] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (null) [root@localhost ~]# dd if=/dev/urandom of=/mnt/100M bs=1M count=100; sync 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.660273 s, 159 MB/s [root@localhost ~]# hexdump /mnt/100M | head -10 0000000 25fc 1cf5 28bb 6dd5 2236 2493 c578 c7ff 0000010 45a9 95ba ea17 4222 7cc2 71ed 3c12 c2a5 0000020 4511 a292 79f0 2e64 a02f e771 3841 389c 0000030 9f00 3756 3206 1653 bba5 217e b857 efac 0000040 a474 81ff 014e 2028 580f 658f 7b3b f7a9 0000050 65f8 39fd 01b6 ac6c 1327 e738 d5cb c5ae 0000060 0066 7a32 b020 7a05 a31f 13c6 8e64 1f2e 0000070 8bf9 95fc dbad 06a6 305f 11ba 58a4 e409 0000080 5417 e768 5827 c54e 79d7 ef32 aa3f 8260 0000090 0d16 3236 c1a0 7789 799d a849 c3d3 ab70 4. check the vdb.qcow2's 'disk size' (105M now): [root@dell-per740xd-10 ~]# qemu-img info /var/lib/libvirt/images/vdb.qcow2 -U image: /var/lib/libvirt/images/vdb.qcow2 file format: qcow2 virtual size: 5 GiB (5368709120 bytes) disk size: 105 MiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false 5. do a full backup for vdb [root@dell-per740xd-10 ~]# cat backup.xml <domainbackup mode='push'> <disks> <disk name='vdb' type='file'> <target file='/home/images/vdb.full.backup'/> <driver type='qcow2'/> </disk> <disk name='vda' backup='no'/> </disks> </domainbackup> [root@dell-per740xd-10 ~]# cat checkpoint.xml <domaincheckpoint> <name>check_full</name> </domaincheckpoint> 6. check the backup image's 'disk size', which is 128M (>105M and <5G, acceptable): [root@dell-per740xd-10 ~]# qemu-img info /home/images/vdb.full.backup -U image: /home/images/vdb.full.backup file format: qcow2 virtual size: 5 GiB (5368709120 bytes) disk size: 128 MiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false 7. udr vdb.full.backup as vm's image and check its hex data, it's same as step 3. [root@dell-per740xd-10 ~]# virsh edit vm1 Domain vm1 XML configuration edited. [root@dell-per740xd-10 ~]# virsh destroy vm1 vDomain vm1 destroyed [root@dell-per740xd-10 ~]# virsh start vm1 Domain vm1 started [root@dell-per740xd-10 ~]# virsh domblklist vm1 Target Source -------------------------------------------------------- vda /var/lib/libvirt/images/jeos-27-x86_64.qcow2 vdb /home/images/vdb.full.backup [root@localhost ~]# mount /dev/vdb /mnt/ [ 21.611599] EXT4-fs (vdb): recovery complete [ 21.613673] EXT4-fs (vdb): mounted filesystem with ordered data mode. Opts: (null) [root@localhost ~]# ll /mnt/ total 102416 -rw-r--r--. 1 root root 104857600 Sep 16 11:33 100M drwx------. 2 root root 16384 Sep 16 11:33 lost+found [root@localhost ~]# hexdump /mnt/100M | head -10 0000000 25fc 1cf5 28bb 6dd5 2236 2493 c578 c7ff 0000010 45a9 95ba ea17 4222 7cc2 71ed 3c12 c2a5 0000020 4511 a292 79f0 2e64 a02f e771 3841 389c 0000030 9f00 3756 3206 1653 bba5 217e b857 efac 0000040 a474 81ff 014e 2028 580f 658f 7b3b f7a9 0000050 65f8 39fd 01b6 ac6c 1327 e738 d5cb c5ae 0000060 0066 7a32 b020 7a05 a31f 13c6 8e64 1f2e 0000070 8bf9 95fc dbad 06a6 305f 11ba 58a4 e409 0000080 5417 e768 5827 c54e 79d7 ef32 aa3f 8260 0000090 0d16 3236 c1a0 7789 799d a849 c3d3 ab70
*** This bug has been marked as a duplicate of bug 1802401 ***