Description of problem: RHV is switching scratch disks on shared storage (NFS, GlusterFs, iSCSI, FC). To use scratch disk on block storage (LVM), RHV needs to create large scratch disk that can hold maximum all the data referenced by the backup dirty bitmap or worse when backing up raw volumes, in the entire disk size. This make it harder to create the scratch disk on the best storage, since it may not have enough space, or may actually fail the backup since no storage has enough space. RHV wants to use the same mechanism, used for qcow2 disks on block storage, create a small initial disk, and extend it as needed. For this RHV uses virDomainSetBlockThreshold to set a threshold, and extend the disk when it reaches the threshold. RHV cannot use this mechanism with current libvirt since the scratch disks are not exposed in the xml. I think this can be solved by exposing the scratch disks in the disk backing chain or or otherwise (maybe as separate element?), and assigning and index to the scratch disk, so RHV can use virDomainSetBlockThreshold("vda[4]", ...). This must work for file and block storage. network disk types were never fully supported in RHV. We had ceph network disk as tech preview, which was replaced by kernel rbd support, and Gluster libgafapi, which was never fully supported in libvirt, and I think it deprecated in gluster. Supporting network disk is nice to have feature.
Bug 1874483 explains why RHV switches to shared storage for scratch disks. We plan to implemnt this for RHV 4.4.5 which will be shipped before RHEL 8.4. If this bug will be fixed in 8.4, we may be able to use this in RHV 4.4.6 which will require RHEL 8.4.
It's now possible in upstream libvirt. The index is exposed in the backup job XML: $ virsh backup-dumpxml backup-test <domainbackup mode='pull'> <server transport='tcp' name='localhost' port='1234'/> <disks> <disk name='vda' backup='yes' type='file' backupmode='full' exportname='vda' index='4'> <driver type='qcow2'/> <scratch file='/tmp/backup-test-images/scratch-vda.qcow2'/> </disk> <disk name='hda' backup='no'/> </disks> </domainbackup> $ virsh domblkthreshold backup-test 'vda[4]' 123 $virsh event --loop --all event 'block-threshold' for domain backup-test: dev: vda[4](/tmp/backup-test-images/scratch-vda.qcow2) 123 196485 ^C Following commits implement the requested functionality: commit e3922af17c9fe0a699fcc98ad7ca8600ba32ecdc Author: Peter Krempa <pkrempa> Date: Mon Dec 7 12:38:50 2020 +0100 conf: backup: Format index of 'store' Similarly to other disk-related stuff, the index is useful when you want to refer to the image in APIs such as virDomainSetBlockThreshold. For internal use we also need to parse it inside of the status XML. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit 40242b7452b99857a4c79553a481f9d70ad6da9b Author: Peter Krempa <pkrempa> Date: Mon Dec 7 12:38:43 2020 +0100 qemuDomainGetStorageSourceByDevstr: Lookup also backup 'store' nodenames Nodename may be asociated to a disk backup job, add support to looking up in that chain too. This is specifically useful for the BLOCK_WRITE_THRESHOLD event which can be registered for any nodename. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit c1720b9ac752cd586fe63cefbd01a4f3ced78385 Author: Peter Krempa <pkrempa> Date: Mon Dec 7 12:38:43 2020 +0100 qemuDomainDiskLookupByNodename: Lookup also backup 'store' nodenames Nodename may be asociated to a disk backup job, add support to looking up in that chain too. This is specifically useful for the BLOCK_WRITE_THRESHOLD event which can be registered for any nodename. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit a0a2eb12abc5ba5b5c324042d28af48184af81ea Author: Peter Krempa <pkrempa> Date: Mon Dec 7 13:19:18 2020 +0100 qemuDomainGetStorageSourceByDevstr: Avoid logged errors 'virStorageFileChainLookup' reports an error when the lookup of the backing chain entry is unsuccessful. Since we possibly use it multiple times when looking up backing for 'disk->mirror' the function can report error which won't be actually reported. Replace the call to virStorageFileChainLookup by lookup in the chain by index. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit 047b45f35915b4a92f0b1fce4638430092999d03 Author: Peter Krempa <pkrempa> Date: Mon Dec 7 14:03:22 2020 +0100 virDomainBackupDiskDefParseXML: Use virDomainStorageSourceParseBase Don't duplicate code to parse the virStorageSource basics. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit d46512fc954ef9091c485b001af50860d42297b7 Author: Peter Krempa <pkrempa> Date: Mon Dec 7 17:05:40 2020 +0100 backup: Move file format check from parser to qemu driver It's a technical detail in qemu that QCOW2 is needed for a pull-mode backup. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit a0a2eb12abc5ba5b5c324042d28af48184af81ea Author: Peter Krempa <pkrempa> Date: Mon Dec 7 13:19:18 2020 +0100 qemuDomainGetStorageSourceByDevstr: Avoid logged errors 'virStorageFileChainLookup' reports an error when the lookup of the backing chain entry is unsuccessful. Since we possibly use it multiple times when looking up backing for 'disk->mirror' the function can report error which won't be actually reported. Replace the call to virStorageFileChainLookup by lookup in the chain by index. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit 4c4c07b94169ad52f6fc3378f397528a698cf5cf Author: Peter Krempa <pkrempa> Date: Mon Dec 7 13:04:29 2020 +0100 qemuDomainGetStorageSourceByDevstr: Use virDomainDiskByTarget The function replaces the open-coded block. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> commit c3bb2b2d5d0098b556a88b696540f1a65641a7c2 Author: Peter Krempa <pkrempa> Date: Mon Dec 7 12:24:31 2020 +0100 qemuDomainDiskLookupByNodename: Simplify node name lookup Use dummy variable to fill 'src' so that access to it doesn't need to be conditionalized and use temporary variable for 'disk' rather than dereferencing the array multiple times. Signed-off-by: Peter Krempa <pkrempa> Reviewed-by: Ján Tomko <jtomko> v6.10.0-135-ge3922af17c
PreVerified on upstream libvirt-6.10.0-1.fc34.x86_64, result is PASS Steps: 1. have a running vm with 2 virtual disks [vda, vdb] ➜ fedora virsh domstate vm1 running ➜ fedora virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vda.qcow2' index='2'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vdb.qcow2' index='1'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> 2. create diskonly snapshot for vm1 ➜ fedora virsh snapshot-create-as vm1 snap1 --disk-only Domain snapshot snap1 created ➜ fedora virsh dumpxml vm1 | awk '/<disk/,/<\/disk/' <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vda.snap1' index='3'/> <backingStore type='file' index='2'> <format type='qcow2'/> <source file='/var/lib/libvirt/images/vda.qcow2'/> <backingStore/> </backingStore> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/vdb.snap1' index='4'/> <backingStore type='file' index='1'> <format type='qcow2'/> <source file='/var/lib/libvirt/images/vdb.qcow2'/> <backingStore/> </backingStore> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/> </disk> 3. in vm, dd some data in vda and vdb [root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=256; sync 256+0 records in 256+0 records out 268435456 bytes (268 MB, 256 MiB) copied, 7.61955 s, 35.2 MB/s [root@localhost ~]# dd if=/dev/urandom of=/tmp/file_on_vda bs=1M count=256; sync 256+0 records in 256+0 records out 268435456 bytes (268 MB, 256 MiB) copied, 6.92628 s, 38.8 MB/s 4. prepare backup xml ➜ fedora cat backup.xml <domainbackup mode='pull'> <server transport='tcp' name='localhost' port='1234'/> <disks> <disk name='vda' backup='yes' type='file' backupmode='full' exportname='vda'> <driver type='qcow2'/> <scratch file='/tmp/scratch-vda.qcow2'/> </disk> <disk name='vdb' backup='yes' type='file' backupmode='full' exportname='vdb'> <driver type='qcow2'/> <scratch file='/tmp/scratch-vdb.qcow2'/> </disk> </disks> </domainbackup> ➜ fedora cat checkpoint.xml <domaincheckpoint> <name>checkpoint_0</name> <description>desc of cp_0</description> <disks> <disk checkpoint="bitmap" name="vda" /> <disk checkpoint="bitmap" name="vdb" /> </disks> </domaincheckpoint> 5. start pull mode backup ➜ fedora virsh backup-begin vm1 backup.xml checkpoint.xml Backup started 6. get the block-dev index of [vda, vdb] ➜ fedora virsh backup-dumpxml vm1 <domainbackup mode='pull'> <server transport='tcp' name='localhost' port='1234'/> <disks> <disk name='vda' backup='yes' type='file' backupmode='full' exportname='vda' index='7'> <driver type='qcow2'/> <scratch file='/tmp/scratch-vda.qcow2'/> </disk> <disk name='vdb' backup='yes' type='file' backupmode='full' exportname='vdb' index='8'> <driver type='qcow2'/> <scratch file='/tmp/scratch-vdb.qcow2'/> </disk> </disks> </domainbackup> 7. set domblkthreshold for vda[7] and vdb[8] ➜ fedora virsh domblkthreshold vm1 'vda[7]' 222 ➜ fedora virsh domblkthreshold vm1 'vdb[8]' 333 8. start to monitor the threshold in another termimal ➜ fedora virsh event --loop --all 9. dd some data to vda, check the event triggered [root@localhost ~]# dd if=/dev/urandom of=/tmp/file_on_vda bs=1M count=256; sync 333+0 records in~]# dd if=/dev/urandom of=/tmp/file_on_vda bs=1M count=333; sync 333+0 records out 349175808 bytes (349 MB, 333 MiB) copied, 4.12547 s, 84.6 MB/s ➜ fedora virsh event --loop --all event 'block-threshold' for domain 'vm1': dev: vda[7](/tmp/scratch-vda.qcow2) 222 196386 10. dd some data to vdb, check the event triggered 56oot@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=444; sync 444+0 records in 444+0 records out 465567744 bytes (466 MB, 444 MiB) copied, 7.33163 s, 63.5 MB/s ➜ fedora virsh event --loop --all event 'block-threshold' for domain 'vm1': dev: vda[7](/tmp/scratch-vda.qcow2) 222 196386 event 'block-threshold' for domain 'vm1': dev: vdb[8](/tmp/scratch-vdb.qcow2) 333 196275
Verified on: libvirt-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098