Bug 1913315
| Summary: | incremental-backup: RFE: Allow monitoring backup scratch disk block threshold | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux Advanced Virtualization | Reporter: | Nir Soffer <nsoffer> |
| Component: | libvirt | Assignee: | Peter Krempa <pkrempa> |
| Status: | CLOSED ERRATA | QA Contact: | yisun |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 8.4 | CC: | aliang, coli, jdenemar, ngu, pkrempa, virt-maint, xuzhang, yafu, yalzhang |
| Target Milestone: | rc | Keywords: | FutureFeature |
| Target Release: | 8.4 | Flags: | pm-rhel:
mirror+
|
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-7.0.0-1.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-05-25 06:46:31 UTC | Type: | Feature Request |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | 7.0.0 |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1913387 | ||
Bug 1874483 explains why RHV switches to shared storage for scratch disks. We plan to implemnt this for RHV 4.4.5 which will be shipped before RHEL 8.4. If this bug will be fixed in 8.4, we may be able to use this in RHV 4.4.6 which will require RHEL 8.4. It's now possible in upstream libvirt. The index is exposed in the backup job XML:
$ virsh backup-dumpxml backup-test
<domainbackup mode='pull'>
<server transport='tcp' name='localhost' port='1234'/>
<disks>
<disk name='vda' backup='yes' type='file' backupmode='full' exportname='vda' index='4'>
<driver type='qcow2'/>
<scratch file='/tmp/backup-test-images/scratch-vda.qcow2'/>
</disk>
<disk name='hda' backup='no'/>
</disks>
</domainbackup>
$ virsh domblkthreshold backup-test 'vda[4]' 123
$virsh event --loop --all
event 'block-threshold' for domain backup-test: dev: vda[4](/tmp/backup-test-images/scratch-vda.qcow2) 123 196485
^C
Following commits implement the requested functionality:
commit e3922af17c9fe0a699fcc98ad7ca8600ba32ecdc
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 12:38:50 2020 +0100
conf: backup: Format index of 'store'
Similarly to other disk-related stuff, the index is useful when you want
to refer to the image in APIs such as virDomainSetBlockThreshold.
For internal use we also need to parse it inside of the status XML.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit 40242b7452b99857a4c79553a481f9d70ad6da9b
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 12:38:43 2020 +0100
qemuDomainGetStorageSourceByDevstr: Lookup also backup 'store' nodenames
Nodename may be asociated to a disk backup job, add support to looking
up in that chain too. This is specifically useful for the
BLOCK_WRITE_THRESHOLD event which can be registered for any nodename.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit c1720b9ac752cd586fe63cefbd01a4f3ced78385
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 12:38:43 2020 +0100
qemuDomainDiskLookupByNodename: Lookup also backup 'store' nodenames
Nodename may be asociated to a disk backup job, add support to looking
up in that chain too. This is specifically useful for the
BLOCK_WRITE_THRESHOLD event which can be registered for any nodename.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit a0a2eb12abc5ba5b5c324042d28af48184af81ea
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 13:19:18 2020 +0100
qemuDomainGetStorageSourceByDevstr: Avoid logged errors
'virStorageFileChainLookup' reports an error when the lookup of the
backing chain entry is unsuccessful. Since we possibly use it multiple
times when looking up backing for 'disk->mirror' the function can report
error which won't be actually reported.
Replace the call to virStorageFileChainLookup by lookup in the chain by
index.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit 047b45f35915b4a92f0b1fce4638430092999d03
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 14:03:22 2020 +0100
virDomainBackupDiskDefParseXML: Use virDomainStorageSourceParseBase
Don't duplicate code to parse the virStorageSource basics.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit d46512fc954ef9091c485b001af50860d42297b7
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 17:05:40 2020 +0100
backup: Move file format check from parser to qemu driver
It's a technical detail in qemu that QCOW2 is needed for a pull-mode
backup.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit a0a2eb12abc5ba5b5c324042d28af48184af81ea
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 13:19:18 2020 +0100
qemuDomainGetStorageSourceByDevstr: Avoid logged errors
'virStorageFileChainLookup' reports an error when the lookup of the
backing chain entry is unsuccessful. Since we possibly use it multiple
times when looking up backing for 'disk->mirror' the function can report
error which won't be actually reported.
Replace the call to virStorageFileChainLookup by lookup in the chain by
index.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit 4c4c07b94169ad52f6fc3378f397528a698cf5cf
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 13:04:29 2020 +0100
qemuDomainGetStorageSourceByDevstr: Use virDomainDiskByTarget
The function replaces the open-coded block.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
commit c3bb2b2d5d0098b556a88b696540f1a65641a7c2
Author: Peter Krempa <pkrempa>
Date: Mon Dec 7 12:24:31 2020 +0100
qemuDomainDiskLookupByNodename: Simplify node name lookup
Use dummy variable to fill 'src' so that access to it doesn't need to be
conditionalized and use temporary variable for 'disk' rather than
dereferencing the array multiple times.
Signed-off-by: Peter Krempa <pkrempa>
Reviewed-by: Ján Tomko <jtomko>
v6.10.0-135-ge3922af17c
PreVerified on upstream libvirt-6.10.0-1.fc34.x86_64, result is PASS
Steps:
1. have a running vm with 2 virtual disks [vda, vdb]
➜ fedora virsh domstate vm1
running
➜ fedora virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/vda.qcow2' index='2'/>
<backingStore/>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/vdb.qcow2' index='1'/>
<backingStore/>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</disk>
2. create diskonly snapshot for vm1
➜ fedora virsh snapshot-create-as vm1 snap1 --disk-only
Domain snapshot snap1 created
➜ fedora virsh dumpxml vm1 | awk '/<disk/,/<\/disk/'
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/vda.snap1' index='3'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/vda.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/vdb.snap1' index='4'/>
<backingStore type='file' index='1'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/vdb.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</disk>
3. in vm, dd some data in vda and vdb
[root@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=256; sync
256+0 records in
256+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 7.61955 s, 35.2 MB/s
[root@localhost ~]# dd if=/dev/urandom of=/tmp/file_on_vda bs=1M count=256; sync
256+0 records in
256+0 records out
268435456 bytes (268 MB, 256 MiB) copied, 6.92628 s, 38.8 MB/s
4. prepare backup xml
➜ fedora cat backup.xml
<domainbackup mode='pull'>
<server transport='tcp' name='localhost' port='1234'/>
<disks>
<disk name='vda' backup='yes' type='file' backupmode='full' exportname='vda'>
<driver type='qcow2'/>
<scratch file='/tmp/scratch-vda.qcow2'/>
</disk>
<disk name='vdb' backup='yes' type='file' backupmode='full' exportname='vdb'>
<driver type='qcow2'/>
<scratch file='/tmp/scratch-vdb.qcow2'/>
</disk>
</disks>
</domainbackup>
➜ fedora cat checkpoint.xml
<domaincheckpoint>
<name>checkpoint_0</name>
<description>desc of cp_0</description>
<disks>
<disk checkpoint="bitmap" name="vda" />
<disk checkpoint="bitmap" name="vdb" />
</disks>
</domaincheckpoint>
5. start pull mode backup
➜ fedora virsh backup-begin vm1 backup.xml checkpoint.xml
Backup started
6. get the block-dev index of [vda, vdb]
➜ fedora virsh backup-dumpxml vm1
<domainbackup mode='pull'>
<server transport='tcp' name='localhost' port='1234'/>
<disks>
<disk name='vda' backup='yes' type='file' backupmode='full' exportname='vda' index='7'>
<driver type='qcow2'/>
<scratch file='/tmp/scratch-vda.qcow2'/>
</disk>
<disk name='vdb' backup='yes' type='file' backupmode='full' exportname='vdb' index='8'>
<driver type='qcow2'/>
<scratch file='/tmp/scratch-vdb.qcow2'/>
</disk>
</disks>
</domainbackup>
7. set domblkthreshold for vda[7] and vdb[8]
➜ fedora virsh domblkthreshold vm1 'vda[7]' 222
➜ fedora virsh domblkthreshold vm1 'vdb[8]' 333
8. start to monitor the threshold in another termimal
➜ fedora virsh event --loop --all
9. dd some data to vda, check the event triggered
[root@localhost ~]# dd if=/dev/urandom of=/tmp/file_on_vda bs=1M count=256; sync
333+0 records in~]# dd if=/dev/urandom of=/tmp/file_on_vda bs=1M count=333; sync
333+0 records out
349175808 bytes (349 MB, 333 MiB) copied, 4.12547 s, 84.6 MB/s
➜ fedora virsh event --loop --all
event 'block-threshold' for domain 'vm1': dev: vda[7](/tmp/scratch-vda.qcow2) 222 196386
10. dd some data to vdb, check the event triggered
56oot@localhost ~]# dd if=/dev/urandom of=/dev/vdb bs=1M count=444; sync
444+0 records in
444+0 records out
465567744 bytes (466 MB, 444 MiB) copied, 7.33163 s, 63.5 MB/s
➜ fedora virsh event --loop --all
event 'block-threshold' for domain 'vm1': dev: vda[7](/tmp/scratch-vda.qcow2) 222 196386
event 'block-threshold' for domain 'vm1': dev: vdb[8](/tmp/scratch-vdb.qcow2) 333 196275
Verified on: libvirt-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098 |
Description of problem: RHV is switching scratch disks on shared storage (NFS, GlusterFs, iSCSI, FC). To use scratch disk on block storage (LVM), RHV needs to create large scratch disk that can hold maximum all the data referenced by the backup dirty bitmap or worse when backing up raw volumes, in the entire disk size. This make it harder to create the scratch disk on the best storage, since it may not have enough space, or may actually fail the backup since no storage has enough space. RHV wants to use the same mechanism, used for qcow2 disks on block storage, create a small initial disk, and extend it as needed. For this RHV uses virDomainSetBlockThreshold to set a threshold, and extend the disk when it reaches the threshold. RHV cannot use this mechanism with current libvirt since the scratch disks are not exposed in the xml. I think this can be solved by exposing the scratch disks in the disk backing chain or or otherwise (maybe as separate element?), and assigning and index to the scratch disk, so RHV can use virDomainSetBlockThreshold("vda[4]", ...). This must work for file and block storage. network disk types were never fully supported in RHV. We had ceph network disk as tech preview, which was replaced by kernel rbd support, and Gluster libgafapi, which was never fully supported in libvirt, and I think it deprecated in gluster. Supporting network disk is nice to have feature.