Bug 1847090
| Summary: | [RFE] Support transferring snapshots using raw format (NBD backend) | ||
|---|---|---|---|
| Product: | [oVirt] ovirt-imageio | Reporter: | Nir Soffer <nsoffer> |
| Component: | Common | Assignee: | Nir Soffer <nsoffer> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Ilan Zuckerman <izuckerm> |
| Severity: | medium | Docs Contact: | |
| Priority: | high | ||
| Version: | 2.0.0 | CC: | bugs, mtessun, nsoffer, royoung, sfishbai, tnisan |
| Target Milestone: | ovirt-4.4.4 | Keywords: | FutureFeature, Reopened |
| Target Release: | --- | Flags: | pm-rhel:
ovirt-4.4+
izuckerm: testing_plan_complete+ mtessun: planning_ack+ pm-rhel: devel_ack+ pm-rhel: testing_ack+ |
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ovirt-engine-4.4.3.10 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-12-21 12:36:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I posted some of the changes in imageio. We need more work to add backing_chain argument in clinet.upload() and client.download(). We need also more changes in related components: - ovirt-engine-api-model (add backing_chain option to ImageTransfer) - ovirt-engine (use transfer.backing_chain when starting NBD server) - vdsm - support backing_chain argument in NBD.start_server Proposing for 4.4.2 since this is useful for backup vendors and relatively small change which is partly done. First imageio patches merged, but more work is needed, leaving in POST. See comment 1 for extra work needed. Moving to NEW to prevent future confusion. Bumping priority since this is needed for backup vendors backing up snapshots. Without this they cannot use imageio python library to download snapshots. New patches handle uploading snapshots using client.upload(). More work is needed for downloading snapshots: - api model: Add ImageTransfer.backing_chain - engine: when using: format="raw", backing_chain=False, The transfer will expose a single snapshot for reading, instead of the entire chain starting at the snapshot. - vdsm: add backing_chain argument to vdsm NBD.start_server. When backing_chain=False, the nbd server expose only the requested volume instead of the entire chain starting at the volume. With these changes, client.download() will fetch a single snapshot using the NBD backend. Tal, I suggest we schedule this to 4.4.3 since this can help backup vendors support for 4.4. Another task to complete this work - handle properly holes in imageio internal copy code. Currently when downloading images, we skip extents with zero=True leaving these areas unallocated. This is not correct when downloading snapshots. They may have zero extents that must hide data in previous layers. When downloading single layer we should: - zero extents with zero=True, hole=False - skip extents with zero=True, hole=True. Fixed in: - vdsm >= 4.40.32 - ovirt-engine >= 4.4.3.7 - ovirt-engine-sdk >= 4.4.5 - ovirt-imageio-client >= 2.1.0-1 Required versions are available in rhv-4.4.3-9. Nir, are the verification steps in comment #5 are still valid? If not, please update them. Thanks. (In reply to Ilan Zuckerman from comment #14) > Nir, are the verification steps in comment #5 are still valid? > If not, please update them. > Thanks. No, downlaod_disk_snaphosts.py needs a rewrite. Note that is was renamed to download_all_disk_snapshots.py. We have a new "downlaod_disk_snapshot.py": https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/download_disk_snapshot.py To verify this change, you can do: 1. Start a vm with one non-empty disk 2. Create snapshot 1 3. Created snapshot 2 4. List disk snapshots using list_disk_snapshots.py $ ./list_disk_snapshots.py -c engine-dev 4b62aa6d-3bdd-4db3-b26f-0484c4124631 [ { "actual_size": 1769902080, "format": "cow", "id": "20391fbc-fa77-4fef-9aea-cf59f27f90b5", "parent": null, "status": "ok" }, { "actual_size": 458752, "format": "cow", "id": "13ad63bf-71e2-4243-8967-ab4324818b31", "parent": "20391fbc-fa77-4fef-9aea-cf59f27f90b5", "status": "ok" }, { "actual_size": 4325376, "format": "cow", "id": "b5e56953-bf86-4496-a6bd-19132cba9763", "parent": "13ad63bf-71e2-4243-8967-ab4324818b31", "status": "ok" } ] 5. Download disk snapshot 20391fbc-fa77-4fef-9aea-cf59f27f90b5 $ ./download_disk_snapshot.py -c engine-dev nfs1 20391fbc-fa77-4fef-9aea-cf59f27f90b5 snap1.qcow2 [ 0.0 ] Connecting... [ 0.1 ] Creating image transfer... [ 1.4 ] Transfer ID: bc7e2379-08c1-4112-8628-b55ab2752c90 [ 1.4 ] Transfer host name: host3 [ 1.4 ] Downloading image... [ 100.00% ] 6.00 GiB, 5.19 seconds, 1.16 GiB/s [ 6.6 ] Finalizing image transfer... 6. Download disk snapshot 13ad63bf-71e2-4243-8967-ab4324818b31, rebasing it on top of snap1.qcow2 $ ./download_disk_snapshot.py -c engine-dev nfs1 13ad63bf-71e2-4243-8967-ab4324818b31 snap2.qcow2 --backing-file snap1.qcow2 [ 0.0 ] Connecting... [ 0.1 ] Creating image transfer... [ 1.3 ] Transfer ID: db912896-1e8f-4441-bbc8-4887168d07b9 [ 1.3 ] Transfer host name: host3 [ 1.3 ] Downloading image... [ 100.00% ] 6.00 GiB, 0.26 seconds, 22.80 GiB/s [ 1.6 ] Finalizing image transfer... This creates the chain: $ qemu-img info --backing-chain snap2.qcow2 image: snap2.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 4.82 MiB cluster_size: 65536 backing file: snap1.qcow2 backing file format: qcow2 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false image: snap1.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 1.65 GiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false Note that the top image is very small, it contains only the data written after the first snapshot was created, until the second snapshot was created. 7. To verify that the image downloaded as separate snapshots is correct, download the entire image at the same point in time by downloading disk snapshot without --backing-chain: $ ./download_disk_snapshot.py -c engine-dev nfs1 13ad63bf-71e2-4243-8967-ab4324818b31 snap2-full.qcow2 [ 0.0 ] Connecting... [ 0.1 ] Creating image transfer... [ 1.4 ] Transfer ID: 3874ae55-bb76-428a-999a-37ca0ea4749e [ 1.4 ] Transfer host name: host4 [ 1.4 ] Downloading image... [ 100.00% ] 6.00 GiB, 11.35 seconds, 541.45 MiB/s [ 12.7 ] Finalizing image transfer... This creates: $ qemu-img info --backing-chain snap2-full.qcow2 image: snap2-full.qcow2 file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 1.65 GiB cluster_size: 65536 Format specific information: compat: 1.1 compression type: zlib lazy refcounts: false refcount bits: 16 corrupt: false 8. Verify that both downloads have the same data $ qemu-img compare snap2.qcow2 snap2-full.qcow2 -s Images are identical. Note about comparing images: -s flag (strict mode) works only if the source base volume is also qcow2. If the base image is raw you need to compare without -s. To ensure a VM uses qcow2 disk on file based storage, select "enable incremental backup" when creating the disk, or created the VM from uploaded disk, where you can control the format of the disk. Verified on rhv-release-4.4.4-2
1. Start a vm with one non-empty disk
2. Create snapshot 1
3. Created snapshot 2
4. List disk snapshots using list_disk_snapshots.py
```
[root@storage-ge5-vdsm3 examples]# python3 list_disk_snapshots.py -c engine dfad7cbe-f932-4bbf-b62f-338102510b37
[
{
"actual_size": 10871635968,
"format": "cow",
"id": "a724c554-79b9-47f8-8537-6cad2c56dc1f",
"parent": null,
"status": "ok"
},
{
"actual_size": 1073741824,
"format": "cow",
"id": "ddd9c387-2770-4522-be62-7b8449f8ed86",
"parent": "a724c554-79b9-47f8-8537-6cad2c56dc1f",
"status": "ok"
},
{
"actual_size": 1073741824,
"format": "cow",
"id": "5a7ef718-976d-4c8e-a2de-e3796af89e92",
"parent": "ddd9c387-2770-4522-be62-7b8449f8ed86",
"status": "ok"
}
]
```
5. Download disk snapshot a724c554-79b9-47f8-8537-6cad2c56dc1f
```
[root@storage-ge5-vdsm3 examples]# python3 download_disk_snapshot.py -c engine iscsi_0 a724c554-79b9-47f8-8537-6cad2c56dc1f snap1.qcow2
[ 0.0 ] Connecting...
[ 0.3 ] Creating image transfer...
[ 3.6 ] Transfer ID: 484127d4-8523-4283-b12c-d13848e08689
[ 3.6 ] Transfer host name: host_mixed_3
[ 3.6 ] Downloading image...
[ 100.00% ] 10.00 GiB, 15.78 seconds, 648.74 MiB/s
[ 19.4 ] Finalizing image transfer...
```
6. Download disk snapshot 13ad63bf-71e2-4243-8967-ab4324818b31, rebasing
it on top of snap1.qcow2
```
[root@storage-ge5-vdsm3 examples]# python3 download_disk_snapshot.py -c engine iscsi_0 ddd9c387-2770-4522-be62-7b8449f8ed86 snap2.qcow2 --backing-file snap1.qcow2
[ 0.0 ] Connecting...
[ 0.3 ] Creating image transfer...
[ 3.5 ] Transfer ID: c8a6bded-e31c-4cce-a9a0-1db48134a452
[ 3.5 ] Transfer host name: host_mixed_3
[ 3.5 ] Downloading image...
[ 100.00% ] 10.00 GiB, 0.13 seconds, 74.58 GiB/s
[ 3.6 ] Finalizing image transfer...
```
This creates the chain:
```
[root@storage-ge5-vdsm3 examples]# qemu-img info --backing-chain snap2.qcow2
image: snap2.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 704 KiB
cluster_size: 65536
backing file: snap1.qcow2
backing file format: qcow2
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
image: snap1.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.22 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
```
7. To verify that the image downloaded as separate snapshots is correct,
download the entire image at the same point in time by downloading
disk snapshot without --backing-chain:
```
[root@storage-ge5-vdsm3 examples]# python3 download_disk_snapshot.py -c engine iscsi_0 ddd9c387-2770-4522-be62-7b8449f8ed86 snap2-full.qcow2
[ 0.0 ] Connecting...
[ 0.3 ] Creating image transfer...
[ 3.6 ] Transfer ID: c6628315-3bba-4add-8720-61ee48cfa0c9
[ 3.6 ] Transfer host name: host_mixed_3
[ 3.6 ] Downloading image...
[ 100.00% ] 10.00 GiB, 15.41 seconds, 664.44 MiB/s
[ 19.0 ] Finalizing image transfer...
```
This created:
```
[root@storage-ge5-vdsm3 examples]# qemu-img info --backing-chain snap2-full.qcow2
image: snap2-full.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.22 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
```
8. Verify that both downloads have the same data
```
[root@storage-ge5-vdsm3 examples]# qemu-img compare snap2.qcow2 snap2-full.qcow2 -s
Images are identical.
```
This bugzilla is included in oVirt 4.4.4 release, published on December 21st 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.4 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report. |
Description of problem: oVirt uses 2 backends for image transfers: 1. file Write user data as is to storage on upload, read disk data as is on download. When uploading qcow2 image, the user header overwrites the qcow2 header created by oVirt. The file backend is the only option in 4.4 to upload or download single snapshot. oVirt verifies that qcow2 header uploaded by the user matches the qcow2 header created by oVirt, but this validation is the wrong approach and should be avoided when possible. The right way is to store user data in the image as done by the nbd backend, keeping the qcow2 header created by oVirt. 2. nbd Upload and download always use raw guest data. write user data to disk using the underlying disk format (raw or qcow2) on upload. Read raw guest data from disk using the underlying disk format, and return raw guest data. Add support for uploading and downloading snapshot using the nbd backend. Download snapshot flow: 1. User creates qcow2 image on local host, with backing file if this is a snapshot. 2. User start image transfer using format=raw, backing_chain=False 3. Vdsm starts qemu-nbd with backing file disabled: qemu-nbd ... 'json:{"driver": "qcow2", "file": {"driver":"file","filename":"file.qcow2"},"backing": null}' 4. User download snapshot using imageio client, specifying backing_chain=False 5. imageio client download extents using this logic: - "zero": false -> write to qcow2 image - "zero": true, "hole": false -> zero area in qcow2 image - "zero": true, "hole": true -> skip, leave area unallocated Upload snapshot flow: 1. User creates a new VM and upload base image normally 2. User creates a new snapshot using oVirt API 3. User creates image transfer using raw format for snapshot image 4. User uploads snapshot data using imageio client, specifying backing_chain=False 5. imageio client run qemu-nbd with backing disabled: qemu-nbd ... 'json:{"driver": "qcow2", "file": {"driver":"file","filename":"file.qcow2"},"backing": null}' 6. imageio client upload extents using this logic: - "zero": false -> write to server - "zero": true, "hole": false -> zero area on server - "zero": true, "hole": true -> skip, leave area unallocated 7. When upload is done, engine can skip image verification since qcow2 header can not be modified by the client. Changes needed: - Report holes in extents API - Add backing_chain option to imageio client upload/download - Add backing_chain option to ImageTransfer API - Add backing_chain option to vdsm NBD.start_server - Update download_disk_snapshots.py and upload_disk_snapshots.py examples to use imageio client instead of outdated HTTP code. Reporting extents for a single snapshot was requested in the past by backup vendors, and may help vendors to perform backup in more efficient way when using backup appliance. This change does not eliminate the need for the file backend when uploading images from the UI. To eliminate this we need a qcow2 parser implemented in javascript.