Bug 1847090 - [RFE] Support transferring snapshots using raw format (NBD backend)
Summary: [RFE] Support transferring snapshots using raw format (NBD backend)
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-imageio
Classification: oVirt
Component: Common
Version: 2.0.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ovirt-4.4.4
: ---
Assignee: Nir Soffer
QA Contact: Ilan Zuckerman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-15 15:55 UTC by Nir Soffer
Modified: 2020-12-21 12:36 UTC (History)
6 users (show)

Fixed In Version: ovirt-engine-4.4.3.10
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-12-21 12:36:05 UTC
oVirt Team: Storage
Embargoed:
pm-rhel: ovirt-4.4+
izuckerm: testing_plan_complete+
mtessun: planning_ack+
pm-rhel: devel_ack+
pm-rhel: testing_ack+

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 109579 0 master MERGED qemu_nbd: Add backing_chain argument 2021-02-02 10:46:16 UTC
oVirt gerrit 109580 0 master MERGED nbd: Add Extent.hole property 2021-02-02 10:46:16 UTC
oVirt gerrit 109581 0 master MERGED backends: image: Add *Extent.to_dict() *Extent.from_dict() 2021-02-02 10:46:16 UTC
oVirt gerrit 109582 0 master MERGED extents: Report holes in zero extents 2021-02-02 10:46:16 UTC
oVirt gerrit 111310 0 master MERGED tests: Prepare for testing with nbd backend 2021-02-02 10:46:16 UTC
oVirt gerrit 111311 0 master MERGED client: Support upload snapshot using NBD backend 2021-02-02 10:46:17 UTC
oVirt gerrit 111312 0 master MERGED ImageTransfer: Support shallow download 2021-02-02 10:46:17 UTC
oVirt gerrit 111324 0 master MERGED nbd: Allow exporting a volume without the backing chain 2021-02-02 10:46:17 UTC
oVirt gerrit 111339 0 master MERGED core: Support shallow download 2021-02-02 10:46:17 UTC
oVirt gerrit 111426 0 master MERGED imagetransfer: Use disk= instead of deprecated image= 2021-02-02 10:46:17 UTC
oVirt gerrit 111428 0 master MERGED imagetransfer: Support download of disk snapshots 2021-02-02 10:46:17 UTC
oVirt gerrit 111429 0 master ABANDONED download_disk: Support shallow download 2021-02-02 10:46:18 UTC
oVirt gerrit 111460 0 master MERGED imagetransfer: Support shallow transfer 2021-02-02 10:46:19 UTC
oVirt gerrit 111461 0 master MERGED examples: Show how to download a disk snapshot 2021-02-02 10:46:19 UTC
oVirt gerrit 111463 0 master MERGED core: Add option to list active disk snapshots 2021-02-02 10:46:19 UTC
oVirt gerrit 111464 0 master MERGED snapshots: Add option to include active snapshots 2021-02-02 10:46:19 UTC
oVirt gerrit 111465 0 master MERGED examples: Show how to list disk snapshots 2021-02-02 10:46:19 UTC
oVirt gerrit 111476 0 master MERGED qemu_img: Add backing_format option 2021-02-02 10:46:20 UTC
oVirt gerrit 111477 0 master MERGED client: Support download with a backing file 2021-02-02 10:46:20 UTC
oVirt gerrit 111478 0 master MERGED client: Fix hole handling 2021-02-02 10:46:20 UTC
oVirt gerrit 111479 0 master MERGED DiskSnapshot: Add disk snapshot parent link 2021-02-02 10:46:20 UTC
oVirt gerrit 111480 0 master MERGED core: Expose disk snapshot parent 2021-02-02 10:46:20 UTC
oVirt gerrit 111486 0 master ABANDONED RFC: API: Allow client control for hole punching 2021-02-02 10:46:20 UTC
oVirt gerrit 111529 0 master MERGED Upgrade to model 4.4.19, Metamodel 1.3.3 2021-02-02 10:46:21 UTC
oVirt gerrit 111538 0 master MERGED tests: Cleanup io tests 2021-02-02 10:46:21 UTC
oVirt gerrit 111539 0 master MERGED tests: Add failing tests for handling holes 2021-02-02 10:46:21 UTC

Description Nir Soffer 2020-06-15 15:55:33 UTC 
Description of problem:

oVirt uses 2 backends for image transfers:

1. file

Write user data as is to storage on upload, read disk data as is on
download. When uploading qcow2 image, the user header overwrites the
qcow2 header created by oVirt.

The file backend is the only option in 4.4 to upload or download single 
snapshot.

oVirt verifies that qcow2 header uploaded by the user matches the qcow2
header created by oVirt, but this validation is the wrong approach and
should be avoided when possible. The right way is to store user data
in the image as done by the nbd backend, keeping the qcow2 header
created by oVirt.

2. nbd

Upload and download always use raw guest data. write user data to disk
using the underlying disk format (raw or qcow2) on upload. Read raw guest
data from disk using the underlying disk format, and return raw guest
data.

Add support for uploading and downloading snapshot using the nbd backend.

Download snapshot flow:

1. User creates qcow2 image on local host, with backing file if this is
   a snapshot.

2. User start image transfer using format=raw, backing_chain=False

3. Vdsm starts qemu-nbd with backing file disabled:

    qemu-nbd ... 'json:{"driver": "qcow2", "file": {"driver":"file","filename":"file.qcow2"},"backing": null}'

4. User download snapshot using imageio client, specifying backing_chain=False

5. imageio client download extents using this logic:
   - "zero": false -> write to qcow2 image
   - "zero": true, "hole": false -> zero area in qcow2 image
   - "zero": true, "hole": true -> skip, leave area unallocated

Upload snapshot flow:

1. User creates a new VM and upload base image normally

2. User creates a new snapshot using oVirt API

3. User creates image transfer using raw format for snapshot image

4. User uploads snapshot data using imageio client, specifying
   backing_chain=False

5. imageio client run qemu-nbd with backing disabled:

    qemu-nbd ... 'json:{"driver": "qcow2", "file": {"driver":"file","filename":"file.qcow2"},"backing": null}'

6. imageio client upload extents using this logic:
   - "zero": false -> write to server
   - "zero": true, "hole": false -> zero area on server
   - "zero": true, "hole": true -> skip, leave area unallocated

7. When upload is done, engine can skip image verification
   since qcow2 header can not be modified by the client.


Changes needed:

- Report holes in extents API
- Add backing_chain option to imageio client upload/download
- Add backing_chain option to ImageTransfer API
- Add backing_chain option to vdsm NBD.start_server
- Update download_disk_snapshots.py and upload_disk_snapshots.py
  examples to use imageio client instead of outdated HTTP code.

Reporting extents for a single snapshot was requested in the past by
backup vendors, and may help vendors to perform backup in more
efficient way when using backup appliance.

This change does not eliminate the need for the file backend when
uploading images from the UI. To eliminate this we need a qcow2
parser implemented in javascript.
Comment 1 Nir Soffer 2020-06-15 16:09:50 UTC 
I posted some of the changes in imageio. We need more work to 
add backing_chain argument in clinet.upload() and client.download().

We need also more changes in related components:
- ovirt-engine-api-model (add backing_chain option to ImageTransfer)
- ovirt-engine (use transfer.backing_chain when starting NBD server)
- vdsm - support backing_chain argument in NBD.start_server
Comment 2 Nir Soffer 2020-06-17 12:23:52 UTC 
Proposing for 4.4.2 since this is useful for backup vendors and relatively
small change which is partly done.
Comment 3 Nir Soffer 2020-06-17 15:15:19 UTC 
First imageio patches merged, but more work is needed,
leaving in POST.
Comment 6 Nir Soffer 2020-07-28 18:16:52 UTC 
See comment 1 for extra work needed. Moving to NEW to prevent future confusion.
Comment 7 Nir Soffer 2020-08-18 19:32:59 UTC 
Bumping priority since this is needed for backup vendors backing up snapshots.
Without this they cannot use imageio python library to download snapshots.
Comment 8 Nir Soffer 2020-09-15 21:57:38 UTC 
New patches handle uploading snapshots using client.upload().

More work is needed for downloading snapshots:

- api model: Add ImageTransfer.backing_chain

- engine: when using:

   format="raw",
   backing_chain=False,

The transfer will expose a single snapshot for reading, instead of 
the entire chain starting at the snapshot.

- vdsm: add backing_chain argument to vdsm NBD.start_server.

When backing_chain=False, the nbd server expose only the requested volume
instead of the entire chain starting at the volume.

With these changes, client.download() will fetch a single snapshot
using the NBD backend.
Comment 9 Nir Soffer 2020-09-15 22:05:01 UTC 
Tal, I suggest we schedule this to 4.4.3 since this can help
backup vendors support for 4.4.
Comment 10 Nir Soffer 2020-09-16 19:33:19 UTC 
Another task to complete this work - handle properly holes
in imageio internal copy code.

Currently when downloading images, we skip extents with zero=True
leaving these areas unallocated. This is not correct when downloading
snapshots. They may have zero extents that must hide data in previous
layers.

When downloading single layer we should:
- zero extents with zero=True, hole=False
- skip extents with zero=True, hole=True.
Comment 11 Nir Soffer 2020-10-14 20:39:35 UTC 
Fixed in:
- vdsm >= 4.40.32
- ovirt-engine >= 4.4.3.7
- ovirt-engine-sdk >= 4.4.5
- ovirt-imageio-client >= 2.1.0-1
Comment 12 Nir Soffer 2020-10-15 16:31:25 UTC 
Required versions are available in rhv-4.4.3-9.
Comment 14 Ilan Zuckerman 2020-10-29 08:23:17 UTC 
Nir, are the verification steps in comment #5 are still valid?
If not, please update them.
Thanks.
Comment 16 Nir Soffer 2020-10-29 12:15:04 UTC 
(In reply to Ilan Zuckerman from comment #14)
> Nir, are the verification steps in comment #5 are still valid?
> If not, please update them.
> Thanks.

No, downlaod_disk_snaphosts.py needs a rewrite. Note that is was renamed to
download_all_disk_snapshots.py.

We have a new "downlaod_disk_snapshot.py":
https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/download_disk_snapshot.py

To verify this change, you can do:

1. Start a vm with one non-empty disk
2. Create snapshot 1
3. Created snapshot 2
4. List disk snapshots using list_disk_snapshots.py

$ ./list_disk_snapshots.py -c engine-dev 4b62aa6d-3bdd-4db3-b26f-0484c4124631
[
  {
    "actual_size": 1769902080,
    "format": "cow",
    "id": "20391fbc-fa77-4fef-9aea-cf59f27f90b5",
    "parent": null,
    "status": "ok"
  },
  {
    "actual_size": 458752,
    "format": "cow",
    "id": "13ad63bf-71e2-4243-8967-ab4324818b31",
    "parent": "20391fbc-fa77-4fef-9aea-cf59f27f90b5",
    "status": "ok"
  },
  {
    "actual_size": 4325376,
    "format": "cow",
    "id": "b5e56953-bf86-4496-a6bd-19132cba9763",
    "parent": "13ad63bf-71e2-4243-8967-ab4324818b31",
    "status": "ok"
  }
]

5. Download disk snapshot 20391fbc-fa77-4fef-9aea-cf59f27f90b5

$ ./download_disk_snapshot.py -c engine-dev nfs1 20391fbc-fa77-4fef-9aea-cf59f27f90b5 snap1.qcow2
[   0.0 ] Connecting...
[   0.1 ] Creating image transfer...
[   1.4 ] Transfer ID: bc7e2379-08c1-4112-8628-b55ab2752c90
[   1.4 ] Transfer host name: host3
[   1.4 ] Downloading image...
[ 100.00% ] 6.00 GiB, 5.19 seconds, 1.16 GiB/s                                 
[   6.6 ] Finalizing image transfer...

6. Download disk snapshot 13ad63bf-71e2-4243-8967-ab4324818b31, rebasing
   it on top of snap1.qcow2

$ ./download_disk_snapshot.py -c engine-dev nfs1 13ad63bf-71e2-4243-8967-ab4324818b31 snap2.qcow2 --backing-file snap1.qcow2
[   0.0 ] Connecting...
[   0.1 ] Creating image transfer...
[   1.3 ] Transfer ID: db912896-1e8f-4441-bbc8-4887168d07b9
[   1.3 ] Transfer host name: host3
[   1.3 ] Downloading image...
[ 100.00% ] 6.00 GiB, 0.26 seconds, 22.80 GiB/s                                
[   1.6 ] Finalizing image transfer...

This creates the chain:

$ qemu-img info --backing-chain snap2.qcow2 
image: snap2.qcow2
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 4.82 MiB
cluster_size: 65536
backing file: snap1.qcow2
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: snap1.qcow2
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.65 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Note that the top image is very small, it contains only the data
written after the first snapshot was created, until the second snapshot
was created.

7. To verify that the image downloaded as separate snapshots is correct,
   download the entire image at the same point in time by downloading
   disk snapshot without --backing-chain:

$ ./download_disk_snapshot.py -c engine-dev nfs1 13ad63bf-71e2-4243-8967-ab4324818b31 snap2-full.qcow2
[   0.0 ] Connecting...
[   0.1 ] Creating image transfer...
[   1.4 ] Transfer ID: 3874ae55-bb76-428a-999a-37ca0ea4749e
[   1.4 ] Transfer host name: host4
[   1.4 ] Downloading image...
[ 100.00% ] 6.00 GiB, 11.35 seconds, 541.45 MiB/s                              
[  12.7 ] Finalizing image transfer...

This creates:

$ qemu-img info --backing-chain snap2-full.qcow2
image: snap2-full.qcow2
file format: qcow2
virtual size: 6 GiB (6442450944 bytes)
disk size: 1.65 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

8. Verify that both downloads have the same data

$ qemu-img compare snap2.qcow2 snap2-full.qcow2 -s
Images are identical.
Comment 17 Nir Soffer 2020-10-29 13:11:45 UTC 
Note about comparing images: -s flag (strict mode) works only
if the source base volume is also qcow2. If the base image is raw
you need to compare without -s.

To ensure a VM uses qcow2 disk on file based storage, select
"enable incremental backup" when creating the disk, or created
the VM from uploaded disk, where  you can control the format
of the disk.
Comment 22 Ilan Zuckerman 2020-11-23 11:49:16 UTC 
Verified on rhv-release-4.4.4-2

1. Start a vm with one non-empty disk
2. Create snapshot 1
3. Created snapshot 2
4. List disk snapshots using list_disk_snapshots.py

```
[root@storage-ge5-vdsm3 examples]# python3 list_disk_snapshots.py -c engine dfad7cbe-f932-4bbf-b62f-338102510b37
[
  {
    "actual_size": 10871635968,
    "format": "cow",
    "id": "a724c554-79b9-47f8-8537-6cad2c56dc1f",
    "parent": null,
    "status": "ok"
  },
  {
    "actual_size": 1073741824,
    "format": "cow",
    "id": "ddd9c387-2770-4522-be62-7b8449f8ed86",
    "parent": "a724c554-79b9-47f8-8537-6cad2c56dc1f",
    "status": "ok"
  },
  {
    "actual_size": 1073741824,
    "format": "cow",
    "id": "5a7ef718-976d-4c8e-a2de-e3796af89e92",
    "parent": "ddd9c387-2770-4522-be62-7b8449f8ed86",
    "status": "ok"
  }
]
```

5. Download disk snapshot a724c554-79b9-47f8-8537-6cad2c56dc1f

```
[root@storage-ge5-vdsm3 examples]# python3 download_disk_snapshot.py -c engine iscsi_0 a724c554-79b9-47f8-8537-6cad2c56dc1f snap1.qcow2
[   0.0 ] Connecting...
[   0.3 ] Creating image transfer...
[   3.6 ] Transfer ID: 484127d4-8523-4283-b12c-d13848e08689
[   3.6 ] Transfer host name: host_mixed_3
[   3.6 ] Downloading image...
[ 100.00% ] 10.00 GiB, 15.78 seconds, 648.74 MiB/s                             
[  19.4 ] Finalizing image transfer...
```

6. Download disk snapshot 13ad63bf-71e2-4243-8967-ab4324818b31, rebasing
   it on top of snap1.qcow2
```
[root@storage-ge5-vdsm3 examples]# python3 download_disk_snapshot.py -c engine iscsi_0 ddd9c387-2770-4522-be62-7b8449f8ed86 snap2.qcow2 --backing-file snap1.qcow2
[   0.0 ] Connecting...
[   0.3 ] Creating image transfer...
[   3.5 ] Transfer ID: c8a6bded-e31c-4cce-a9a0-1db48134a452
[   3.5 ] Transfer host name: host_mixed_3
[   3.5 ] Downloading image...
[ 100.00% ] 10.00 GiB, 0.13 seconds, 74.58 GiB/s                               
[   3.6 ] Finalizing image transfer...
```

This creates the chain:
```
[root@storage-ge5-vdsm3 examples]# qemu-img info --backing-chain snap2.qcow2
image: snap2.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 704 KiB
cluster_size: 65536
backing file: snap1.qcow2
backing file format: qcow2
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

image: snap1.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.22 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
```

7. To verify that the image downloaded as separate snapshots is correct,
   download the entire image at the same point in time by downloading
   disk snapshot without --backing-chain:
```
[root@storage-ge5-vdsm3 examples]# python3 download_disk_snapshot.py -c engine iscsi_0 ddd9c387-2770-4522-be62-7b8449f8ed86 snap2-full.qcow2
[   0.0 ] Connecting...
[   0.3 ] Creating image transfer...
[   3.6 ] Transfer ID: c6628315-3bba-4add-8720-61ee48cfa0c9
[   3.6 ] Transfer host name: host_mixed_3
[   3.6 ] Downloading image...
[ 100.00% ] 10.00 GiB, 15.41 seconds, 664.44 MiB/s                             
[  19.0 ] Finalizing image transfer...
```

This created:
```
[root@storage-ge5-vdsm3 examples]# qemu-img info --backing-chain snap2-full.qcow2
image: snap2-full.qcow2
file format: qcow2
virtual size: 10 GiB (10737418240 bytes)
disk size: 2.22 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
```

8. Verify that both downloads have the same data
```
[root@storage-ge5-vdsm3 examples]# qemu-img compare snap2.qcow2 snap2-full.qcow2 -s
Images are identical.
```
Comment 23 Sandro Bonazzola 2020-12-21 12:36:05 UTC 
This bugzilla is included in oVirt 4.4.4 release, published on December 21st 2020.

Since the problem described in this bug report should be resolved in oVirt 4.4.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.
Note You need to log in before you can comment on or make changes to this bug.