Description of problem: As ISO domains are deprecated, new ISOs should be uploaded to a DATA domain , but the upload via Web Admin Interface does not allow to provide the checksum of the image, nor the admin has any means (via WEB) to get the checksum of the ISO in order to verify that the upload was successfull. Please provide a way to verify that the ISO on the DATA domain has the checksum that the admin has obtained from the ISO source. Version-Release number of selected component (if applicable): ovirt-engine-4.3.7.2-1.el7.noarch ovirt-engine-webadmin-portal-4.3.7.2-1.el7.noarch How reproducible: Always Steps to Reproduce: 1.Go to Admin Portal -> Storage -> Disks 2.Select "Upload" -> Start 3.Add 2 (optional) fields -> "checksum type" & "checksum value" 4.When selecting the image in "Disks" the checksums "md5" , "sha1" , "sha256" & "sha512" are provided (precalculated after the upload) Actual results: There is no way to verify the checksum via WEB . Only doable solution is going to the storage and manually finding out the image and do the checksum Expected results: Upload to fail if checksum was provided. Provide checksums for "md5" , "sha1" , "sha256" & "sha512" which were calculated after the upload.
I didn't properly documented the reproducal steps - currently only step 1 &2 are valid. Step 3 & 4 would be used only if the RFE is implemented.
This request is not currently committed to 4.4.z, moving it to 4.5
The request is too specific. I think a more generic solution is needed, being able to verify image content using checksum without downloading the image. This is needed for any type of image, not only ISO images, and it should support uploading from the API/SDK, which is the recommenced way to upload images. The specific use case of uploading ISO images using the UI is the one that we are less likey to support. We have 2 issues: - We cannot run programs from the browser, so it will be hard to compute shasum of the uploaded image - ISO images are typically aligned to 2048 byes, but in oVirt all disks are aligned to 4k or 128m (depends on the storage type). The original size of the ISO is lost once the disk is created. What we can do now is: - Provide checksum for existing disks for any storage and allocation policy - Provide command line tool for computing checksum of guest data, so qcow2 and compressed qcow2 images can be verified. Known issues: - raw images not aligned to storage alignment (4k or 128m) will return different checksum compared with the original image. I think this can be solved by limting the checksum calcuation by length. Example use cases: - Verify that image was uploaded correctly without downloading the image - Verify that downloaded image have the same content as the original image - Check if disk was changed after it was backed up I think this will be very useful for testing.
How checksum work? - imageio server can compute checksum for disks. - imageio client can compute checksum for local images. - checksum is based on guest visible content of the image, not host data - checksum can be computed for any image format (raw, qcow2) and supports qcow2 chain (checksum computed for entire chain). - local checksum supports also images inside ova file. - computing checksum is expensive, so we don't do this by default in any flow. Users can use the new API if needed. - Comparing checksum is similar to "qemu-img compare", but we can compare local and remote images, or create the checksum on different machines and different time, and compare the checksum later. - Creating checksum of sparse images reads only the data extents of a disk, so it should be faster than tools that do not understand sparseness. - Creating checksum of disk guest visible content is slower compared with creating checksum of qcow2 image downloaded from same disk. Known limitations - Only API is supported, there is no UI. Users can use the new API in applications. - In cases where oVirt aligns uploaded image size to storage extent size the checksum of the uploaded image and the disk will not match. Examples: - upload raw image to raw disk on file storage, when image size is not aligned to 4096 bytes. - upload raw image to raw disk on block storage, when image size is not aligned to 128 MiB. - In both cases aligning local image before computing the checksum should work. - Computing a checksum of a single image in an image chain is not supported. How to test: Getting checksum for remote image: 1. Start image transfer for download: $ ./image_transfer.py --engine-url https://engine3 --username admin@internal --password engine3-password --cafile engine3.pem download 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 0.0 ] Connecting to engine... [ 0.0 ] Looking up disk 58daea80-1229-4c6b-b33c-1a4e568c8ad7 [ 0.2 ] Creating image transfer for download [ 1.6 ] Transfer ID: 25d6fa17-f7b0-4a2b-98eb-2cdfa20a6abf [ 1.6 ] Transfer host name: host4 [ 1.6 ] Transfer URL: https://host4:54322/images/c4cf32b4-b526-4d69-88d5-1d84375cff2c [ 1.6 ] Proxy URL: https://engine3:54323/images/c4cf32b4-b526-4d69-88d5-1d84375cff2c [ 1.6 ] Conneted to imageio server 2. Compute checksum: $ curl -k https://host4:54322/images/c4cf32b4-b526-4d69-88d5-1d84375cff2c/checksum {"checksum": "8b75ea5b7d976eb25acf3721cb1ee5ba7e173bc1", "algorithm": "sha1"} 3. Finalize the download If using image_transfer.py, interrupt the script. Creating checksum for local images using imageio client: Creating test images: # Raw image $ virt-builder fedora-32 -o disk.raw # Qcow2 image $ qemu-img convert -f raw -O qcow2 disk.raw disk.qcow2 # Compressed qcow2 image $ qemu-img convert -f raw -O qcow2 -c disk.raw disk-compressed.qcow2 # Qcow2 image with a backing file $ qemu-img create -f qcow2 -b disk.raw -F raw chain.qcow2 # OVA archive $ echo "this is not real ovf" > vm.ovf $ tar cf vm.ova vm.ovf disk-compressed.qcow2 Creating checksum from python shell - all checksums should be the same since the guest visible content is the same. $ python >>> from ovirt_imageio import client >>> client.checksum("disk.raw") '2df96976518821e000fcf92fae0bfc6a7fb5b2d2' >>> client.checksum("disk.qcow2") '2df96976518821e000fcf92fae0bfc6a7fb5b2d2' >>> client.checksum("chain.qcow2") '2df96976518821e000fcf92fae0bfc6a7fb5b2d2' >>> client.checksum("disk-compressed.qcow2") '2df96976518821e000fcf92fae0bfc6a7fb5b2d2' >>> client.checksum("vm.ova", member="disk-compressed.qcow2") '2df96976518821e000fcf92fae0bfc6a7fb5b2d2' Cancellation: - Cancelling a transfer in the middle of a long checksum calculation (g.e. 100g disk) will abort the checksum request with "403 Foribidden" error, with message "Ticket xxx-yyy was canceled". To test: 1. start image transfer 2. start long checksum request 3. Cancel the transfer from API or SDK using the transfer id. Performance testing: - Time to create checksum for preallocated image should be similar to existing tools like sha1sum. (Note that shasum is about twice slower compared to sha1sum) - Time to create checksum for same image in qcow2 format or raw sparse format should be lower than existing tools like sha1sun, when using same storage. Usage in automated tests: - In cases where you download an image for comparing image checksums or running "qemu-img compare" for verifying upload image, you can replace the download with checksum request. This will be best if the source image checksum is always the same and its checksum is pre-calculated. - Can be used to verify offline storage operations like deleting snapshot, restoring vm from backup, or restore vm to previous snapshot, assuming that you have a checksum of the image chain. - Can be used during full backup to compute a checksum of the current vm disk content before the data is downloaded. - Cannot be used on a snapshot of a running vm - this calculates a checksum of the entire chain starting at the given snapshot, not of a single snapshot.
(In reply to Nir Soffer from comment #4) Nir, please see my comment bellow. The message is not the same as you described. > Cancellation: > > - Cancelling a transfer in the middle of a long checksum calculation > (g.e. 100g disk) will abort the checksum request with "403 Foribidden" > error, with message "Ticket xxx-yyy was canceled". > Starting a transfer, and cancelling it in a process of checksum calculation: [root@storage-ge13-vdsm1 examples]# python3 image_transfer.py -c engine download 260523eb-0927-439d-bfd6-e5d4d692a649 [ 0.1 ] Connecting to engine... [ 0.1 ] Looking up disk 260523eb-0927-439d-bfd6-e5d4d692a649 [ 0.2 ] Creating image transfer for download [ 2.1 ] Transfer ID: 6b8cbf2a-23a2-46e5-95fc-5a78ffefb328 [ 2.1 ] Transfer host name: host_mixed_3 [ 2.1 ] Transfer URL: https://storage-ge13-vdsm3.scl.lab.tlv.redhat.com:54322/images/e6c9ca43-9a16-4836-95b1-248319375e68 [ 2.1 ] Proxy URL: https://storage-ge-13.scl.lab.tlv.redhat.com:54323/images/e6c9ca43-9a16-4836-95b1-248319375e68 [ 2.1 ] Conneted to imageio server ^C [ 36.4 ] Finalizing image transfer.. The actual message when aborting the transfer is: [root@storage-ge13-vdsm1 examples]# curl -k https://storage-ge13-vdsm3.scl.lab.tlv.redhat.com:54322/images/e6c9ca43-9a16-4836-95b1-248319375e68/checksum You are not allowed to access this resource: Ticket e6c9ca43-9a16-4836-95b1-248319375e68 was canceled
(In reply to Ilan Zuckerman from comment #5) > (In reply to Nir Soffer from comment #4) > Nir, please see my comment bellow. The message is not the same as you > described. > > > Cancellation: > > > > - Cancelling a transfer in the middle of a long checksum calculation > > (g.e. 100g disk) will abort the checksum request with "403 Foribidden" > > error, with message "Ticket xxx-yyy was canceled". > > > > Starting a transfer, and cancelling it in a process of checksum calculation: > > [root@storage-ge13-vdsm1 examples]# python3 image_transfer.py -c engine > download 260523eb-0927-439d-bfd6-e5d4d692a649 > [ 0.1 ] Connecting to engine... > [ 0.1 ] Looking up disk 260523eb-0927-439d-bfd6-e5d4d692a649 > [ 0.2 ] Creating image transfer for download > [ 2.1 ] Transfer ID: 6b8cbf2a-23a2-46e5-95fc-5a78ffefb328 > [ 2.1 ] Transfer host name: host_mixed_3 > [ 2.1 ] Transfer URL: > https://storage-ge13-vdsm3.scl.lab.tlv.redhat.com:54322/images/e6c9ca43-9a16- > 4836-95b1-248319375e68 > [ 2.1 ] Proxy URL: > https://storage-ge-13.scl.lab.tlv.redhat.com:54323/images/e6c9ca43-9a16-4836- > 95b1-248319375e68 > [ 2.1 ] Conneted to imageio server > ^C > [ 36.4 ] Finalizing image transfer.. We have now checksum_disk.py example: https://github.com/oVirt/ovirt-engine-sdk/blob/master/sdk/examples/checksum_disk.py You can use it instead for testing the new checksum API. $ ./checksum_disk.py -c engine 58daea80-1229-4c6b-b33c-1a4e568c8ad7 { "algorithm": "blake2b", "checksum": "05953b09bb83ba258e027d202fc6dd3170c75d2f", "block_size": 4194304 } You can compare to image checksum using checksum_image.py: https://github.com/oVirt/ovirt-engine-sdk/blame/master/sdk/examples/checksum_image.py $ ./checksum_image.py fedora-32.qcow2 { "algorithm": "blake2b", "checksum": "05953b09bb83ba258e027d202fc6dd3170c75d2f", "block_size": 4194304 } The output from checksum_disk.py and checksum_image.py should be the same. > The actual message when aborting the transfer is: > > [root@storage-ge13-vdsm1 examples]# curl -k > https://storage-ge13-vdsm3.scl.lab.tlv.redhat.com:54322/images/e6c9ca43-9a16- > 4836-95b1-248319375e68/checksum > You are not allowed to access this resource: Ticket > e6c9ca43-9a16-4836-95b1-248319375e68 was canceled This is what I described. You got a "403 Forbidden" response (use curl --include o4194304ption to show response headers) with the message I described: Ticket e6c9ca43-9a16-4836-95b1-248319375e68 was canceled This is also a good test for cancellation, related to other bug you work on. Note that you should test this with latest 4.4.3 build, using imageio 2.1.0 both on server and client side. Older versions used different checksum algorithm.
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
Verified on rhv-4.4.3-9 according guidelines from comment #6
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.