Description of problem: Currently uploading disks from ova (tar file) requires writing manual HTTP code for reading data from tar and uploading it to imageio server. This is inefficient and incorrect: - using single thread with synchronous I/O - sending zero over wire - always creating fully allocated disks instead of sparse disks This is also hard to do right, and the current examples in the sdk are all outdated and broken in various ways, and generally bad example code. The right way to do this via imageio client: client.upload() But it does not support reading disks from ova file. Add this support so user can specify the offset and size of the image in the file, and the format of the disk. For example, for this ova file: $ tar -tvf vm.ova -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk1.qcow2 -rw-r--r-- nsoffer/nsoffer 454144 2020-06-22 21:34 disk2.qcow2 $ python -c 'import tarfile; print(list({"name": m.name, "offset": m.offset_data, "size": m.size} for m in tarfile.open("vm.ova")))' [{'name': 'disk1.qcow2', 'offset': 512, 'size': 454144}, {'name': 'disk2.qcow2', 'offset': 455168, 'size': 454144}] The user will call client.upload() with: When uploading disk1.qcow2: client.upload( "vm.ova", "https://host:54322/images/xxx-yyy", offset=512, size=454144, fmt="qcow2") When uploading disk2.qcow2: client.upload( "vm.ova", "https://host:54322/images/yyy-zzz", offset=455168, size=454144, fmt="qcow2") imageio client will configure qemu-nbd to open the file using: 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 512, "size": 454144, "file": {"driver", "file", "filename": "vm.ova"}}}' This expose the raw guest data from the disk which can be used by imageio client to upload the disk to imageio server normally. Another solution is to add support from uploading from URL (bug 1829047) and expose the disk using another qemu-nbd or nbdkit tar plugin. But this will preform worse, is more complex, harder to use, and add more dependencies.
*** Bug 1849992 has been marked as a duplicate of this bug. ***
We can support only these disk formats: - qcow2 - qcow2 compressed - raw - other formats such as vmdk may work but are not tested We cannot support raw compressed disks since qemu-nbd does not support decompressing raw disk contents. To upload OVA with compressed raw disks you need to unpack the disks and uncompress them before the upload. We may be able to support raw compressed disk using external nbd server supporting decompression, but I'm not sure such server exists. Another way to support compressed raw images is to support gzip content encoding in the server side and decompress data before writing to storage.
Uploads works now, but there are some issues. When uploading from ova, the disk provisioned-size is stored in the ova, but the size may not match the actual image size in the qcow2 image. The right size to use is the size from the qcow2 image since this is what vdsm will use later. To get the virtual size we need to tell "qemu-img info" to look into the tar at the right offset, and use qcow2 driver: $ qemu-img info 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 1536, "file": {"driver": "file", "filename": "fedora-32.ova"}}}' image: json:{"driver": "qcow2", "file": {"offset": 1536, "driver": "raw", "file": {"driver": "file", "filename": "fedora-32.ova"}}} file format: qcow2 virtual size: 6 GiB (6442450944 bytes) disk size: 645 MiB cluster_size: 65536 Format specific information: compat: 1.1 lazy refcounts: false refcount bits: 16 corrupt: false When uploading to qcow2 disk we need to measure the disk allocation. This also requires the same json spec: $ qemu-img measure -O qcow2 'json:{"driver": "qcow2", "file": {"driver": "raw", "offset": 1536, "file": {"driver": "file", "filename": "fedora-32.ova"}}}' required size: 1381302272 fully allocated size: 6443696128 The json format above is correct only to qcow2 image inside OVA. For raw image inside OVA we need: 'json:{"driver": "raw", "offset": 1536, "file": {"driver": "file", "filename": "fedora-32.ova"}}'
The API was simplified to accept the name of the disk in the OVA file. client.upload() will find the right offset and size: Locating disk name: $ tar tf fedora-32.ova vm.ovf fedora-32.qcow2 Uploading disk from OVA file: client.upload( "fedora-32.ova", transfer_url, cafile, fmt="qcow2", member="fedora-32.qcow2")
This might be useful: Description of problem: Currently, the ovirt-imageio is not supporting ova file uploading for ovirt 4.4, due to the deprecation of the ovirt-imageio proxy, at least for the default installation. The client.upload API has been added which allows for the upload_disk.py script of the ovirt-engine-sdk to upload disk images directly from the disk image file by providing the file path and name. For the upload_ova_as_template.py and upload_ova_as_vm.py scripts, the disk image resides in the ova and worked via obtaining the file descriptor. The client.upload api does not support file descriptors, however. Therefore the ovirt-imageio client needs to be modified to support ova uploads. One option is to pass the file offset of the image within the client.upload API and Image size if supporting the file descriptor is not feasible. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Run the upload_ova_as_template.py from the ovirt-engine-sdk 2. 3. Actual results: The script fails to upload the disk images of the ova file because the proxy is not available. Expected results: The upload_ova_as_template.py script shall upload all disk images and the template successfully. Additional info:
(In reply to Nir Soffer from comment #4) > Uploading disk from OVA file: > > client.upload( > "fedora-32.ova", > transfer_url, > cafile, > fmt="qcow2", > member="fedora-32.qcow2") Upload was simplified, disk format is detected automaticlally, so the fmt argument was removed. client.upload( "fedora-32.ova", transfer_url, cafile, member="fedora-32.qcow2")
All patches merged. Will be available in ovirt-imageio-client 2.0.9-1. Available now via ovirt-release-master.rpm in ovirt-imageio-client >= 2.0.9-0.202006300947.git91b1d59
QE storage team does not have the capacity for non-urgent customer bugs to verify at this moment. retargeting back to 4.4.3 as it was in the original plan. If this is urgent or customer-related existing issue in any way feel free to retarget it back. Also 2.0.9 is not yet in QE hands , moving back to MODIFIED.
I checked this for my usage, and the script fails: sudo python3 ./upload_from_ova.py --engine-url http://localhost:8080 --username admin@internal --sd-name engine_master -c /home/srosenbe/ovirt_engine_master/etc/pki/ovirt-engine/ca.pem --disk-sparse --ova-disk-name 2824e752-9a2d-44af-81d5-30763e79afe9 VM5.ova Checking image... Traceback (most recent call last): File "./upload_from_ova.py", line 204, in <module> image_info = client.info(args.ova_file, member=args.ova_disk_name) AttributeError: module 'ovirt_imageio.client' has no attribute 'info' The VM has two disks and I am specifying one, but it is not clear if this is the issue.
(In reply to Steven Rosenberg from comment #10) OVA support requires ovirt-imageio-client >= 2.0.9-1 Which version are you testing?
(In reply to Nir Soffer from comment #11) > (In reply to Steven Rosenberg from comment #10) > OVA support requires ovirt-imageio-client >= 2.0.9-1 > > Which version are you testing? I believe updating the daemon solved the problem on my development environment. It now succeeds: sudo python3 ./upload_from_ova.py --engine-url http://localhost:8080 --username admin@internal --sd-name engine_master -c /home/srosenbe/ovirt_engine_master/etc/pki/ovirt-engine/ca.pem --disk-sparse --ova-disk-name 2824e752-9a2d-44af-81d5-30763e79afe9 /ova/VM5.ova Checking image... Image format: qcow2 Disk name: 2824e752-9a2d-44af-81d5-30763e79afe9.qcow2 Disk format: cow Disk provisioned size: 16106127360 Disk initial size: 2686976 Disk backup: None Password: Creating disk... Disk ID: d8b21f1a-9507-4589-9e4e-54e0e2f7ad4b Creating image transfer... Transfer ID: 00f8e690-4d3a-44c3-b41f-3384598c6ee3 Transfer host name: Host1 Uploading disk 2824e752-9a2d-44af-81d5-30763e79afe9 from VM5.ova... [ 100.00% ] 15.00 GiB, 0.23 seconds, 64.12 GiB/s Finalizing image transfer... Upload completed successfully Thank you.
Verified on rhv-4.4.1-10 (imageio-2.0.9-1)
This bug is targeted to 4.4.3 but verified in 4.4.1, can you please check targeting?
This bugzilla is included in oVirt 4.4.3 release, published on November 10th 2020. Since the problem described in this bug report should be resolved in oVirt 4.4.3 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days