Created attachment 1874246 [details] Contains all files mentioned in the description Description of problem: We are testing to implement support for a new Managed Block Storage storage domain support using a Lightbits Labs storage cluster. During our testing we have tried to upload an image directly to the MBS we created (called lights) and also tried to copy an image from a local storage domain to the MBS we created. Both attempts failed Version-Release number of selected component (if applicable): 2.4.3 ovirt-imageio 4.5 overt engine How reproducible: Easy Steps to Reproduce: 1.Create a local storage domain 2.Create a MBS 3.Try uploading a VM image to the MBS directly Actual results: Upload image fails with no details Expected results: The image is uploaded Additional info: Attached the log info from the engine.log (UploadImageDirectToMBS.txt) Also tried copying the image from the local storage domain to the MBS -> volume gets created in the Lightbits Cluster backend but the copy fails. The new volume created on the MBS has a status that says 'Illegal' Info for this use case Engine.log -> CopyoVirtImage-LD2MBS.txt Image Copy Dialog Window -> CopyoVirtImage-LD2MBS-CopyDialog.png Disk status window -> CopyoVirtImage-LD2MBS-AfterStatus.png Backend volume status after copy request -> CopyoVirtImage-LD2MBS-BackendStatus.png
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.
ovirt-imageio supports can write to any storage supported by qemu-nbd. The issue is in engine, responsible for orchestration of image transfers. The problem is that engine does not support image transfer to managed block storage domains. Adding support for MBS disks should not be hard, it is just missing feature. With the current system the only way to upload images is to create a data domain using another type of storage (e.g. NFS), and upload the images to the data domain, and move the disk to the MBS domain. To implement this support for MBS disks, we need: Engine UI: - Prevent upload of qcow2 disks to MBS domain, since we don't support converting the format during the upload from the UI. Engine backend: - Instead of preparing the image before the upload, attach the disk to the host - Instead of tearing down the image after the upload, detach the disk to the host Vdsm: - Starting NBD server in assumes an image (PDIV) add support for MBS volume (basically attached volume at /run/vdsm/managedvolume/uuid)
JC, coping an image from data storage domain (for example the master domain) to MBS should work (but of course never tested with NVMe/tcp storage. Please report another bug for this, and include complete vdsm log showing the copy flow. We should see the logs attaching the volume to the host, and the copy_data logs. Grepping for the volume UUID in the log should show the right log file(s).
Vdsm API need to change to support MBS volume. Currently is expects PDIV: NBDServerConfig: &NBDServerConfig added: '4.3' description: NBD server configuration. name: NBDServerConfig properties: - description: The storage domain UUID. name: sd_id type: *UUID - description: The image UUID. name: img_id type: *UUID - description: The volume UUID. name: vol_id type: *UUID - defaultvalue: false description: Allow read only access. name: readonly type: boolean - defaultvalue: false description: Pass discard requests to underlying filesystem. By default discard requests are ignored. Recommended when uploading to sparse volume. name: discard type: boolean - defaultvalue: false description: Try to keep the volume sparse by converting zeroes in write commands to optimized write zeroes commands. If discard is enabled, zeroes writes deallocate space. By default zeroes are not detected. Recommended when uploading to a sparse volume. name: detect_zeroes type: boolean added: '4.5' - defaultvalue: true description: If true, expose the backing chain of volume using qcow2 format. If false, expose only the specified volume. name: backing_chain type: boolean - defaultvalue: null description: If set, the NBD server exports the specified dirty bitmap. NBD client can use the dirty bitmap to download changed blocks when a VM is powered off. name: bitmap type: *UUID We need to add support for external volumes, similar to the way we support external volumes in SDM.copy_data(). However in this case we don't have an endpoint. We can deprecate "sd_id", "img_id", "vol_id" and add "endpoint" which can use the DIVEndpoint and ExternalEndpoint like SDM.copy_data. Engine can use the endpoint API with MBS volume, and the older API for images.
Hi Nir, created https://bugzilla.redhat.com/show_bug.cgi?id=2077649 for the copy failure. Included in the new BZ the UUIDs for localdomain, MBS domain, source and target image. Look for line 37542 for the beginning of the copy sequence Extract below 37542 2022-04-21 15:30:01,899-0400 INFO (jsonrpc/3) [api.storage] START attach_volume(vol_id='f7b88e89-14b8-4384-ac3a-76a29c465bf1', connection_info={'driver_volume_type': 'lightos', 'data': {'lightos_nodes': {'10.100.10.101': {'target_portal': '10.100.10.101', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10 .103': {'target_portal': '10.100.10.103', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10.102': {'target_portal': '10.100.10.102', 'target_port': 8009, 'transport_type': 'tcp'}}, 'uuid': '3ae21b9e-a59d-4be7-b44f-80d1462d4d94', 'subsysnqn': 'nqn.2016-01.com.lightbitslabs:uuid:e42ee459-90d6-481f-a096-63aa 95248ba1', 'access_mode': 'rw'}}) from=::ffff:10.20.60.35,48428, flow_id=2d9862bf (api:48) 37543 2022-04-21 15:30:01,900-0400 INFO (jsonrpc/3) [storage.managevolumedb] Adding volume f7b88e89-14b8-4384-ac3a-76a29c465bf1 connection_info={'driver_volume_type': 'lightos', 'data': {'lightos_nodes': {'10.100.10.101': {'target_portal': '10.100.10.101', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10.103' : {'target_portal': '10.100.10.103', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10.102': {'target_portal': '10.100.10.102', 'target_port': 8009, 'transport_type': 'tcp'}}, 'uuid': '3ae21b9e-a59d-4be7-b44f-80d1462d4d94', 'subsysnqn': 'nqn.2016-01.com.lightbitslabs:uuid:e42ee459-90d6-481f-a096-63aa95248 ba1', 'access_mode': 'rw'}} (managedvolumedb:181) Best regards JC (In reply to Nir Soffer from comment #3) > JC, coping an image from data storage domain (for example the master domain) > to MBS > should work (but of course never tested with NVMe/tcp storage. > > Please report another bug for this, and include complete vdsm log showing > the copy > flow. We should see the logs attaching the volume to the host, and the > copy_data > logs. Grepping for the volume UUID in the log should show the right log > file(s).
Bug 2077666 tracks the copy issue.
This sounds like a reasonable change for upstream, filed https://github.com/oVirt/ovirt-engine/issues/313 - contribution is welcome Until then, we better block uploading an image to MBS domain
This bugzilla is included in oVirt 4.5.2 release, published on August 10th 2022. Since the problem described in this bug report should be resolved in oVirt 4.5.2 release, it has been closed with a resolution of CURRENT RELEASE. If the solution does not work for you, please open a new bug report.