Bug 2077649 - Block image transfer to/from Managed Block Storage domain
Summary: Block image transfer to/from Managed Block Storage domain
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.5.0.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ovirt-4.5.2
: ---
Assignee: Artiom Divak
QA Contact: Avihai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-04-21 19:54 UTC by JC Lopez
Modified: 2022-08-30 08:49 UTC (History)
8 users (show)

Fixed In Version: ovirt-engine-4.5.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-08-08 08:17:16 UTC
oVirt Team: Storage
Embargoed:
pm-rhel: ovirt-4.5?
pm-rhel: planning_ack?
pm-rhel: devel_ack+
pm-rhel: testing_ack?


Attachments (Terms of Use)
Contains all files mentioned in the description (983.76 KB, application/zip)
2022-04-21 19:54 UTC, JC Lopez
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github oVirt ovirt-engine pull 476 0 None Merged core: Block MBS Image upload 2022-07-31 11:53:33 UTC
Red Hat Issue Tracker RHV-45849 0 None None None 2022-04-21 20:10:55 UTC

Description JC Lopez 2022-04-21 19:54:56 UTC
Created attachment 1874246 [details]
Contains all files mentioned in the description

Description of problem:
We are testing to implement support for a new Managed Block Storage storage domain support using a Lightbits Labs storage cluster.

During our testing we have tried to upload an image directly to the MBS we created (called lights) and also tried to copy an image from a local storage domain to the MBS we created.

Both attempts failed

Version-Release number of selected component (if applicable):
2.4.3 ovirt-imageio
4.5 overt engine

How reproducible:
Easy

Steps to Reproduce:
1.Create a local storage domain
2.Create a MBS
3.Try uploading a VM image to the MBS directly

Actual results:
Upload image fails with no details

Expected results:
The image is uploaded

Additional info:
Attached the log info from the engine.log (UploadImageDirectToMBS.txt)

Also tried copying the image from the local storage domain to the MBS -> volume gets created in the Lightbits Cluster backend but the copy fails. The new volume created on the MBS has a status that says 'Illegal'

Info for this use case
Engine.log -> CopyoVirtImage-LD2MBS.txt
Image Copy Dialog Window -> CopyoVirtImage-LD2MBS-CopyDialog.png
Disk status window -> CopyoVirtImage-LD2MBS-AfterStatus.png
Backend volume status after copy request -> CopyoVirtImage-LD2MBS-BackendStatus.png

Comment 1 RHEL Program Management 2022-04-21 20:04:36 UTC
The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Comment 2 Nir Soffer 2022-04-21 20:17:34 UTC
ovirt-imageio supports can write to any storage supported by qemu-nbd.
The issue is in engine, responsible for orchestration of image transfers.

The problem is that engine does not support image transfer to managed
block storage domains. Adding support for MBS disks should not be hard,
it is just missing feature.

With the current system the only way to upload images is to create a data
domain using another type of storage (e.g. NFS), and upload the images
to the data domain, and move the disk to the MBS domain.

To implement this support for MBS disks, we need:

Engine UI:
- Prevent upload of qcow2 disks to MBS domain, since we don't support converting
  the format during the upload from the UI.

Engine backend:
- Instead of preparing the image before the upload, attach the disk to the host
- Instead of tearing down the image after the upload, detach the disk to the host

Vdsm:
- Starting NBD server in assumes an image (PDIV) add support for MBS volume
  (basically attached volume at /run/vdsm/managedvolume/uuid)

Comment 3 Nir Soffer 2022-04-21 20:23:59 UTC
JC, coping an image from data storage domain (for example the master domain) to MBS
should work (but of course never tested with NVMe/tcp storage.

Please report another bug for this, and include complete vdsm log showing the copy
flow. We should see the logs attaching the volume to the host, and the copy_data
logs. Grepping for the volume UUID in the log should show the right log file(s).

Comment 4 Nir Soffer 2022-04-21 20:47:52 UTC
Vdsm API need to change to support MBS volume. Currently is expects
PDIV:

    NBDServerConfig: &NBDServerConfig
        added: '4.3' 
        description: NBD server configuration.
        name: NBDServerConfig
        properties:
        -   description: The storage domain UUID. 
            name: sd_id 
            type: *UUID 

        -   description: The image UUID. 
            name: img_id
            type: *UUID 

        -   description: The volume UUID. 
            name: vol_id
            type: *UUID 

        -   defaultvalue: false 
            description: Allow read only access.
            name: readonly
            type: boolean

        -   defaultvalue: false 
            description: Pass discard requests to underlying filesystem. By
                default discard requests are ignored. Recommended when
                uploading to sparse volume.
            name: discard
            type: boolean

        -   defaultvalue: false 
            description: Try to keep the volume sparse by converting zeroes in
              write commands to optimized write zeroes commands. If discard is
              enabled, zeroes writes deallocate space. By default zeroes are
              not detected. Recommended when uploading to a sparse volume.
            name: detect_zeroes
            type: boolean
            added: '4.5' 

        -   defaultvalue: true
            description: If true, expose the backing chain of volume using 
                qcow2 format. If false, expose only the specified volume.
            name: backing_chain
            type: boolean

        -   defaultvalue: null
            description: If set, the NBD server exports the specified dirty 
              bitmap. NBD client can use the dirty bitmap to download changed
              blocks when a VM is powered off.
            name: bitmap
            type: *UUID 

We need to add support for external volumes, similar to the way we support 
external volumes in SDM.copy_data(). However in this case we don't have 
an endpoint. We can deprecate "sd_id", "img_id", "vol_id" and add "endpoint"
which can use the DIVEndpoint and ExternalEndpoint like SDM.copy_data. Engine
can use the endpoint API with MBS volume, and the older API for images.

Comment 5 JC Lopez 2022-04-21 20:54:23 UTC
Hi Nir,

created https://bugzilla.redhat.com/show_bug.cgi?id=2077649 for the copy failure.

Included in the new BZ the UUIDs for localdomain, MBS domain, source and target image.

Look for line 37542 for the beginning of the copy sequence

Extract below
37542 2022-04-21 15:30:01,899-0400 INFO  (jsonrpc/3) [api.storage] START attach_volume(vol_id='f7b88e89-14b8-4384-ac3a-76a29c465bf1', connection_info={'driver_volume_type': 'lightos', 'data': {'lightos_nodes': {'10.100.10.101': {'target_portal': '10.100.10.101', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10      .103': {'target_portal': '10.100.10.103', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10.102': {'target_portal': '10.100.10.102', 'target_port': 8009, 'transport_type': 'tcp'}}, 'uuid': '3ae21b9e-a59d-4be7-b44f-80d1462d4d94', 'subsysnqn': 'nqn.2016-01.com.lightbitslabs:uuid:e42ee459-90d6-481f-a096-63aa      95248ba1', 'access_mode': 'rw'}}) from=::ffff:10.20.60.35,48428, flow_id=2d9862bf (api:48)
37543 2022-04-21 15:30:01,900-0400 INFO  (jsonrpc/3) [storage.managevolumedb] Adding volume f7b88e89-14b8-4384-ac3a-76a29c465bf1 connection_info={'driver_volume_type': 'lightos', 'data': {'lightos_nodes': {'10.100.10.101': {'target_portal': '10.100.10.101', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10.103'      : {'target_portal': '10.100.10.103', 'target_port': 8009, 'transport_type': 'tcp'}, '10.100.10.102': {'target_portal': '10.100.10.102', 'target_port': 8009, 'transport_type': 'tcp'}}, 'uuid': '3ae21b9e-a59d-4be7-b44f-80d1462d4d94', 'subsysnqn': 'nqn.2016-01.com.lightbitslabs:uuid:e42ee459-90d6-481f-a096-63aa95248      ba1', 'access_mode': 'rw'}} (managedvolumedb:181)

Best regards
JC

(In reply to Nir Soffer from comment #3)
> JC, coping an image from data storage domain (for example the master domain)
> to MBS
> should work (but of course never tested with NVMe/tcp storage.
> 
> Please report another bug for this, and include complete vdsm log showing
> the copy
> flow. We should see the logs attaching the volume to the host, and the
> copy_data
> logs. Grepping for the volume UUID in the log should show the right log
> file(s).

Comment 6 Nir Soffer 2022-04-21 21:58:55 UTC
Bug 2077666 tracks the copy issue.

Comment 7 Arik 2022-04-25 14:26:13 UTC
This sounds like a reasonable change for upstream, filed https://github.com/oVirt/ovirt-engine/issues/313 - contribution is welcome
Until then, we better block uploading an image to MBS domain

Comment 9 Sandro Bonazzola 2022-08-30 08:49:07 UTC
This bugzilla is included in oVirt 4.5.2 release, published on August 10th 2022.
Since the problem described in this bug report should be resolved in oVirt 4.5.2 release, it has been closed with a resolution of CURRENT RELEASE.
If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.