Bug 1615124

Engine should mark tickets as sparse when the underlying disk is considered as thin
and storage is file based.

Engine must not mark a ticket as sparse for block storage. We may limit this 
restriction in the future by checking if the user selected wipe-after-delete
for the underlying disk.

Idan, do we have an easy way to detect a disk on block storage that can be safely
discarded during upload?

Yes, a disk can be discarded if it's not marked with WAD and its storage domain supports discard.

(In reply to Nir Soffer from comment #1)
> Engine should mark tickets as sparse when the underlying disk is considered
> as thin
> and storage is file based.
> 
> Engine must not mark a ticket as sparse for block storage. We may limit this 
> restriction in the future by checking if the user selected wipe-after-delete
> for the underlying disk.
> 
> Idan, do we have an easy way to detect a disk on block storage that can be
> safely
> discarded during upload?

@Nir - we can check if wipe-after-delete is selected on the disk (we have it also in upload flow). Should we do it now or just allow sparse for thin disk on file domain?

(In reply to Daniel Erez from comment #3)
> @Nir - we can check if wipe-after-delete is selected on the disk (we have it
> also in upload flow). Should we do it now or just allow sparse for thin disk
> on file domain?

I would defer this to another RFE, since on imageio side we don't support yet
trim for block storage, and I don't want to risk this feature for 4.2.6.

(In reply to Nir Soffer from comment #4)
> (In reply to Daniel Erez from comment #3)
> > @Nir - we can check if wipe-after-delete is selected on the disk (we have it
> > also in upload flow). Should we do it now or just allow sparse for thin disk
> > on file domain?
> 
> I would defer this to another RFE, since on imageio side we don't support yet
> trim for block storage, and I don't want to risk this feature for 4.2.6.

ok, opened an RFE for that: https://bugzilla.redhat.com/show_bug.cgi?id=1619006

Daniel, can you please provide the scenario for verification?
Thanks

(In reply to Elad from comment #6)
> Daniel, can you please provide the scenario for verification?
> Thanks

We currently only use it in v2v flow, i.e. sparse is not supported using engine api/ui. So the best way to verify it is by checking AddImageTicket in the log (e.g. [*]). It should be true for thin disks on file domain.

[*]
INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-17) [b32b5c96-6d02-4eac-acad-b90520c1b43b] START, AddImageTicketVDSCommand(HostName = host1, AddImageTicketVDSCommandParameters:{hostId='0e7696d2-c45a-4436-ac2c-392581f56e99', ticketId='0fa87884-f87a-4c58-951b-237bace1b6a6', timeout='300', operations='[write]', size='46137344', url='file:///rhev/data-center/mnt/derezengine.usersys:_home_data_nfs_data1/8f3a6be6-c65e-4640-91aa-31580fe91262/images/5a2203bc-395b-4ba1-8bbb-f3118ed48b5c/4b5e174d-118c-45d0-ade3-67355b862977', filename='null', sparse='true'}), log id: 40a62d65

Even better, check that ovirt-imageio-daemon got a ticket with "sparse": True.

1. Start upload from the UI to NFS storage domain
2. Look in daemon log for the ADD log message for this transfer:

2018-08-19 10:15:47,714 INFO    (Thread-1) [tickets] [local] ADD ticket={u'url': u'file:///rhev/data-center/mnt/blockSD/bee83d01-aab7-4581-921f-d0145235c757/images/b2f50c31-c878-4a1b-96ed-010060e6ca49/f96d3922-03cc-4169-ae9f-6e37d520a30a', u'ops': [u'write'], u'uuid': u'4fe94bbe-db25-465c-a2a5-e83e254882b0', u'timeout': 300, u'size': 107374182400}

There should be a u'sparse': True key in the ticket dict.

Checking that imageio does the right thing for sparse image should be done at
bug 1615122.

To check the entire flow, we need the imageio client, supporting sparse uploads.

Does download sprase require a different RFE?

(In reply to Yaniv Lavi from comment #9)
> Does download sprase require a different RFE?

Yes, should open a new RFE for it.

Thanks Yaniv and Daniel.Veritas will open a new RFE for this.

Best,
Pavan.

Verified using:
ovirt-engine-4.2.6.4-0.0.master.20180823141941.gitb3a5b18.el7.noarch
vdsm-4.20.38-1.gitb49b34d.el7.x86_64

Uploaded the next image using WebAdmin:
# qemu-img info temp_img.qcow2 
image: temp_img.qcow2
file format: qcow2
virtual size: 10G (10737418240 bytes)
disk size: 50M
cluster_size: 65536
Format specific information:
    compat: 0.10
    refcount bits: 16


From engine log:

2018-08-28 11:50:34,337+03 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.AddImageTicketVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-87) [e6365c22-a317-4f08-8d8d-728b248b867f] START, AddImageTicketVDSCommand(HostName = host_mixed_2, AddImageTicketVDSCommandParameters:{hostId='e03db7f3-8d22-4644-b6b3-e1df54d294f7', ticketId='b6339314-e880-4765-bf93-fc3abefd2ed1', timeout='300', operations='[write]', size='10737418240', url='file:///rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Storage__NFS_storage__local__ge16__nfs__1/45c4a7f6-394e-4583-92c8-1bd73ce43dac/images/5d6afe53-0f21-4d7c-81e4-2acc7f2305b7/efe38bf9-47fb-4858-b457-1e7f6d89c30d', filename='null', sparse='true'}), log id: 739c7704


From daemon log:

2018-08-28 11:50:34,373 INFO    (Thread-1) [tickets] [local] ADD ticket={u'uuid': u'b6339314-e880-4765-bf93-fc3abefd2ed1', u'ops': [u'write'], u'url': u'file:///rhev/data-center/mnt/yellow-vdsb.qa.lab.tlv.redhat.com:_Storage__NFS_storage__local__ge16__nfs__1/45c4a7f6-394e-4583-92c8-1bd73ce43dac/images/5d6afe53-0f21-4d7c-81e4-2acc7f2305b7/efe38bf9-47fb-4858-b457-1e7f6d89c30d', u'sparse': True, u'timeout': 300, u'size': 10737418240}

QE verification bot: the bug was verified upstream