Bug 1851708

Description of problem:

Downloading a template disk using upload_disk.py example from the ovirt engine
sdk fails.

Creating transfer step times out after 60 seconds, and canceling the transfer
fails (this is another bug):

$ ./download_disk.py \
    --engine-url https://engine3 \
    --username admin@internal \
    --password-file engine3-password \
    --cafile engine3.pem \
    9b62b5fa-920e-4d0c-baf6-40406106e48e \
    download.img
Connecting...
Creating image transfer...
Traceback (most recent call last):
  File "./download_disk.py", line 163, in <module>
    connection, disk, types.ImageTransferDirection.DOWNLOAD, host=host)
  File "/home/nsoffer/src/ovirt-engine-sdk/sdk/examples/helpers/imagetransfer.py", line 203, in create_transfer
    transfer_service.cancel()
  File "/usr/local/lib64/python3.7/site-packages/ovirtsdk4/services.py", line 13748, in cancel
    return self._internal_action(action, 'cancel', None, headers, query, wait)
  File "/usr/local/lib64/python3.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action
    return future.wait() if wait else future
  File "/usr/local/lib64/python3.7/site-packages/ovirtsdk4/service.py", line 55, in wait
    return self._code(response)
  File "/usr/local/lib64/python3.7/site-packages/ovirtsdk4/service.py", line 296, in callback
    self._check_fault(response)
  File "/usr/local/lib64/python3.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault
    self._raise_error(response, body)
  File "/usr/local/lib64/python3.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error
    raise error
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is "[Cannot cancel image transfer from any source other than the browser.]". HTTP response code is 409.

The transfer remains in "initializing" state forever.

# https://engine3/ovirt-engine/api/imagetransfers/

<image_transfers>
<image_transfer href="/ovirt-engine/api/imagetransfers/c2f33238-2ab6-4cff-bfce-857350773b84" id="c2f33238-2ab6-4cff-bfce-857350773b84">
<actions>
<link href="/ovirt-engine/api/imagetransfers/c2f33238-2ab6-4cff-bfce-857350773b84/cancel" rel="cancel"/>
<link href="/ovirt-engine/api/imagetransfers/c2f33238-2ab6-4cff-bfce-857350773b84/resume" rel="resume"/>
<link href="/ovirt-engine/api/imagetransfers/c2f33238-2ab6-4cff-bfce-857350773b84/pause" rel="pause"/>
<link href="/ovirt-engine/api/imagetransfers/c2f33238-2ab6-4cff-bfce-857350773b84/extend" rel="extend"/>
<link href="/ovirt-engine/api/imagetransfers/c2f33238-2ab6-4cff-bfce-857350773b84/finalize" rel="finalize"/>
</actions>
<active>false</active>
<direction>download</direction>
<format>raw</format>
<inactivity_timeout>60</inactivity_timeout>
<phase>initializing</phase>
<transferred>0</transferred>
<image id="9b62b5fa-920e-4d0c-baf6-40406106e48e"/>
</image_transfer>
</image_transfers>


In vdsm log we can see that engine starts the nbd server with readonly=False
and vdsm reject the request (as it should):

2020-06-27 07:38:24,400+0300 INFO  (jsonrpc/7) [vdsm.api] START start_nbd_server(server_id='cd8192b2-292b-4e85-9cfd-eb6df547ca2a', config={'discard': False, 'readonly': Fals
e, 'sd_id': '56ecc03c-4bb5-4792-8971-3c51ea924d2e', 'img_id': '9b62b5fa-920e-4d0c-baf6-40406106e48e', 'vol_id': 'ed6dc087-632c-40d9-9b0d-9ccc7dcce0e8'}) from=::ffff:192.168.
122.12,53836, flow_id=03da0db8-4f7e-41b4-a06a-ad01963a49c3, task_id=a0be2c02-f920-4738-b2dc-1864d51abf62 (api:48)

2020-06-27 07:38:24,403+0300 INFO  (jsonrpc/7) [vdsm.api] FINISH start_nbd_server error=Shared volume is read only: ['sdUUID: 56ecc03c-4bb5-4792-8971-3c51ea924d2e', 'imgUUID: 9b62b5fa-920e-4d0c-baf6-40406106e48e', 'volUUID: ed6dc087-632c-40d9-9b0d-9ccc7dcce0e8'] from=::ffff:192.168.122.12,53836, flow_id=03da0db8-4f7e-41b4-a06a-ad01963a49c3, task_id=a0be2c02-f920-4738-b2dc-1864d51abf62 (api:52)

2020-06-27 07:38:24,403+0300 ERROR (jsonrpc/7) [storage.TaskManager.Task] (Task='a0be2c02-f920-4738-b2dc-1864d51abf62') Unexpected error (task:880)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run
    return fn(*args, **kargs)
  File "<decorator-gen-221>", line 2, in start_nbd_server
  File "/usr/lib/python3.6/site-packages/vdsm/common/api.py", line 50, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/hsm.py", line 3771, in start_nbd_server
    url = nbd.start_server(server_id, config)
  File "/usr/lib/python3.6/site-packages/vdsm/storage/nbd.py", line 95, in start_server
    raise se.SharedVolumeNonWritable(vol)
vdsm.storage.exception.SharedVolumeNonWritable: Shared volume is read only: ['sdUUID: 56ecc03c-4bb5-4792-8971-3c51ea924d2e', 'imgUUID: 9b62b5fa-920e-4d0c-baf6-40406106e48e',
 'volUUID: ed6dc087-632c-40d9-9b0d-9ccc7dcce0e8']

Version-Release number of selected component (if applicable):
4.4.1.5-0.0.master.20200624174458.git572f5c80257.el8

How reproducible:
Always

Steps to Reproduce:
1. Find a template disk UUID
2. Try to download the disk

$ ./download_disk.py \
    --engine-url https://engine3 \
    --username admin@internal \
    --password-file engine3-password \
    --cafile engine3.pem \
    d84456fb-36da-47e1-8728-1df0c2b58871 \
    download.img


Actual results:
Creating transfer times out, transfer remain in intiializing status.

Expected results:
Transfer should be created successfully and image downloaded.

We have 2 issues:

- NBD server is started incorrectly - for downloads the server must be
  started in readonly mode.

- Error starting NBD server is ignored. The transfer should fail and
  and perform the clean up.

- No way to recover by cancelling the transfer. Maybe finalize works,
  but it does not make sense to finalize a transfer that is not in 
  transferring phase.

Marking as urgent since this breaks existing backup flows.