Bug 1365237 - Upload image doesn't notice that the disk image was removed, it finalizes the upload and marks it as OK
Summary: Upload image doesn't notice that the disk image was removed, it finalizes the...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.0.2.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ovirt-4.1.3
: 4.1.3
Assignee: Daniel Erez
QA Contact: Natalie Gavrielov
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-08 17:00 UTC by Natalie Gavrielov
Modified: 2017-07-06 13:42 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-07-06 13:42:39 UTC
oVirt Team: Storage
Embargoed:
rule-engine: ovirt-4.1+
rule-engine: planning_ack+
rule-engine: devel_ack+
ratamir: testing_ack+


Attachments (Terms of Use)
logs (1.61 MB, application/x-gzip)
2016-08-08 17:00 UTC, Natalie Gavrielov
no flags Details
logs: engine, imageio-proxy, imageio-daemon (531.01 KB, application/zip)
2017-04-30 15:09 UTC, Natalie Gavrielov
no flags Details
snapshot (399.72 KB, image/png)
2017-04-30 15:13 UTC, Natalie Gavrielov
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 73584 0 master MERGED webadmin: detect file read error during upload 2017-03-06 10:25:01 UTC
oVirt gerrit 73586 0 ovirt-engine-4.1 MERGED webadmin: detect file read error during upload 2017-03-06 10:57:35 UTC
oVirt gerrit 76793 0 master MERGED webadmin: image upload - xhr timeout handling 2017-05-14 14:56:02 UTC
oVirt gerrit 76801 0 ovirt-engine-4.1 MERGED webadmin: image upload - xhr timeout handling 2017-05-15 08:21:13 UTC

Description Natalie Gavrielov 2016-08-08 17:00:56 UTC
Created attachment 1188862 [details]
logs

Description of problem:
Upload image should have a way of telling if it uploaded the whole disk image or just a part of it (in case something went wrong, and somehow the disk image disappeared from it's location.

Version-Release number of selected component:
rhevm-4.0.2.4-0.1.el7ev.noarch
ovirt-imageio-common-0.3.0-0.el7ev.noarch
ovirt-imageio-proxy-0.3.0-0.el7ev.noarch
vdsm-4.18.10-1.el7ev.x86_64
ovirt-imageio-daemon-0.3.0-0.el7ev.noarch

How reproducible:
100%

Steps to Reproduce:
1. Upload a disk image.
2. While upload image takes place (is active) delete the disk image from its location.

Actual results:
The moment file was deleted, status bar changes to Finalizing then Complete and eventually OK.

Expected results:

A way for the upload to notice that something went wrong - even pause the upload if it can't alert the user of the strange situation.

Additional info:
Happens both for qcow2 disk images and raw.

Comment 1 Allon Mureinik 2016-09-12 07:58:06 UTC
Does this happen on both file and block based domains?

Comment 2 Natalie Gavrielov 2016-09-15 12:03:29 UTC
(In reply to Allon Mureinik from comment #1)
> Does this happen on both file and block based domains?

I couldn't remember whether I tested this on both block and file or just one of them, so I retested it now:
I used the same scenario described in comment #0, but with 4.0.4 builds.
Got completely different results this time:
Seems it doesn't have any timeout now.
for example, I deleted the file being uploaded when the UI showed:
"Sent 280 of 3072 MB"
And it stays that way for more than half an hour now (I have a feeling it will stay that way..)

Comment 3 Natalie Gavrielov 2017-04-30 15:09:12 UTC
Created attachment 1275290 [details]
logs: engine, imageio-proxy, imageio-daemon

Performed scenario described in comment 0, for both block and file storage types, in both cases the upload remains as if it is still active (does not change state in any point, shows: Sent x of y MB in the progress bar).

Now, the first time I tested this, apparently the engine's date and time wasn't in sync, so I fixed that and once the time changed the two uploads switched state to failed.
*But* when I performed the test again with engine showing the correct time, the uploads just won't switch to failed - and are shown as active.

Comment 4 Natalie Gavrielov 2017-04-30 15:10:41 UTC
Version tested:
rhevm-4.1.2-0.1.el7.noarch
ovirt-imageio-proxy-1.0.0-0.el7ev.noarch

Comment 5 Red Hat Bugzilla Rules Engine 2017-04-30 15:10:46 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 6 Natalie Gavrielov 2017-04-30 15:13:22 UTC
Created attachment 1275292 [details]
snapshot

Comment 7 Daniel Erez 2017-05-07 09:17:03 UTC
(In reply to Natalie Gavrielov from comment #3)
> Created attachment 1275290 [details]
> logs: engine, imageio-proxy, imageio-daemon
> 
> Performed scenario described in comment 0, for both block and file storage
> types, in both cases the upload remains as if it is still active (does not
> change state in any point, shows: Sent x of y MB in the progress bar).
> 
> Now, the first time I tested this, apparently the engine's date and time
> wasn't in sync, so I fixed that and once the time changed the two uploads
> switched state to failed.
> *But* when I performed the test again with engine showing the correct time,
> the uploads just won't switch to failed - and are shown as active.

@Natalie:
* Can you please attach the ui.log as well? 
* In which browsers have you tried it (and which versions)?

Comment 8 Daniel Erez 2017-05-14 09:16:12 UTC
Seems that the current fix is functioning in Chrome but not FireFox.
Thus, added another solution for FireFox: handling the timeout error raised by file removal. I.e. removing the file during upload will raise an XHR timeout error configurable by UploadImageXhrTimeoutInSeconds value - 2 minutes by default.

Comment 9 Natalie Gavrielov 2017-06-07 15:21:06 UTC
Partially verified:

Firefox 52.1.1 (64-bit) - uploaded both qcow2 v3 and raw to both nfs and iscsi
While upload is in progress, I moved the files to a different location and the upload switched state to "Paused by the system", then resumed the upload, and performed the move/resume a few more times.
Works just fine.

I was unable to test this on my chrome browser, because of bug 1459610.

Comment 10 Natalie Gavrielov 2017-06-07 15:22:35 UTC
(In reply to Natalie Gavrielov from comment #9)
> Partially verified:
> 
> Firefox 52.1.1 (64-bit) - uploaded both qcow2 v3 and raw to both nfs and
> iscsi
> While upload is in progress, I moved the files to a different location and
> the upload switched state to "Paused by the system", then resumed the
> upload, and performed the move/resume a few more times.
> Works just fine.
> 
> I was unable to test this on my chrome browser, because of bug 1459610.

Builds used:
ovirt-engine-4.1.3.1-0.1.el7.noarch
ovirt-imageio-proxy-1.0.0-0.el7ev.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch

Comment 11 Natalie Gavrielov 2017-06-11 12:31:36 UTC
Verified also using Chrome 55.0.2883.87 (64-bit)
and builds: 
ovirt-engine-4.1.3.2-0.1.el7.noarch
ovirt-imageio-common-1.0.0-0.el7ev.noarch
ovirt-imageio-proxy-1.0.0-0.el7ev.noarch
vdsm-4.19.18-1.el7ev.x86_64
ovirt-imageio-daemon-1.0.0-0.el7ev.noarch


Note You need to log in before you can comment on or make changes to this bug.