Created attachment 1181076 [details] engine.log, vdsm.log, image-proxy.log Description of problem: Uploading image with Image Type raw cause image verification failure. Version-Release number of selected component: rhevm-4.0.2-0.2.rc1.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch ovirt-imageio-proxy-0.3.0-0.el7ev.noarch vdsm-4.18.6-1.el7ev.x86_64 ovirt-imageio-daemon-0.3.0-0.el7ev.noarch How reproducible: 100% Steps to Reproduce: Upload image - 1. Image Type: QCOW2 2. Image Type: Raw Actual results: 1. Seems that when uploading image type QCOW2 passed successfully - image status when done is "OK". 2. When finishing: status failed then finalizing failure, and then Illegal. From engine.log: 2016-07-18 16:15:18,764 ERROR [org.ovirt.engine.core.bll.storage.disk.image.UploadDiskImageCommand] (DefaultQuartzScheduler8) [790e9c2a] Failed to verify uploaded image: {}: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to VerifyUntrustedVolumeVDS, error = Image verification failed: "reason=Volume's format specified by QEMU is qcow2, while the format specified in VDSM metadata is raw", code = 484 (Failed with error unexpected and code 16) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:114) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.verifyImage(UploadImageCommand.java:286) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.handleFinalizingSuccess(UploadImageCommand.java:266) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.executeStateHandler(UploadImageCommand.java:147) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommand.proceedCommandExecution(UploadImageCommand.java:116) [bll.jar:] at org.ovirt.engine.core.bll.storage.disk.image.UploadImageCommandCallback.doPolling(UploadImageCommandCallback.java:13) [bll.jar:] at org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:113) [bll.jar:] at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) [:1.8.0_101] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_101] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_101] at org.ovirt.engine.core.utils.timer.JobWrapper.invokeMethod(JobWrapper.java:77) [scheduler.jar:] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:51) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_101] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_101] Expected results: For the upload to work, and in the end display status "OK". Additional info:
Natalie, I'm afraid I don't understand the flow here. Are you selecting QCow in the GUI and then feeding it a RAW file? If not, could you clarify the stages you executed? Thanks!
Created attachment 1181123 [details] libvirtd.log - just in case
(In reply to Allon Mureinik from comment #1) > Natalie, I'm afraid I don't understand the flow here. > Are you selecting QCow in the GUI and then feeding it a RAW file? If not, > could you clarify the stages you executed? > > Thanks! Steps are: upload a file using UI, select image type - raw. When the upload is done, there is the following error: Failed to VerifyUntrustedVolumeVDS, error = Image verification failed: "reason=Volume's format specified by QEMU is qcow2, while the format specified in VDSM metadata is raw Note, that when performing the same scenario using image type QCOW2 - it finishes with status OK.
After talking with Natalie, the system seems to behave as it was designed. The system failed verification after uploading about 2mb of 3gb image, because the actual image format was qcow, while the selected image format was raw. Issues that we may need to investigate: - According to Natalie, it took 2-3 minutes until the upload failed - I would expect the failure after several seconds. - The UI does not give any clue about what happens. I would expect to see this: 1. Click upload 2. Show "verifying image" indeterminate progress (.e.g spinner) 3. If verification failed, show error dialog in the ui 4. When verification finish, show "uploading image" progress (progress bar) Since this was not specified when we started to work on this, we will have to improve this in the next version.
Hi Natalie, Are you using NFS or iSCSI? Thin or Preallocated? Note that unsupported combinations are now blocked from the UI (should be available on upcoming builds).
(In reply to Nir Soffer from comment #4) > After talking with Natalie, the system seems to behave as it was designed. > > The system failed verification after uploading about 2mb of 3gb image, > because > the actual image format was qcow, while the selected image format was raw. > > Issues that we may need to investigate: > > - According to Natalie, it took 2-3 minutes until the upload failed - I would > expect the failure after several seconds. > > - The UI does not give any clue about what happens. I would expect to see > this: > > 1. Click upload > 2. Show "verifying image" indeterminate progress (.e.g spinner) > 3. If verification failed, show error dialog in the ui > 4. When verification finish, show "uploading image" progress (progress bar) > > Since this was not specified when we started to work on this, we will have > to > improve this in the next version. I think there might be some misunderstandings here.. It all started when I tried to upload a QCOW disk as Raw and saw this fails within a few minutes (2-3) with no explanation - which is why I thought it is a bug, and opened bug 1357269. Later on, for reasons I can't explain.. behaviour was changed (even before I added the patch Nir sent me for the logging, and NO, I did not change/touch anything in the environment). How behaviour was changed? It started uploading QCOW2 files as Raw: It uploads the WHOLE file.. and then switches it's state to Illegal.. And then I opened THIS bug. So right now.. behaviour is wrong - we upload QCOW2 disk as Raw - all the 3GB.. not just 2-3 MB (eventually it fails.. which is good, but why upload the whole thing?!).
@Natalie - is it relevant only for file domains? If so, then it's already handled by bug 1353229 . I.e. uploading as QCOW2 on a file domain is not supported and thus disabled. BTW, the format verification can currently be done only after uploading the entire file, cause we need the file in the domain in order to test it.. So that's by design.
Another thing, when I've tried to reproduce it I got a proper event message in the webadmin [*], did you get it as well? [*] "VDSM localhost command failed: Image verification failed: "reason=Volume's format specified by QEMU is raw, while the format specified in VDSM metadata is qcow2""
(In reply to Daniel Erez from comment #8) > Another thing, when I've tried to reproduce it I got a proper event message > in the webadmin [*], did you get it as well? > > [*] "VDSM localhost command failed: Image verification failed: > "reason=Volume's format specified by QEMU is raw, while the format specified > in VDSM metadata is qcow2"" Yes, I did.
(In reply to Daniel Erez from comment #5) > Hi Natalie, > > Are you using NFS or iSCSI? Thin or Preallocated? > Note that unsupported combinations are now blocked from the UI (should be > available on upcoming builds). and (In reply to Daniel Erez from comment #7) > @Natalie - is it relevant only for file domains? 1. QCOW2 image disk, uploaded as Raw image type, preallocated, block 2. QCOW2 image disk, uploaded as Raw image type, thin provision, file 3. QCOW2 image disk, uploaded as Raw image type, preallocated, file For all these option it uploads the whole file, then switches state to Illegal. For the remaining option: QCOW2 image disk, uploaded as Raw image type, thin provision, block there is a message in the UI: Error while executing action: No Message
(In reply to Natalie Gavrielov from comment #10) > (In reply to Daniel Erez from comment #5) > > Hi Natalie, > > > > Are you using NFS or iSCSI? Thin or Preallocated? > > Note that unsupported combinations are now blocked from the UI (should be > > available on upcoming builds). > > and > > (In reply to Daniel Erez from comment #7) > > @Natalie - is it relevant only for file domains? > > > 1. QCOW2 image disk, uploaded as Raw image type, preallocated, block > 2. QCOW2 image disk, uploaded as Raw image type, thin provision, file > 3. QCOW2 image disk, uploaded as Raw image type, preallocated, file > > For all these option it uploads the whole file, then switches state to > Illegal. > > For the remaining option: QCOW2 image disk, uploaded as Raw image type, thin > provision, block > there is a message in the UI: Error while executing action: No Message I suspect that what happens here is this: 1. Staring first upload 2. First image verification fails after the first chunk of the image reach storage 3. Upload is paused, but the system remember the uploaded size (8MiB?) 4. Second upload to same image, system resume upload from last uploaded offset 5. Since we do not upload the first chunk, there is no verification of the first chunk 6. Upload finish 7. Last image verification fails, and image is paused again So the bug is in step 3 - when upload verification fails, the system must set the uploaded size to zero. Retrying the upload must start from the first chunk. Amit, can you confirm this?
(In reply to Nir Soffer from comment #11) > (In reply to Natalie Gavrielov from comment #10) > > (In reply to Daniel Erez from comment #5) > > > Hi Natalie, > > > > > > Are you using NFS or iSCSI? Thin or Preallocated? > > > Note that unsupported combinations are now blocked from the UI (should be > > > available on upcoming builds). > > > > and > > > > (In reply to Daniel Erez from comment #7) > > > @Natalie - is it relevant only for file domains? > > > > > > 1. QCOW2 image disk, uploaded as Raw image type, preallocated, block > > 2. QCOW2 image disk, uploaded as Raw image type, thin provision, file > > 3. QCOW2 image disk, uploaded as Raw image type, preallocated, file > > > > For all these option it uploads the whole file, then switches state to > > Illegal. > > > > For the remaining option: QCOW2 image disk, uploaded as Raw image type, thin > > provision, block > > there is a message in the UI: Error while executing action: No Message > > I suspect that what happens here is this: > > 1. Staring first upload > 2. First image verification fails after the first chunk of the image reach > storage > 3. Upload is paused, but the system remember the uploaded size (8MiB?) > 4. Second upload to same image, system resume upload from last uploaded > offset > 5. Since we do not upload the first chunk, there is no verification of the > first chunk > 6. Upload finish > 7. Last image verification fails, and image is paused again > > So the bug is in step 3 - when upload verification fails, the system must > set the > uploaded size to zero. Retrying the upload must start from the first chunk. > > Amit, can you confirm this? No, we don't have the initial verification implemented yet, Bug 1344285 should handle that. Natalie, if this bug is about making the upload fail faster when the format is incorrect, then we can set it to be a duplicate of Bug 1344285. Otherwise we might consider closing the bug, as its description and title describes a desired flow.
Irrelevant now.. closing.
Reopening - will be used as an RFE for client side verification.
Derez, do you remember to backport this patch?
(In reply to Amit Aviram from comment #15) > Derez, do you remember to backport this patch? Not sure we should. @Allon - is it 4.0 material or should be postponed to 4.1?
(In reply to Daniel Erez from comment #16) > (In reply to Amit Aviram from comment #15) > > Derez, do you remember to backport this patch? > > Not sure we should. > > @Allon - is it 4.0 material or should be postponed to 4.1? 4.0.4 material, like the target says. (Assuming we trust this patch, of course)
Daniel, I've noticed that format verification on client's side can only tell if the file is qcow2.. all other files are catalogued as raw.. is this true?
(In reply to Natalie Gavrielov from comment #18) > Daniel, > > I've noticed that format verification on client's side can only tell if the > file is qcow2.. all other files are catalogued as raw.. is this true? Yes, that's ok. RAW type is just a bunch of bits- without any format.
(In reply to Amit Aviram from comment #19) > (In reply to Natalie Gavrielov from comment #18) > > Daniel, > > > > I've noticed that format verification on client's side can only tell if the > > file is qcow2.. all other files are catalogued as raw.. is this true? > > Yes, that's ok. RAW type is just a bunch of bits- without any format. Another thing, qcow is recognized as qcow2..
Verified, Everything that is qcow2 / qcow2 v3 - recognized as qcow2, with the right compatibility mode. qcow is also recognized as qcow2 - I will open a separate bug. Everything else - recognized as raw. Builds used: rhevm-4.0.4-0.1.el7ev.noarch vdsm-4.18.12-1.el7ev.x86_64 ovirt-imageio-proxy-0.3.0-0.el7ev.noarch ovirt-imageio-common-0.3.0-0.el7ev.noarch ovirt-imageio-daemon-0.3.0-0.el7ev.noarch