Bug 984491

Summary: Exceptions thrown when attempting to end a task successfully result in the task looping forever.
Product: Red Hat Enterprise Virtualization Manager Reporter: Lee Yarwood <lyarwood>
Component: ovirt-engineAssignee: Omer Frenkel <ofrenkel>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: 3.2.0CC: acathrow, audgiri, iheim, lpeer, lsvaty, michal.skrivanek, Rhev-m-bugs, yeylon
Target Milestone: ---Keywords: Triaged
Target Release: 3.3.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: virt
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-07-29 13:12:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lee Yarwood 2013-07-15 11:32:17 UTC
Description of problem:

Exceptions thrown when attempting to end a task successfully result in the task looping forever, for example with an import task :

2013-07-06 21:06:15,423 ERROR [org.ovirt.engine.core.bll.EntityAsyncTask] (pool-3-thread-49) EntityAsyncTask::EndCommandAction [within thread]: EndAction for action type ImportVm threw an exception: org.ovirt.engine.core.compat.CompatException: java.io.IOException: Permission denied
        at org.ovirt.engine.core.compat.backendcompat.Path.GetTempFileName(Path.java:23) [engine-compat.jar:]
        at org.ovirt.engine.core.utils.ovf.OvfWriter.<init>(OvfWriter.java:34) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.ovf.OvfVmWriter.<init>(OvfVmWriter.java:27) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.ovf.OvfManager.ExportVm(OvfManager.java:22) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.VmCommand.UpdateVmInSpm(VmCommand.java:182) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.VmCommand.UpdateVmInSpm(VmCommand.java:156) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.ImportVmCommand.UpdateVmImSpm(ImportVmCommand.java:984) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.ImportVmCommand.endVmRelatedOps(ImportVmCommand.java:974) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.ImportVmCommand.EndImportCommand(ImportVmCommand.java:965) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.ImportVmCommand.endSuccessfully(ImportVmCommand.java:924) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:512) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:459) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.throw vdsBrokerCommand.getVDSReturnValue().getExceptionObject();(CommandBase.java:1374) [engine-bll.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInNewTransaction(TransactionSupport.java:204) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInRequired(TransactionSupport.java:142) [engine-utils.jar:]
        at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:109) [engine-utils.jar:]
        at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:405) [engine-bll.jar:]
        at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:369) [engine-bll.jar:]

# zgrep -in 'EndAction for action type ImportVm threw an exception' var/log/ovirt-engine/engine* | wc -l
22932

Should we retry a sensible number of times before the task the fails?

Version-Release number of selected component (if applicable):
rhevm-3.1.0-50.el6ev.noarch                                 Sun 24 Mar 2013 08:42:37 PM GMT
rhevm-backend-3.1.0-50.el6ev.noarch                         Sun 24 Mar 2013 08:42:30 PM GMT

How reproducible:
Always

Steps to Reproduce:
1. Break /tmp permissions with chmod 1770 etc.
2. Try to import/export, edit the guest in some way that will result in OVF modification etc.
3. Task will loop attempting to end successfully.

Actual results:
Task loops attempting to end successfully.

Expected results:
Task attempts to end successfully a set number of times before failing.

Additional info:

Comment 4 Omer Frenkel 2013-07-28 08:58:45 UTC
Both problems reported here were solved in 3.3:
1. IO error "Permission denied" when creating temp file:
The code handling XMLs changed in 3.3 and this cannot happen anymore, since it's not using temp file anymore.
this was fixed in http://gerrit.ovirt.org/#/c/11221

2. failure when ending Import-Vm command:
The mechanism tries to rerun end logic in case of failure, and when we encounter endless retries we try to have a specific solution (move error-proned code from end logic, or stop retry for specific errors..)
in this case the error-prone code (which is update the SPM with the new imported vm OVF) was moved already in 3.2 to one general place (OvfDataUpdater) that execute this update out of the end logic, so this error should not be seen anymore in 3.2
this was fixed in http://gerrit.ovirt.org/#/c/9340

Comment 5 Michal Skrivanek 2013-07-29 13:09:53 UTC
*** Bug 984503 has been marked as a duplicate of this bug. ***

Comment 6 Michal Skrivanek 2013-07-29 13:12:07 UTC
the specific case is fixed in 3.2, complete fix for other flows in 3.3