Bug 1238795

Summary: Upgrade from 3.5.2 to 3.5.3 experiencing database execution error
Product: [Retired] oVirt Reporter: srepetsk <srepetsk>
Component: ovirt-engine-installerAssignee: Sandro Bonazzola <sbonazzo>
Status: CLOSED CURRENTRELEASE QA Contact: Lukas Svaty <lsvaty>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.5CC: bugs, didi, ecohen, emesika, gklein, lsurette, lveyde, oourfali, rbalakri, sbonazzo, stirabos, yeylon, ylavi
Target Milestone: ---Keywords: CodeChange, Regression
Target Release: 3.5.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: integration
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: if an async task with VdcActionType 0 was running when executing engine-setup, the upgrade to 3.5.3 fails with a KeyError exception. Consequence: user may hit an upgrade failure until the async task complete or, if zombie, it's manually removed from the DB. Fix: a key for VdcActionType 0 has been added Result: VdcActionType 0 tasks are now correctly handled.
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-11-09 09:11:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description srepetsk 2015-07-02 15:59:30 UTC
Description of problem:
When running `engine-setup` on the Hosted Engine, the upgrade fails due to a method exception in the database portion of the process.


Version-Release number of selected component (if applicable):
Running oVirt version 3.5.2 and attempting to upgrade to version 3.5.3

How reproducible:


Steps to Reproduce:
1.Run yum upgrade in order to upgrade the repository RPMs
2.yum update ovirt-engine-setup
3.engine-setup
4.Follow the prompts, and attempt to perform the upgrade.

Actual results:

2015-07-01 14:21:36 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2015-07-01 14:21:36 DEBUG otopi.context context.dumpEnvironment:500 ENV OVESETUP_DIALOG/confirmSettings=bool:'True'
2015-07-01 14:21:36 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END
2015-07-01 14:21:36 DEBUG otopi.context context._executeMethod:138 Stage validation METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks.Plugin._validateAsyncTasks
2015-07-01 14:21:36 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks asynctasks._validateAsyncTasks:441 Cleaning async tasks and compensations
2015-07-01 14:21:36 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:164 Database: 'None', Statement: '
                select
                async_tasks.action_type,
                async_tasks.task_id,
                async_tasks.started_at,
                storage_pool.name
                from async_tasks, storage_pool
                where async_tasks.storage_pool_id = storage_pool.id
            ', args: {}
2015-07-01 14:21:36 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:169 Creating own connection
2015-07-01 14:21:36 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:214 Result: [{'started_at': datetime.datetime(2015, 6, 29, 15, 6, 43, 749000, tzinfo=<psycopg2.tz.FixedOffsetTimezone object at 0x293bb10>), 'task_id': 'ad9ebb04-8d73-42b6-b7dd-c619fc2e7403', 'action_type': 0, 'name': 'Default'}, {'started_at': datetime.datetime(2015, 6, 29, 16, 11, 0, 291000, tzinfo=<psycopg2.tz.FixedOffsetTimezone object at 0x2710750>), 'task_id': '6ec188ee-9bb3-4193-aa1a-0a6571ccb3f9', 'action_type': 1011, 'name': 'Default'}]
2015-07-01 14:21:36 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 448, in _validateAsyncTasks
    ) = self._checkRunningTasks()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 341, in _checkRunningTasks
    self._getRunningTasks(dbstatement),
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 202, in _getRunningTasks
    for entry in tasks
KeyError: '0'
2015-07-01 14:21:36 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': '0'

See the full output: https://gist.githubusercontent.com/srepetsk/a0e9b9b6b662273c1e4c/raw/74f6442b934ec2afc30d037207bf94c0f3ebdfa9/gistfile1.txt

Expected results:
The software is supposed to perform the upgrade of the engine

Additional info:

Comment 1 Sandro Bonazzola 2015-07-07 13:52:24 UTC
Looks like the issue here is that the first async task detected has action_type 0 which is not in ASYNC_TASKS_MAP. 

Eli, Simone, is an action_type 0 supposed to exist? If so, how should it be mapped?

it has been returned by:
select
 async_tasks.action_type,
 async_tasks.task_id,
 async_tasks.started_at,
 storage_pool.name
 from async_tasks, storage_pool
 where async_tasks.storage_pool_id = storage_pool.id ;

Comment 2 Eli Mesika 2015-07-08 06:41:16 UTC
(In reply to Sandro Bonazzola from comment #1)
> Looks like the issue here is that the first async task detected has
> action_type 0 which is not in ASYNC_TASKS_MAP. 
> 
> Eli, Simone, is an action_type 0 supposed to exist? If so, how should it be
> mapped?

In VdcActionType enum the first entry is :

 Unknown(0, QuotaDependency.NONE)

So, AFAIK , since there can be entries with this value, it should also be included in the ASYNC_TASKS_MAP in async_tasks_map.py

Comment 3 Sandro Bonazzola 2015-07-08 07:17:11 UTC
(In reply to Eli Mesika from comment #2)
> In VdcActionType enum the first entry is :
> 
>  Unknown(0, QuotaDependency.NONE)
> 
> So, AFAIK , since there can be entries with this value, it should also be
> included in the ASYNC_TASKS_MAP in async_tasks_map.py

In 3.6/master there are 369 values which are in VdcActionType and not in ASYNC_TASKS_MAP. Just to be sure, must all the values in VdcActionType be mapped there?

Comment 4 Eli Mesika 2015-07-08 07:32:05 UTC
(In reply to Sandro Bonazzola from comment #3)
> (In reply to Eli Mesika from comment #2)
> > In VdcActionType enum the first entry is :
> > 
> >  Unknown(0, QuotaDependency.NONE)
> > 
> > So, AFAIK , since there can be entries with this value, it should also be
> > included in the ASYNC_TASKS_MAP in async_tasks_map.py
> 
> In 3.6/master there are 369 values which are in VdcActionType and not in
> ASYNC_TASKS_MAP. Just to be sure, must all the values in VdcActionType be
> mapped there?

No, I don't think so ...
But 0 (Unknown) should be added IMO 

Putting also needinfo on didi who can elaborate on that

Comment 5 Sandro Bonazzola 2015-07-08 07:48:57 UTC
Adding also Oved.

Comment 6 Yedidyah Bar David 2015-07-08 08:04:46 UTC
Sorry, no idea.

Someone who knows async tasks better than me should reply.

It might be better to just drop this file and keep the id/name/desc list in the database, if it's not already there, or find some other way to allow engine-setup read this enum.

Comment 7 Sandro Bonazzola 2015-07-08 08:13:21 UTC
So adding 0 for now, opening an RFE for getting these values in a place where both java and python can read them.

Comment 8 Oved Ourfali 2015-07-27 11:08:50 UTC
What's the open question here?

Comment 9 Eli Mesika 2015-07-27 11:54:51 UTC
AFAIK , there is no open question here ...

Comment 10 Sandro Bonazzola 2015-07-29 07:10:16 UTC
No open question left, discussion moved on bug #1240940

Comment 11 Yaniv Lavi 2015-09-01 11:43:13 UTC
Can you please help us with the SQL code needed to reproduce and test this issue?

Comment 12 Eli Mesika 2015-09-01 12:34:41 UTC
(In reply to Yaniv Dary from comment #11)
> Can you please help us with the SQL code needed to reproduce and test this
> issue?

Please note that the fix was in the python code by adding the UNKNOWN entry, no SQL reproduce is available 

see
https://gerrit.ovirt.org/#/c/43301/2/packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/upgrade/async_tasks_map.py

Comment 13 Lukas Svaty 2015-10-01 13:52:36 UTC
verified in rhevm-3.5.5-0.1.el6ev.noarch