Bug 1161012

Moving back to assigned, as http://gerrit.ovirt.org/35220 broke upgrade from 3.4. see bug 1167769.

To know which version you upgrade from, you can use self.environment[osetupcons.CoreEnv.ORIGINAL_GENERATED_BY_VERSION] - should contain a string, e.g. '3.4.4' or '3.5.0'.

but perhaps better to condition this on existence of the table 'command_entities'.

(In reply to Yedidyah Bar David from comment #3)
> but perhaps better to condition this on existence of the table
> 'command_entities'.

Indeed, and this is what I plan to do.
Thanks!

oVirt 3.5.1 has been released and since this bug is targeted 3.5.1 and in modified state, it should be included in this release.
Please re-target and move nack to modified if this assumption is not valid for this bug.

*** Bug 1190228 has been marked as a duplicate of this bug. ***

any update on this bug? it's currently failing a ci job:
http://jenkins.ovirt.org/job/ovirt-engine_3.5_upgrade-from-3.4_merged/

this ovirt bug was fixed during 3.5.1 cycle and is included in the build, and therefore should be verified.

*** Bug 1200454 has been marked as a duplicate of this bug. ***

I installed version of oVirt: ovirt-engine-3.5.1.1-1.el6.noarch (there isn't patch yet)
I sucessfully added a Host, Storage domain and created 1 VM
After I started making a template from VM, I turned down the network of the Host to create a zombie task.
I removed the host from engine for sure.
Then I want to upgrade the engine to version
ovirt-engine-setup-3.5.2-0.0.master.20150226114525.el6.noarch.rpm (Patched)
but when making the setup of the engine I got error:

        Please confirm installation settings (OK, Cancel) [OK]: 
[ INFO  ] Cleaning async tasks and compensations
[ ERROR ] Failed to execute stage 'Setup validation': 201
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20150311132601-nkrll6.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20150311132820-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed

In log is this error:

2015-03-11 13:28:20 DEBUG otopi.context context._executeMethod:138 Stage validation METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks.Plugin._validateAsyncTasks
2015-03-11 13:28:20 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.upgrade.asynctasks asynctasks._validateAsyncTasks:457 Cleaning async tasks and compensations
2015-03-11 13:28:20 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:164 Database: 'None', Statement: '
                select
                async_tasks.action_type,
                async_tasks.task_id,
                async_tasks.started_at,
                storage_pool.name
                from async_tasks, storage_pool
                where async_tasks.storage_pool_id = storage_pool.id
            ', args: {}
2015-03-11 13:28:20 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:169 Creating own connection
2015-03-11 13:28:20 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:214 Result: [{'started_at': datetime.datetime(2015, 3, 11, 13, 11, 2, 801000, tzinfo=<psycopg2.tz.FixedOffsetTimezone obje
ct at 0x1703750>), 'task_id': '8e43abf3-75a1-411e-b5ab-d7df01eebb1b', 'action_type': 201, 'name': 'Default'}]
2015-03-11 13:28:20 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 464, in _validateAsyncTasks
    ) = self._checkRunningTasks()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 357, in _checkRunningTasks
    self._getRunningTasks(dbstatement),
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 218, in _getRunningTasks
    for entry in tasks
KeyError: 201
2015-03-11 13:28:20 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': 201

attached logs from engine.

Created attachment 1000417 [details]
logs from engine

We found other issues handling AsyncTaskZombieTaskLifeInMinutes to identify the zombies. Now it should work.

Moving to integration, and assigning to you, Simone.

Oved, can you please exact flow that you want QE to test here?
There is a lot of code changes related to this bug and we do not want to miss regressions on this.

(In reply to Yaniv Dary from comment #16)
> Oved, can you please exact flow that you want QE to test here?
> There is a lot of code changes related to this bug and we do not want to
> miss regressions on this.

Exactly what Petr already did when he tried to verify it in the past. See Comment 12.

(In reply to Oved Ourfali from comment #17)
> (In reply to Yaniv Dary from comment #16)
> > Oved, can you please exact flow that you want QE to test here?
> > There is a lot of code changes related to this bug and we do not want to
> > miss regressions on this.
> 
> Exactly what Petr already did when he tried to verify it in the past. See
> Comment 12.

I guess this change goes deeper than just that flow. Please provide list of affected areas to test that changed.

It doesn't go deeper. Petr can contact us if he needs to.

I tried to verificate this bug during the upgrade from latest_av and from vt13.11.
I created a zombie by comment #12 (template). While I did the upgrade of the engine from latest_av to vt14.2: 

          Please confirm installation settings (OK, Cancel) [OK]: 
[ INFO  ] Cleaning async tasks and compensations
          The following system tasks have been found running in the system:
          Task ID:           5d58e691-da49-422e-b27e-122adebf6e38
          Task Name:         AddVmTemplateCommand          
          Task Description:  Adding a template             
          Started at:        30
          DC Name:           Default                       
[ ERROR ] Failed to execute stage 'Setup validation': 
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20150407092843-wk5tdd.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20150407092913-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed

In log:
2015-04-07 09:29:13 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 454, in _validateAsyncTasks
    compensations,
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/upgrade/asynctasks.py", line 305, in _askUserToWaitForTasks
    commands='\n'.join(runningCommands),
TypeError
2015-04-07 09:29:13 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Setup validation': 

While I did the upgrade of the engine from vt13.11 to vt14.2, behaviour was a little different: 

          Please confirm installation settings (OK, Cancel) [OK]: 
[ INFO  ] Cleaning async tasks and compensations
          The following system tasks have been found running in the system:
          Task ID:           cd472cc8-2dff-47d8-b54a-6f6e829898f3
          Task Name:         AddVmTemplateCommand          
          Task Description:  Adding a template             
          Started at:        30
          DC Name:           Default                       
          The following commands have been found running in the system:
          The following compensations have been found running in the system:
          Would you like to try to wait for that?
          (Answering "no" will stop the upgrade (Yes, No)

But, it's a zombie and I have only two options: 
- stop the upgrade
- forever wait

I thing it doesn't recognize that it is a zombie.

logs from both "latest_av" and "vt13.11" attached

Created attachment 1011641 [details]
latest_av

Created attachment 1011642 [details]
vt13.11

Please, verify on VT14.2. VT 13.11 is known to be affected.

I tried two upgrade: latest_av >> vt14.2 and vt13.11 >> vt14.2. So moving to ASSIGNED

Verified with template zombie in vt14.2 (rhevm 3.5.1-0.3.el6ev) 

There is the timeout after that the tasks is marked as zombie.
For testing I changed the value of AsyncTaskZombieTaskLifeInMinutes in table vdc_options from 3000 to 30 minutes.

ovirt 3.5.2 was GA'd. closing current release.