Bugzilla will be upgraded to version 5.0 on December 2, 2018. The outage period for the upgrade will start at 0:00 UTC and have a duration of 12 hours
Bug 970518 - engine: can't clear LSM tasks that failed during upgrade
engine: can't clear LSM tasks that failed during upgrade
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
3.2.0
x86_64 Linux
unspecified Severity high
: ---
: 3.3.0
Assigned To: Nobody's working on this, feel free to take it
Pavel Stehlik
infra
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-06-04 04:44 EDT by Dafna Ron
Modified: 2016-02-10 14:11 EST (History)
9 users (show)

See Also:
Fixed In Version: is2
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-21 17:18:42 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Infra
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
logs (1.34 MB, application/x-gzip)
2013-06-04 04:44 EDT, Dafna Ron
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
oVirt gerrit 15133 None None None Never

  None (edit)
Description Dafna Ron 2013-06-04 04:44:40 EDT
Created attachment 756659 [details]
logs

Description of problem:

I ran some tests on LSM and had some tasks in async table that were not properly cleared. 
when I tried to upgrade my setup the installer detects the tasks and fails to clear them (so upgrade cannot be continued without GSS involvement. 

However, when I ran the taskcleaner manually (not with upgrade script) I was able to clear the tasks. 

Version-Release number of selected component (if applicable):

sf17.2 -> 17.4

How reproducible:

100%

Steps to Reproduce:
1. in sf17.2 run a vm as paused and start a LSM on the vm 
2. we will fail to clear the task with ArrayOutOfBound -1 error.
3. try to upgrade to sf17.4
4. after engine fails to clear the tasks, run taskcleaner manually and try to upgrade again

Actual results:

upgrade detects the tasks but fails to clear them
taskcleaner ran manually was able to clear teh tasks and I was able to upgrade the setup. 

Expected results:

we should be able to clear the tasks during upgrade. 

Additional info:


Info: The following tasks have been found running in the system: 

System Tasks:

---- Task ID: b230ab47-21d8-487f-a190-6a1278a58b52 ------- 
Task Name: LiveMigrateDisk               
Task Description: Migrating Live Disk           
Started at: 2013-06-03 15:51:29.047+03    
DC Name: iSCSI                         


---- Task ID: 01b8016a-706c-4681-91bc-91b98bd38ee4 ------- 
Task Name: LiveMigrateDisk               
Task Description: Migrating Live Disk           
Started at: 2013-06-02 14:48:13.137+03    
DC Name: iSCSI                         




[ Jun 04 11:29:52 ] Would you like to proceed and try to stop tasks automatically?
(Answering 'no' will stop the upgrade)? (yes|no): n
Starting DB related services...                      [ DONE ]
Starting ovirt-engine service...                     [ DONE ]

There are still running tasks: 

System Tasks:

---- Task ID: b230ab47-21d8-487f-a190-6a1278a58b52 ------- 
Task Name: LiveMigrateDisk               
Task Description: Migrating Live Disk           
Started at: 2013-06-03 15:51:29.047+03    
DC Name: iSCSI                         


---- Task ID: 01b8016a-706c-4681-91bc-91b98bd38ee4 ------- 
Task Name: LiveMigrateDisk               
Task Description: Migrating Live Disk           
Started at: 2013-06-02 14:48:13.137+03    
DC Name: iSCSI                         



Please make sure that there are no running system tasks before you continue. Stopping upgrade.
Starting ovirt-engine service...                     [ DONE ]
Error: Upgrade failed.
please check log at /var/log/ovirt-engine/ovirt-engine-upgrade_2013_06_04_11_26_05.log
[root@Dafna-32 tmp]# 


ngine=# SELECT * from async_tasks;
               task_id                | action_type | status | result |                               action_parameters                               |                      action_params_class                      |               step_i
d                |              command_id              |         started_at         |           storage_pool_id            | task_type |                                task_parameters                                |                   
    task_params_class                       
--------------------------------------+-------------+--------+--------+-------------------------------------------------------------------------------+---------------------------------------------------------------+---------------------
-----------------+--------------------------------------+----------------------------+--------------------------------------+-----------+-------------------------------------------------------------------------------+-------------------
--------------------------------------------
 b230ab47-21d8-487f-a190-6a1278a58b52 |        1010 |      2 |      0 | {                                                                             | org.ovirt.engine.core.common.action.LiveMigrateDiskParameters | 149a6f40-e139-4ff4-b
b6e-eea172e1cba5 | 1f960ca6-9f89-4e14-952b-e20466ec37e9 | 2013-06-03 15:51:29.047+03 | 7fd33b43-a9f4-4eb7-a885-e9583a929ceb |         8 | {                                                                             | org.ovirt.engine.c
ore.common.action.LiveMigrateDiskParameters
                                                                      :   "@class" : "org.ovirt.engine.core.common.action.LiveMigrateDiskParameters",                                                                                       
                                                                                                                                        :   "@class" : "org.ovirt.engine.core.common.action.LiveMigrateDiskParameters",   
                                                                      :   "commandId" : [ "org.ovirt.engine.core.compat.Guid", {                                




[root@Dafna-32 dbutils]# ./taskcleaner.sh -s localhost -p 5432 -d engine -u engine -R -l taskcleaner.sh.log
 This will remove all async_tasks table content!!! 
Caution, this operation should be used with care. Please contact support prior to running this command
Are you sure you want to proceed? [y/n]
y
NOTICE:  truncate cascades to table "async_tasks_entities"
TRUNCATE TABLE

[root@Dafna-32 dbutils]# psql -U postgres engine
Password for user postgres: 
psql (8.4.13)
Type "help" for help.

engine=# SELECT * from async_tasks;
 task_id | action_type | status | result | action_parameters | action_params_class | step_id | command_id | started_at | storage_pool_id | task_type | task_parameters | task_params_class 
---------+-------------+--------+--------+-------------------+---------------------+---------+------------+------------+-----------------+-----------+-----------------+-------------------
(0 rows)

engine=#
Comment 1 Allon Mureinik 2013-07-09 09:11:52 EDT
The underlying ArrayIndexOutOfBounds issues was solved by the patch mentioned in the external trackers.
Now, the upgrade should succeed too.

If the issue is not solved, please reopen and attach current logs.
Comment 3 Aharon Canan 2013-09-10 09:04:33 EDT
As we can't LSM paused VM in 3.3, this issue is irrelevant from storage perspective for 3.3

marking as verified.
Comment 4 Itamar Heim 2014-01-21 17:18:42 EST
Closing - RHEV 3.3 Released
Comment 5 Itamar Heim 2014-01-21 17:25:02 EST
Closing - RHEV 3.3 Released

Note You need to log in before you can comment on or make changes to this bug.