Bug 1293689

Summary: Leftovers in compensation table
Product: [oVirt] ovirt-engine Reporter: Simone Tiraboschi <stirabos>
Component: BLL.StorageAssignee: Allon Mureinik <amureini>
Status: CLOSED CURRENTRELEASE QA Contact: Karolína Hajná <khajna>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.6.1.3CC: bugs, emesika, masayag, oourfali, pstehlik, rgolan, sbonazzo, tnisan
Target Milestone: ovirt-3.6.2Flags: oourfali: ovirt-3.6.z?
rule-engine: planning_ack?
oourfali: devel_ack+
pstehlik: testing_ack+
Target Release: 3.6.2.5   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-02-18 11:22:07 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Simone Tiraboschi 2015-12-22 17:17:19 UTC
Description of problem:
After upgrading the engine to 3.6.1 some leftover remain in the business_entity_snapshot table.

engine=# select * from business_entity_snapshot;
          id           |      command_id       |    command_type    |       entity_id       |     entity_type     |    entity_snapshot     | snapshot_class | snapshot_type | insertion_order |     started_at     
-----------------------+-----------------------+--------------------+-----------------------+---------------------+------------------------+----------------+---------------+-----------------+--------------------
 5f84f571-cb2b-4186-b8.| ebe288f6-fdf9-4f27-94.| org.ovirt.engine.c.| 77d0d6b1-376f-4ee6-a8.| org.ovirt.engine.co.| [ "org.ovirt.engine.co.| org.ovirt.engi.|             1 |               1 | 2015-12-22 16:41:4.
.3b-35f30a9d2749       |.28-dc7dbfe3754c       |.ore.bll.storage.Ad.|.a3-c998a20e5d69       |.re.common.businesse.|.re.compat.Guid", {    +|.ne.core.compat.|               |                 |.8.703247+00
                       |                       |.dExistingFileStora.|                       |.ntities.StorageDoma.|   "uuid" : "77d0d6b1-3.|..Guid          |               |                 | 
                       |                       |.geDomainCommand    |                       |.inStatic            |.76f-4ee6-a8a3-c998a20e.|                |               |                 | 
                       |                       |                    |                       |                     |.5d69"                 +|                |               |                 | 
                       |                       |                    |                       |                     | } ]                    |                |               |                 | 
 0b333c6b-23f0-486c-95.| ebe288f6-fdf9-4f27-94.| org.ovirt.engine.c.| 77d0d6b1-376f-4ee6-a8.| org.ovirt.engine.co.| [ "org.ovirt.engine.co.| org.ovirt.engi.|             1 |               2 | 2015-12-22 16:41:4.
.64-7dd625a267db       |.28-dc7dbfe3754c       |.ore.bll.storage.Ad.|.a3-c998a20e5d69       |.re.common.businesse.|.re.compat.Guid", {    +|.ne.core.compat.|               |                 |.8.703247+00
                       |                       |.dExistingFileStora.|                       |.ntities.StorageDoma.|   "uuid" : "77d0d6b1-3.|..Guid          |               |                 | 
                       |                       |.geDomainCommand    |                       |.inDynamic           |.76f-4ee6-a8a3-c998a20e.|                |               |                 | 
                       |                       |                    |                       |                     |.5d69"                 +|                |               |                 | 
                       |                       |                    |                       |                     | } ]                    |                |               |                 | 
 aeab1aab-f3a5-4bfb-8a.| ebe288f6-fdf9-4f27-94.| org.ovirt.engine.c.| 00000000-0000-0000-00.| org.ovirt.engine.co.| {                     +| org.ovirt.engi.|             0 |               3 | 2015-12-22 16:41:4.
.24-119ae5a266b7       |.28-dc7dbfe3754c       |.ore.bll.storage.Ad.|.00-000000000000       |.re.common.businesse.|   "availableDiskSize" .|.ne.core.common.|               |                 |.9.83925+00
                       |                       |.dExistingFileStora.|                       |.ntities.StorageDoma.|.: null,               +|..businessentit.|               |                 | 
                       |                       |.geDomainCommand    |                       |.inDynamic           |   "externalStatus" : ".|.ies.StorageDom.|               |                 | 
                       |                       |                    |                       |                     |.Ok",                  +|.ainDynamic     |               |                 | 
                       |                       |                    |                       |                     |   "containsUnregistere.|                |               |                 | 
                       |                       |                    |                       |                     |.dEntities" : false,   +|                |               |                 | 
                       |                       |                    |                       |                     |   "id" : [ "org.ovirt..|                |               |                 | 
                       |                       |                    |                       |                     |.engine.core.compat.Gui.|                |               |                 | 
                       |                       |                    |                       |                     |.d", {                 +|                |               |                 | 
                       |                       |                    |                       |                     |     "uuid" : "00000000.|                |               |                 | 
                       |                       |                    |                       |                     |.-0000-0000-0000-000000.|                |               |                 | 
                       |                       |                    |                       |                     |.000000"               +|                |               |                 | 
                       |                       |                    |                       |                     |   } ],                +|                |               |                 | 
                       |                       |                    |                       |                     |   "usedDiskSize" : nul.|                |               |                 | 
                       |                       |                    |                       |                     |.l,                    +|                |               |                 | 
                       |                       |                    |                       |                     |   "freeDiskPercent" : .|                |               |                 | 
                       |                       |                    |                       |                     |.0.0                   +|                |               |                 | 
                       |                       |                    |                       |                     | }                      |                |               |                 | 
(3 rows)



The correspondent command in command_entities was already dropped.
engine=# select * from command_entities;
 command_id | command_type | root_command_id | command_parameters | command_params_class | created_at | status | callback_enabled | callback_notified | return_value | return_value_class | job_id | step_id | exec
uted | user_id 
------------+--------------+-----------------+--------------------+----------------------+------------+--------+------------------+-------------------+--------------+--------------------+--------+---------+-----
-----+---------
(0 rows)

engine-setup will wait for that to complete:
          The following compensations have been found running in the system:
          org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand org.ovirt.engine.core.common.businessentities.StorageDomainStatic
          org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand org.ovirt.engine.core.common.businessentities.StorageDomainDynamic
          org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand org.ovirt.engine.core.common.businessentities.StorageDomainDynamic
          Would you like to try to wait for that?
          (Answering "no" will stop the upgrade (Yes, No) no
[ ERROR ] Failed to execute stage 'Setup validation': Upgrade cannot be completed; asynchronous tasks or commands or compensations are still running. Please make sure that there are no running tasks before you continue.


Version-Release number of selected component (if applicable):
ovirt-engine.noarch                        3.6.1.3-1.el7.centos       @ovirt-3.6

How reproducible:
Seen more than once.

Steps to Reproduce:
1. upgrade to 3.6.1
2. run engine-setup again just after that
3.

Actual results:
After upgrade to 3.6.1 we found leftovers in the compensation table, this locks next run of engine-setup

Expected results:
No leftovers in the compensation table

Additional info:

There is no foreign key between business_entity_snapshot and command_entities

Engine-setup will delete the zombie commands but here there is no command to delete.

Comment 1 Simone Tiraboschi 2015-12-22 17:22:06 UTC
Workaround: In order to execute engine-setup again is necessary to manually delete them from the business_entity_snapshot table.

Comment 2 Tal Nisan 2015-12-23 12:53:05 UTC
Although the command in the compensation are storage related, it seems to me that the fact they are not cleared is more infra related, Oved, what do you think?

Comment 3 Oved Ourfali 2015-12-23 12:55:20 UTC
Moti/Eli - should those be deleted as part of task cleaner?

Comment 4 Moti Asayag 2015-12-23 13:00:47 UTC
(In reply to Oved Ourfali from comment #3)
> Moti/Eli - should those be deleted as part of task cleaner?

It makes sense to clear this table as a condition prior to running the upgrade process.

If not, any persisted class/entity in that table which was changed during the upgrade, will fail deserialization from the db.

Comment 5 Eli Mesika 2015-12-24 12:02:07 UTC
(In reply to Moti Asayag from comment #4)
> (In reply to Oved Ourfali from comment #3)
> > Moti/Eli - should those be deleted as part of task cleaner?
> 
> It makes sense to clear this table as a condition prior to running the
> upgrade process.
> 
> If not, any persisted class/entity in that table which was changed during
> the upgrade, will fail deserialization from the db.

Agree

Comment 6 Red Hat Bugzilla Rules Engine 2015-12-30 12:55:47 UTC
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 7 Karolína Hajná 2016-02-12 10:45:34 UTC
Verified in rhevm-3.6.3-0.1.el6.noarch