Bug 1555116

Summary: Merge retry validation fails with NPE after previous commit/destroyImage timeout
Product: Red Hat Enterprise Virtualization Manager Reporter: Germano Veit Michel <gveitmic>
Component: ovirt-engineAssignee: Benny Zlotnik <bzlotnik>
Status: CLOSED ERRATA QA Contact: Evelina Shames <eshames>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1.6CC: aefrat, bzlotnik, gveitmic, gwatson, lsurette, marceloltmm, mccarblea, Rhev-m-bugs, rightkicktech, srevivo, tnisan
Target Milestone: ovirt-4.3.3Flags: lsvaty: testing_plan_complete-
Target Release: 4.3.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: ovirt-engine-4.3.3.1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-08 12:37:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
spm vdsm and engine logs
none
bug
none
engine.log
none
user-logs none

Description Germano Veit Michel 2018-03-14 00:59:03 UTC 
Description of problem:

A Live merge completed fine up to the COMMIT/DestroyImage step, which was hung on the SPM due to known storage issues. After some time the engine timed out and suggested the user to retry. But retry fails because the command validation hits a NPE.

So the image was merged and removed from the chain successfully, just the SPM part of removing the image from the SD failed. From what I understand, the NPE comes from a loop in the image chain that was created on the failed COMMIT phase (previous leaf, which failed to be removed from the SD, contains parentid=NULL in the DB).

In more details:

ENGINE: Live merge fails timeout on last step (COMMIT, DestroyImage)

=> EXTEND
2018-03-08 12:15:16,171+13 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Executing Live Merge command step 'EXTEND'
2018-03-08 12:15:16,242+13 INFO  [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-5-thread-9) [c23b59cc-bb79-4486-b031-a09a478e4959] Base and top image sizes are the same; no image size update required

=> MERGE
2018-03-08 12:15:17,257+13 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [c23b59cc-bb79-4486-b031-a09a478e4959] Executing Live Merge command step 'MERGE'
2018-03-08 12:16:12,553+13 INFO  [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler1) [c23b59cc-bb79-4486-b031-a09a478e4959] Merge command (jobId = f4957a7f-7c42-4e8b-acb5-dad3c2292634) has completed for images 'ee0fa36e-c011-452e-b646-e95a1ecb945e'..'bd5e7e03-d3f3-458f-8b5f-f53536df8c10'

=> MERGE_STATUS
2018-03-08 12:16:13,568+13 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Executing Live Merge command step 'MERGE_STATUS'
2018-03-08 12:16:13,662+13 INFO  [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-8) [c23b59cc-bb79-4486-b031-a09a478e4959] Successfully removed volume(s): [bd5e7e03-d3f3-458f-8b5f-f53536df8c10]

=> COMMIT
2018-03-08 12:16:13,662+13 INFO  [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-8) [c23b59cc-bb79-4486-b031-a09a478e4959] Volume merge type 'COMMIT'
2018-03-08 12:16:14,656+13 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (pool-5-thread-4) [c23b59cc-bb79-4486-b031-a09a478e4959] START, DestroyImageVDSCommand( DestroyImageVDSCommandParameters:{runAsync='true', storagePoolId='58117215-0261-0167-0311-0000000002b6', ignoreFailoverLimit='false', storageDomainId='0b0107b2-9d2a-4871-b4d6-76bdffba84cd', imageGroupId='70668240-d413-45a8-b081-c0aeb5b7466f', imageId='00000000-0000-0000-0000-000000000000', imageList='[bd5e7e03-d3f3-458f-8b5f-f53536df8c10]', postZero='false', force='false'}), log id: 4a53b8c9

.... 50 hours ...

2018-03-10 14:15:15,555+13 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Command 'RemoveSnapshot' (id: '03911358-50a8-4b26-8b56-3618dc5f830d') waiting on child command id: '4a0c31d5-12a5-4fd4-8880-8831a2e0f08b' type:'RemoveSnapshotSingleDiskLive' to complete
2018-03-10 14:15:15,555+13 WARN  [org.ovirt.engine.core.bll.tasks.CommandsRepository] (DefaultQuartzScheduler10) [] Marking expired command as Failed: command 'RemoveSnapshotSingleDiskLive (4a0c31d5-12a5-4fd4-8880-8831a2e0f08b)' that started at 'Thu Mar 08 12:15:15 NZDT 2018' has been marked as Failed.
2018-03-10 14:15:26,533+13 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Snapshot '2859c957-ac4e-48ab-be7c-6bc940fd21dc' images 'bd5e7e03-d3f3-458f-8b5f-f53536df8c10'..'ee0fa36e-c011-452e-b646-e95a1ecb945e' merged, but volume removal failed. Some or all of the following volumes may be orphaned: [bd5e7e03-d3f3-458f-8b5f-f53536df8c10]. Please retry Live Merge on the snapshot to complete the operation.
2018-03-10 14:15:27,573+13 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (DefaultQuartzScheduler5) [c23b59cc-bb79-4486-b031-a09a478e4959] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.



SPM: task doesnt complete (widespread storage access problems)

2018-03-08 12:15:44,991+1300 INFO  (tasks/9) [storage.ThreadPool.WorkerThread] START task 1828df85-8c90-49c5-8cba-fe958e5f14de (cmd=<bound method Task.commit of <storage.task.Task instance at 0x7f56f419eb48>>, args=None) (threadPool:208) 
...
2018-03-08 14:00:58,878+1300 INFO  (jsonrpc/5) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'1828df85-8c90-49c5-8cba-fe958e5f14de': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '1828df85-8c90-49c5-8cba-fe958e5f14de'}, .... } ...



ENGINE RETRY: RemoveSnapshotCommand Validation fails with NPE:

2018-03-13 12:17:32,338+13 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-16) [c356ec77-0dfa-4ea1-960b-982b9b24cdef] Error during ValidateFailure.: java.lang.NullPointerException
        at org.ovirt.engine.core.bll.validator.storage.StorageDomainValidator.getTotalSizeForMerge(StorageDomainValidator.java:121) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.StorageDomainValidator.hasSpaceForMerge(StorageDomainValidator.java:207) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.MultipleStorageDomainsValidator.lambda$allDomainsHaveSpaceForMerge$0(MultipleStorageDomainsValidator.java:128) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.MultipleStorageDomainsValidator.validOrFirstFailure(MultipleStorageDomainsValidator.java:190) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.MultipleStorageDomainsValidator.allDomainsHaveSpaceForMerge(MultipleStorageDomainsValidator.java:125) [bll.jar:]
        at org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand.validateStorageDomains(RemoveSnapshotCommand.java:381) [bll.jar:]
        at org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand.validate(RemoveSnapshotCommand.java:359) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:836) [bll.jar:]

bd5e7e03 was the active image (leaf) before the merge. The merge succeeded and bd5e7e03 was removed from the chain. The leaf is now the image which was the parent of bd5e7e03, which is ee0fa36e. bd5e7e03 is still in the DB as COMMIT did not finish, but its parent was set to 00000000. There is now a loop in the chain in the DB which I understand causes the NULL subchain, trigerring the NPE.


              image_guid              |               parentid               | imagestatus 
--------------------------------------+--------------------------------------+-------------
 [ORPHAN]  --- this parent NULL triggers the NPE due to a null subchain?
 bd5e7e03-d3f3-458f-8b5f-f53536df8c10 | 00000000-0000-0000-0000-000000000000 |           4

 [LEAF]
 ee0fa36e-c011-452e-b646-e95a1ecb945e | 87284902-3827-4541-9465-3f7706d2a6a4 |           1
 87284902-3827-4541-9465-3f7706d2a6a4 | fa2d6a6b-dc17-48e9-a573-2e87b853e53a |           1
 fa2d6a6b-dc17-48e9-a573-2e87b853e53a | 605a5807-4405-4b27-981a-6b137766a502 |           1
 605a5807-4405-4b27-981a-6b137766a502 | 6eac6a71-7cff-4c77-bdb1-3082bc2b91e3 |           1
 6eac6a71-7cff-4c77-bdb1-3082bc2b91e3 | 00000000-0000-0000-0000-000000000000 |           1
 [BASE]

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.6.2-0.1.el7.noarch

How reproducible:
Unknown
Comment 1 Germano Veit Michel 2018-03-14 01:07:55 UTC 
By forcing this in 4.1.9 (db manipulation) I also get a NPE on the exact same spot.
Comment 4 Germano Veit Michel 2018-03-14 05:47:45 UTC 
Not sure why I called this a loop, there is no loop. Its just a single image chain when trying to remove bd5e7e03, which doesn't make sense and is not well handled.
Comment 9 Ala Hino 2018-03-15 11:22:56 UTC 
Moving to 4.2.4 for now as the issue isn't reproducible and need more analysis.
Comment 10 Marcelo Leandro 2018-05-22 14:46:34 UTC 
I am have the same problem in a lot of vms.

attached logs the last vm with problem.

how a can resolve this problem? i need remove de database?

Thanks.
Comment 11 Marcelo Leandro 2018-05-22 14:47:06 UTC 
Created attachment 1440212 [details]
spm vdsm and engine logs
Comment 12 Marcelo Leandro 2018-05-22 15:10:37 UTC 
(In reply to Marcelo Leandro from comment #10)
> I am have the same problem in a lot of vms.
> 
> attached logs the last vm with problem.
> 
> how a can resolve this problem? i need remove de database?
> 
> Thanks.

I think the soluction is similar this :

https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15

but in this case I need the snapshot id , but in the snapshot tab dont have ids.
Comment 13 Marcelo Leandro 2018-05-22 15:35:52 UTC 
(In reply to Marcelo Leandro from comment #12)
> (In reply to Marcelo Leandro from comment #10)
> > I am have the same problem in a lot of vms.
> > 
> > attached logs the last vm with problem.
> > 
> > how a can resolve this problem? i need remove de database?
> > 
> > Thanks.
> 
> I think the soluction is similar this :
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> 
> but in this case I need the snapshot id , but in the snapshot tab dont have
> ids.

I found the snapshot id, can I apply the solution mentioned in comment 12 ?

select snapshot_id,description from snapshots where vm_id = 'c847130e-230d-47ee-a124-54d46e1dd928';
             snapshot_id              | description 
--------------------------------------+-------------
 9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
 a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup

select * from images where vm_snapshot_id = 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
              image_guid              |     creation_date      |     size      |               it_guid                |               parentid               | imagestatus |        lastmodified        |          
  vm_snapshot_id            | volume_type | volume_format |            image_group_id            |         _create_date          |         _update_date          | active | volume_classification | qcow_compat 
--------------------------------------+------------------------+---------------+--------------------------------------+--------------------------------------+-------------+----------------------------+----------
----------------------------+-------------+---------------+--------------------------------------+-------------------------------+-------------------------------+--------+-----------------------+-------------
 db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 | 1209033293824 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:06.869-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 | 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
 a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 |  107374182400 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:06.564-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 | 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
 2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 | 1717986918400 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:07.098-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 | 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
 85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 |  536870912000 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:06.196-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 | 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2


Thanks.
Comment 14 Ala Hino 2018-05-22 15:53:11 UTC 
(In reply to Marcelo Leandro from comment #13)
> (In reply to Marcelo Leandro from comment #12)
> > (In reply to Marcelo Leandro from comment #10)
> > > I am have the same problem in a lot of vms.
> > > 
> > > attached logs the last vm with problem.
> > > 
> > > how a can resolve this problem? i need remove de database?
> > > 
> > > Thanks.
> > 
> > I think the soluction is similar this :
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > 
> > but in this case I need the snapshot id , but in the snapshot tab dont have
> > ids.
> 
> I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> 
> select snapshot_id,description from snapshots where vm_id =
> 'c847130e-230d-47ee-a124-54d46e1dd928';
>              snapshot_id              | description 
> --------------------------------------+-------------
>  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
>  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> 
> select * from images where vm_snapshot_id =
> 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
>               image_guid              |     creation_date      |     size   
> |               it_guid                |               parentid             
> | imagestatus |        lastmodified        |          
>   vm_snapshot_id            | volume_type | volume_format |           
> image_group_id            |         _create_date          |        
> _update_date          | active | volume_classification | qcow_compat 
> --------------------------------------+------------------------+-------------
> --+--------------------------------------+-----------------------------------
> ---+-------------+----------------------------+----------
> ----------------------------+-------------+---------------+------------------
> --------------------+-------------------------------+------------------------
> -------+--------+-----------------------+-------------
>  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:06.869-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
>  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> 107374182400 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:06.564-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
>  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:07.098-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
>  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> 536870912000 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:06.196-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> 
> 
> Thanks.

What's the error that you get when retrying live merge?
Comment 15 Marcelo Leandro 2018-05-22 15:59:23 UTC 
(In reply to Ala Hino from comment #14)
> (In reply to Marcelo Leandro from comment #13)
> > (In reply to Marcelo Leandro from comment #12)
> > > (In reply to Marcelo Leandro from comment #10)
> > > > I am have the same problem in a lot of vms.
> > > > 
> > > > attached logs the last vm with problem.
> > > > 
> > > > how a can resolve this problem? i need remove de database?
> > > > 
> > > > Thanks.
> > > 
> > > I think the soluction is similar this :
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > 
> > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > ids.
> > 
> > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > 
> > select snapshot_id,description from snapshots where vm_id =
> > 'c847130e-230d-47ee-a124-54d46e1dd928';
> >              snapshot_id              | description 
> > --------------------------------------+-------------
> >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > 
> > select * from images where vm_snapshot_id =
> > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> >               image_guid              |     creation_date      |     size   
> > |               it_guid                |               parentid             
> > | imagestatus |        lastmodified        |          
> >   vm_snapshot_id            | volume_type | volume_format |           
> > image_group_id            |         _create_date          |        
> > _update_date          | active | volume_classification | qcow_compat 
> > --------------------------------------+------------------------+-------------
> > --+--------------------------------------+-----------------------------------
> > ---+-------------+----------------------------+----------
> > ----------------------------+-------------+---------------+------------------
> > --------------------+-------------------------------+------------------------
> > -------+--------+-----------------------+-------------
> >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.869-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.564-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:07.098-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.196-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > 
> > 
> > Thanks.
> 
> What's the error that you get when retrying live merge?

in the gui show this.

Error while executing action:

GOV-CAMARUPIM:

    General command validation failure.
Comment 16 mccarblea 2018-06-21 09:46:31 UTC 
Created attachment 1453390 [details]
bug

to soon to tell
Comment 17 Marcelo Leandro 2018-06-26 12:14:43 UTC 
(In reply to Ala Hino from comment #14)
> (In reply to Marcelo Leandro from comment #13)
> > (In reply to Marcelo Leandro from comment #12)
> > > (In reply to Marcelo Leandro from comment #10)
> > > > I am have the same problem in a lot of vms.
> > > > 
> > > > attached logs the last vm with problem.
> > > > 
> > > > how a can resolve this problem? i need remove de database?
> > > > 
> > > > Thanks.
> > > 
> > > I think the soluction is similar this :
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > 
> > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > ids.
> > 
> > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > 
> > select snapshot_id,description from snapshots where vm_id =
> > 'c847130e-230d-47ee-a124-54d46e1dd928';
> >              snapshot_id              | description 
> > --------------------------------------+-------------
> >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > 
> > select * from images where vm_snapshot_id =
> > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> >               image_guid              |     creation_date      |     size   
> > |               it_guid                |               parentid             
> > | imagestatus |        lastmodified        |          
> >   vm_snapshot_id            | volume_type | volume_format |           
> > image_group_id            |         _create_date          |        
> > _update_date          | active | volume_classification | qcow_compat 
> > --------------------------------------+------------------------+-------------
> > --+--------------------------------------+-----------------------------------
> > ---+-------------+----------------------------+----------
> > ----------------------------+-------------+---------------+------------------
> > --------------------+-------------------------------+------------------------
> > -------+--------+-----------------------+-------------
> >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.869-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.564-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:07.098-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.196-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > 
> > 
> > Thanks.
> 
> What's the error that you get when retrying live merge?

Can you help to delete this problem snapshots?

Very Thanks.
Comment 18 Yaniv Lavi 2018-08-13 08:47:47 UTC 
Please reply to comment $17.
Comment 19 Ala Hino 2018-08-14 08:34:04 UTC 
(In reply to Marcelo Leandro from comment #17)
> (In reply to Ala Hino from comment #14)
> > (In reply to Marcelo Leandro from comment #13)
> > > (In reply to Marcelo Leandro from comment #12)
> > > > (In reply to Marcelo Leandro from comment #10)
> > > > > I am have the same problem in a lot of vms.
> > > > > 
> > > > > attached logs the last vm with problem.
> > > > > 
> > > > > how a can resolve this problem? i need remove de database?
> > > > > 
> > > > > Thanks.
> > > > 
> > > > I think the soluction is similar this :
> > > > 
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > > 
> > > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > > ids.
> > > 
> > > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > > 
> > > select snapshot_id,description from snapshots where vm_id =
> > > 'c847130e-230d-47ee-a124-54d46e1dd928';
> > >              snapshot_id              | description 
> > > --------------------------------------+-------------
> > >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> > >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > > 
> > > select * from images where vm_snapshot_id =
> > > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> > >               image_guid              |     creation_date      |     size   
> > > |               it_guid                |               parentid             
> > > | imagestatus |        lastmodified        |          
> > >   vm_snapshot_id            | volume_type | volume_format |           
> > > image_group_id            |         _create_date          |        
> > > _update_date          | active | volume_classification | qcow_compat 
> > > --------------------------------------+------------------------+-------------
> > > --+--------------------------------------+-----------------------------------
> > > ---+-------------+----------------------------+----------
> > > ----------------------------+-------------+---------------+------------------
> > > --------------------+-------------------------------+------------------------
> > > -------+--------+-----------------------+-------------
> > >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:06.869-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> > >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:06.564-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> > >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:07.098-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> > >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:06.196-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > > 
> > > 
> > > Thanks.
> > 
> > What's the error that you get when retrying live merge?
> 
> Can you help to delete this problem snapshots?
> 
> Very Thanks.

Hi,

Do you still need guidelines to delete the snapshot?
Comment 20 Marcelo Leandro 2018-08-14 09:59:44 UTC 
(In reply to Ala Hino from comment #19)
> (In reply to Marcelo Leandro from comment #17)
> > (In reply to Ala Hino from comment #14)
> > > (In reply to Marcelo Leandro from comment #13)
> > > > (In reply to Marcelo Leandro from comment #12)
> > > > > (In reply to Marcelo Leandro from comment #10)
> > > > > > I am have the same problem in a lot of vms.
> > > > > > 
> > > > > > attached logs the last vm with problem.
> > > > > > 
> > > > > > how a can resolve this problem? i need remove de database?
> > > > > > 
> > > > > > Thanks.
> > > > > 
> > > > > I think the soluction is similar this :
> > > > > 
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > > > 
> > > > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > > > ids.
> > > > 
> > > > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > > > 
> > > > select snapshot_id,description from snapshots where vm_id =
> > > > 'c847130e-230d-47ee-a124-54d46e1dd928';
> > > >              snapshot_id              | description 
> > > > --------------------------------------+-------------
> > > >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> > > >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > > > 
> > > > select * from images where vm_snapshot_id =
> > > > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> > > >               image_guid              |     creation_date      |     size   
> > > > |               it_guid                |               parentid             
> > > > | imagestatus |        lastmodified        |          
> > > >   vm_snapshot_id            | volume_type | volume_format |           
> > > > image_group_id            |         _create_date          |        
> > > > _update_date          | active | volume_classification | qcow_compat 
> > > > --------------------------------------+------------------------+-------------
> > > > --+--------------------------------------+-----------------------------------
> > > > ---+-------------+----------------------------+----------
> > > > ----------------------------+-------------+---------------+------------------
> > > > --------------------+-------------------------------+------------------------
> > > > -------+--------+-----------------------+-------------
> > > >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > > > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.869-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > > > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> > > >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > > > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.564-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > > > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> > > >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > > > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:07.098-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > > > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> > > >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > > > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.196-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > > > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > > > 
> > > > 
> > > > Thanks.
> > > 
> > > What's the error that you get when retrying live merge?
> > 
> > Can you help to delete this problem snapshots?
> > 
> > Very Thanks.
> 
> Hi,
> 
> Do you still need guidelines to delete the snapshot?

Good morning, 
Yes, I still need.

Very thanks.
Comment 21 Marcelo Leandro 2018-08-18 07:01:59 UTC 
(In reply to Ala Hino from comment #19)
> (In reply to Marcelo Leandro from comment #17)
> > (In reply to Ala Hino from comment #14)
> > > (In reply to Marcelo Leandro from comment #13)
> > > > (In reply to Marcelo Leandro from comment #12)
> > > > > (In reply to Marcelo Leandro from comment #10)
> > > > > > I am have the same problem in a lot of vms.
> > > > > > 
> > > > > > attached logs the last vm with problem.
> > > > > > 
> > > > > > how a can resolve this problem? i need remove de database?
> > > > > > 
> > > > > > Thanks.
> > > > > 
> > > > > I think the soluction is similar this :
> > > > > 
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > > > 
> > > > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > > > ids.
> > > > 
> > > > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > > > 
> > > > select snapshot_id,description from snapshots where vm_id =
> > > > 'c847130e-230d-47ee-a124-54d46e1dd928';
> > > >              snapshot_id              | description 
> > > > --------------------------------------+-------------
> > > >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> > > >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > > > 
> > > > select * from images where vm_snapshot_id =
> > > > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> > > >               image_guid              |     creation_date      |     size   
> > > > |               it_guid                |               parentid             
> > > > | imagestatus |        lastmodified        |          
> > > >   vm_snapshot_id            | volume_type | volume_format |           
> > > > image_group_id            |         _create_date          |        
> > > > _update_date          | active | volume_classification | qcow_compat 
> > > > --------------------------------------+------------------------+-------------
> > > > --+--------------------------------------+-----------------------------------
> > > > ---+-------------+----------------------------+----------
> > > > ----------------------------+-------------+---------------+------------------
> > > > --------------------+-------------------------------+------------------------
> > > > -------+--------+-----------------------+-------------
> > > >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > > > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.869-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > > > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> > > >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > > > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.564-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > > > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> > > >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > > > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:07.098-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > > > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> > > >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > > > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.196-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > > > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > > > 
> > > > 
> > > > Thanks.
> > > 
> > > What's the error that you get when retrying live merge?
> > 
> > Can you help to delete this problem snapshots?
> > 
> > Very Thanks.
> 
> Hi,
> 
> Do you still need guidelines to delete the snapshot?
Hi Ala Hino,

Can you help me with guidelines to delete the snapshot ?
Comment 22 Ala Hino 2018-08-28 09:01:54 UTC 
Marcelo,

Let's please continue this conversation on bug 1574631, originally reported by you.
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1574631#c45
Comment 23 Alex Kaouris 2018-09-10 15:46:56 UTC 
Created attachment 1482199 [details]
engine.log

I might have hit the same case. 
When I try to delete a snapshot to several VMs I get: General command validation failure.
Comment 25 Elad 2019-01-24 09:33:33 UTC 
Created attachment 1523044 [details]
user-logs

This was seen also by a user, attaching the logs
Comment 26 Sandro Bonazzola 2019-01-28 09:40:28 UTC 
This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.
Comment 28 Elad 2019-03-27 14:43:35 UTC 
Beny, can you please advise on verification steps?
Comment 29 Benny Zlotnik 2019-03-27 14:47:11 UTC 
1. Create multiple snapshots for a disk
2. Break the chain by taking one the images in the chain and set its parent to an empty guid (manually, in the DB)
3. Run live merge
You should see an error in the form of: "The requested snapshot has an invalid parent, either fix the snapshot definition or remove it manually to complete the process."
Comment 31 Evelina Shames 2019-04-07 12:47:09 UTC 
Verified on ovirt-engine-4.3.3.2-0.1.el7.noarch
Comment 33 errata-xmlrpc 2019-05-08 12:37:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085
Comment 34 Daniel Gur 2019-08-28 13:14:03 UTC 
sync2jira
Comment 35 Daniel Gur 2019-08-28 13:18:18 UTC 
sync2jira