1555116 – Merge retry validation fails with NPE after previous commit/destroyImage timeout

Bug 1555116 - Merge retry validation fails with NPE after previous commit/destroyImage timeout

Summary: Merge retry validation fails with NPE after previous commit/destroyImage timeout

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Virtualization Manager
Classification:	Red Hat
Component:	ovirt-engine
Sub Component:
Version:	4.1.6
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	urgent
Target Milestone:	ovirt-4.3.3
Target Release:	4.3.0
Assignee:	Benny Zlotnik
QA Contact:	Evelina Shames
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-03-14 00:59 UTC by Germano Veit Michel
Modified:	2021-06-10 15:27 UTC (History)
CC List:	11 users (show)
Fixed In Version:	ovirt-engine-4.3.3.1
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-05-08 12:37:22 UTC
oVirt Team:	Storage
Target Upstream Version:
Embargoed:
Flags:	lsvaty: testing_plan_complete-

Attachments	(Terms of Use)
spm vdsm and engine logs (1.38 MB, application/x-xz) 2018-05-22 14:47 UTC, Marcelo Leandro	no flags	Details
bug (76 bytes, text/plain) 2018-06-21 09:46 UTC, mccarblea	no flags	Details
engine.log (15.58 KB, text/plain) 2018-09-10 15:46 UTC, Alex Kaouris	no flags	Details
user-logs (1.06 MB, application/gzip) 2019-01-24 09:33 UTC, Elad	no flags	Details
View All

Links
System	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	3380701	None	None	None	2018-03-14 01:23:39 UTC
Red Hat Product Errata	RHEA-2019:1085	None	None	None	2019-05-08 12:37:42 UTC
oVirt gerrit	98666	master	MERGED	core: fail validation when trying to remove snapshot	2020-12-24 17:43:45 UTC
oVirt gerrit	98865	ovirt-engine-4.3	MERGED	core: fail validation when trying to remove snapshot	2020-12-24 17:43:45 UTC

Description Germano Veit Michel 2018-03-14 00:59:03 UTC

Description of problem:

A Live merge completed fine up to the COMMIT/DestroyImage step, which was hung on the SPM due to known storage issues. After some time the engine timed out and suggested the user to retry. But retry fails because the command validation hits a NPE.

So the image was merged and removed from the chain successfully, just the SPM part of removing the image from the SD failed. From what I understand, the NPE comes from a loop in the image chain that was created on the failed COMMIT phase (previous leaf, which failed to be removed from the SD, contains parentid=NULL in the DB).

In more details:

ENGINE: Live merge fails timeout on last step (COMMIT, DestroyImage)

=> EXTEND
2018-03-08 12:15:16,171+13 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Executing Live Merge command step 'EXTEND'
2018-03-08 12:15:16,242+13 INFO  [org.ovirt.engine.core.bll.MergeExtendCommand] (pool-5-thread-9) [c23b59cc-bb79-4486-b031-a09a478e4959] Base and top image sizes are the same; no image size update required

=> MERGE
2018-03-08 12:15:17,257+13 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler9) [c23b59cc-bb79-4486-b031-a09a478e4959] Executing Live Merge command step 'MERGE'
2018-03-08 12:16:12,553+13 INFO  [org.ovirt.engine.core.bll.MergeCommandCallback] (DefaultQuartzScheduler1) [c23b59cc-bb79-4486-b031-a09a478e4959] Merge command (jobId = f4957a7f-7c42-4e8b-acb5-dad3c2292634) has completed for images 'ee0fa36e-c011-452e-b646-e95a1ecb945e'..'bd5e7e03-d3f3-458f-8b5f-f53536df8c10'

=> MERGE_STATUS
2018-03-08 12:16:13,568+13 INFO  [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Executing Live Merge command step 'MERGE_STATUS'
2018-03-08 12:16:13,662+13 INFO  [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-8) [c23b59cc-bb79-4486-b031-a09a478e4959] Successfully removed volume(s): [bd5e7e03-d3f3-458f-8b5f-f53536df8c10]

=> COMMIT
2018-03-08 12:16:13,662+13 INFO  [org.ovirt.engine.core.bll.MergeStatusCommand] (pool-5-thread-8) [c23b59cc-bb79-4486-b031-a09a478e4959] Volume merge type 'COMMIT'
2018-03-08 12:16:14,656+13 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyImageVDSCommand] (pool-5-thread-4) [c23b59cc-bb79-4486-b031-a09a478e4959] START, DestroyImageVDSCommand( DestroyImageVDSCommandParameters:{runAsync='true', storagePoolId='58117215-0261-0167-0311-0000000002b6', ignoreFailoverLimit='false', storageDomainId='0b0107b2-9d2a-4871-b4d6-76bdffba84cd', imageGroupId='70668240-d413-45a8-b081-c0aeb5b7466f', imageId='00000000-0000-0000-0000-000000000000', imageList='[bd5e7e03-d3f3-458f-8b5f-f53536df8c10]', postZero='false', force='false'}), log id: 4a53b8c9

.... 50 hours ...

2018-03-10 14:15:15,555+13 INFO  [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Command 'RemoveSnapshot' (id: '03911358-50a8-4b26-8b56-3618dc5f830d') waiting on child command id: '4a0c31d5-12a5-4fd4-8880-8831a2e0f08b' type:'RemoveSnapshotSingleDiskLive' to complete
2018-03-10 14:15:15,555+13 WARN  [org.ovirt.engine.core.bll.tasks.CommandsRepository] (DefaultQuartzScheduler10) [] Marking expired command as Failed: command 'RemoveSnapshotSingleDiskLive (4a0c31d5-12a5-4fd4-8880-8831a2e0f08b)' that started at 'Thu Mar 08 12:15:15 NZDT 2018' has been marked as Failed.
2018-03-10 14:15:26,533+13 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (DefaultQuartzScheduler10) [c23b59cc-bb79-4486-b031-a09a478e4959] Snapshot '2859c957-ac4e-48ab-be7c-6bc940fd21dc' images 'bd5e7e03-d3f3-458f-8b5f-f53536df8c10'..'ee0fa36e-c011-452e-b646-e95a1ecb945e' merged, but volume removal failed. Some or all of the following volumes may be orphaned: [bd5e7e03-d3f3-458f-8b5f-f53536df8c10]. Please retry Live Merge on the snapshot to complete the operation.
2018-03-10 14:15:27,573+13 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (DefaultQuartzScheduler5) [c23b59cc-bb79-4486-b031-a09a478e4959] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.



SPM: task doesnt complete (widespread storage access problems)

2018-03-08 12:15:44,991+1300 INFO  (tasks/9) [storage.ThreadPool.WorkerThread] START task 1828df85-8c90-49c5-8cba-fe958e5f14de (cmd=<bound method Task.commit of <storage.task.Task instance at 0x7f56f419eb48>>, args=None) (threadPool:208) 
...
2018-03-08 14:00:58,878+1300 INFO  (jsonrpc/5) [vdsm.api] FINISH getAllTasksStatuses return={'allTasksStatus': {'1828df85-8c90-49c5-8cba-fe958e5f14de': {'code': 0, 'message': 'running job 1 of 1', 'taskState': 'running', 'taskResult': '', 'taskID': '1828df85-8c90-49c5-8cba-fe958e5f14de'}, .... } ...



ENGINE RETRY: RemoveSnapshotCommand Validation fails with NPE:

2018-03-13 12:17:32,338+13 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (default task-16) [c356ec77-0dfa-4ea1-960b-982b9b24cdef] Error during ValidateFailure.: java.lang.NullPointerException
        at org.ovirt.engine.core.bll.validator.storage.StorageDomainValidator.getTotalSizeForMerge(StorageDomainValidator.java:121) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.StorageDomainValidator.hasSpaceForMerge(StorageDomainValidator.java:207) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.MultipleStorageDomainsValidator.lambda$allDomainsHaveSpaceForMerge$0(MultipleStorageDomainsValidator.java:128) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.MultipleStorageDomainsValidator.validOrFirstFailure(MultipleStorageDomainsValidator.java:190) [bll.jar:]
        at org.ovirt.engine.core.bll.validator.storage.MultipleStorageDomainsValidator.allDomainsHaveSpaceForMerge(MultipleStorageDomainsValidator.java:125) [bll.jar:]
        at org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand.validateStorageDomains(RemoveSnapshotCommand.java:381) [bll.jar:]
        at org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand.validate(RemoveSnapshotCommand.java:359) [bll.jar:]
        at org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:836) [bll.jar:]

bd5e7e03 was the active image (leaf) before the merge. The merge succeeded and bd5e7e03 was removed from the chain. The leaf is now the image which was the parent of bd5e7e03, which is ee0fa36e. bd5e7e03 is still in the DB as COMMIT did not finish, but its parent was set to 00000000. There is now a loop in the chain in the DB which I understand causes the NULL subchain, trigerring the NPE.


              image_guid              |               parentid               | imagestatus 
--------------------------------------+--------------------------------------+-------------
 [ORPHAN]  --- this parent NULL triggers the NPE due to a null subchain?
 bd5e7e03-d3f3-458f-8b5f-f53536df8c10 | 00000000-0000-0000-0000-000000000000 |           4

 [LEAF]
 ee0fa36e-c011-452e-b646-e95a1ecb945e | 87284902-3827-4541-9465-3f7706d2a6a4 |           1
 87284902-3827-4541-9465-3f7706d2a6a4 | fa2d6a6b-dc17-48e9-a573-2e87b853e53a |           1
 fa2d6a6b-dc17-48e9-a573-2e87b853e53a | 605a5807-4405-4b27-981a-6b137766a502 |           1
 605a5807-4405-4b27-981a-6b137766a502 | 6eac6a71-7cff-4c77-bdb1-3082bc2b91e3 |           1
 6eac6a71-7cff-4c77-bdb1-3082bc2b91e3 | 00000000-0000-0000-0000-000000000000 |           1
 [BASE]

Version-Release number of selected component (if applicable):
ovirt-engine-4.1.6.2-0.1.el7.noarch

How reproducible:
Unknown

Comment 1 Germano Veit Michel 2018-03-14 01:07:55 UTC

By forcing this in 4.1.9 (db manipulation) I also get a NPE on the exact same spot.

Comment 4 Germano Veit Michel 2018-03-14 05:47:45 UTC

Not sure why I called this a loop, there is no loop. Its just a single image chain when trying to remove bd5e7e03, which doesn't make sense and is not well handled.

Comment 9 Ala Hino 2018-03-15 11:22:56 UTC

Moving to 4.2.4 for now as the issue isn't reproducible and need more analysis.

Comment 10 Marcelo Leandro 2018-05-22 14:46:34 UTC

I am have the same problem in a lot of vms.

attached logs the last vm with problem.

how a can resolve this problem? i need remove de database?

Thanks.

Comment 11 Marcelo Leandro 2018-05-22 14:47:06 UTC

Created attachment 1440212 [details]
spm vdsm and engine logs

Comment 12 Marcelo Leandro 2018-05-22 15:10:37 UTC

(In reply to Marcelo Leandro from comment #10)
> I am have the same problem in a lot of vms.
> 
> attached logs the last vm with problem.
> 
> how a can resolve this problem? i need remove de database?
> 
> Thanks.

I think the soluction is similar this :

https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15

but in this case I need the snapshot id , but in the snapshot tab dont have ids.

Comment 13 Marcelo Leandro 2018-05-22 15:35:52 UTC

(In reply to Marcelo Leandro from comment #12)
> (In reply to Marcelo Leandro from comment #10)
> > I am have the same problem in a lot of vms.
> > 
> > attached logs the last vm with problem.
> > 
> > how a can resolve this problem? i need remove de database?
> > 
> > Thanks.
> 
> I think the soluction is similar this :
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> 
> but in this case I need the snapshot id , but in the snapshot tab dont have
> ids.

I found the snapshot id, can I apply the solution mentioned in comment 12 ?

select snapshot_id,description from snapshots where vm_id = 'c847130e-230d-47ee-a124-54d46e1dd928';
             snapshot_id              | description 
--------------------------------------+-------------
 9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
 a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup

select * from images where vm_snapshot_id = 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
              image_guid              |     creation_date      |     size      |               it_guid                |               parentid               | imagestatus |        lastmodified        |          
  vm_snapshot_id            | volume_type | volume_format |            image_group_id            |         _create_date          |         _update_date          | active | volume_classification | qcow_compat 
--------------------------------------+------------------------+---------------+--------------------------------------+--------------------------------------+-------------+----------------------------+----------
----------------------------+-------------+---------------+--------------------------------------+-------------------------------+-------------------------------+--------+-----------------------+-------------
 db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 | 1209033293824 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:06.869-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 | 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
 a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 |  107374182400 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:06.564-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 | 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
 2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 | 1717986918400 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:07.098-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 | 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
 85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 |  536870912000 | 00000000-0000-0000-0000-000000000000 | 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21 18:03:06.196-03 | a64526ed-
d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 | 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 | 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2


Thanks.

Comment 14 Ala Hino 2018-05-22 15:53:11 UTC

(In reply to Marcelo Leandro from comment #13)
> (In reply to Marcelo Leandro from comment #12)
> > (In reply to Marcelo Leandro from comment #10)
> > > I am have the same problem in a lot of vms.
> > > 
> > > attached logs the last vm with problem.
> > > 
> > > how a can resolve this problem? i need remove de database?
> > > 
> > > Thanks.
> > 
> > I think the soluction is similar this :
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > 
> > but in this case I need the snapshot id , but in the snapshot tab dont have
> > ids.
> 
> I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> 
> select snapshot_id,description from snapshots where vm_id =
> 'c847130e-230d-47ee-a124-54d46e1dd928';
>              snapshot_id              | description 
> --------------------------------------+-------------
>  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
>  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> 
> select * from images where vm_snapshot_id =
> 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
>               image_guid              |     creation_date      |     size   
> |               it_guid                |               parentid             
> | imagestatus |        lastmodified        |          
>   vm_snapshot_id            | volume_type | volume_format |           
> image_group_id            |         _create_date          |        
> _update_date          | active | volume_classification | qcow_compat 
> --------------------------------------+------------------------+-------------
> --+--------------------------------------+-----------------------------------
> ---+-------------+----------------------------+----------
> ----------------------------+-------------+---------------+------------------
> --------------------+-------------------------------+------------------------
> -------+--------+-----------------------+-------------
>  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:06.869-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
>  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> 107374182400 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:06.564-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
>  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:07.098-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
>  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> 536870912000 | 00000000-0000-0000-0000-000000000000 |
> 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> 18:03:06.196-03 | a64526ed-
> d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> 
> 
> Thanks.

What's the error that you get when retrying live merge?

Comment 15 Marcelo Leandro 2018-05-22 15:59:23 UTC

(In reply to Ala Hino from comment #14)
> (In reply to Marcelo Leandro from comment #13)
> > (In reply to Marcelo Leandro from comment #12)
> > > (In reply to Marcelo Leandro from comment #10)
> > > > I am have the same problem in a lot of vms.
> > > > 
> > > > attached logs the last vm with problem.
> > > > 
> > > > how a can resolve this problem? i need remove de database?
> > > > 
> > > > Thanks.
> > > 
> > > I think the soluction is similar this :
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > 
> > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > ids.
> > 
> > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > 
> > select snapshot_id,description from snapshots where vm_id =
> > 'c847130e-230d-47ee-a124-54d46e1dd928';
> >              snapshot_id              | description 
> > --------------------------------------+-------------
> >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > 
> > select * from images where vm_snapshot_id =
> > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> >               image_guid              |     creation_date      |     size   
> > |               it_guid                |               parentid             
> > | imagestatus |        lastmodified        |          
> >   vm_snapshot_id            | volume_type | volume_format |           
> > image_group_id            |         _create_date          |        
> > _update_date          | active | volume_classification | qcow_compat 
> > --------------------------------------+------------------------+-------------
> > --+--------------------------------------+-----------------------------------
> > ---+-------------+----------------------------+----------
> > ----------------------------+-------------+---------------+------------------
> > --------------------+-------------------------------+------------------------
> > -------+--------+-----------------------+-------------
> >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.869-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.564-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:07.098-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.196-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > 
> > 
> > Thanks.
> 
> What's the error that you get when retrying live merge?

in the gui show this.

Error while executing action:

GOV-CAMARUPIM:

    General command validation failure.

Comment 16 mccarblea 2018-06-21 09:46:31 UTC

Created attachment 1453390 [details]
bug

to soon to tell

Comment 17 Marcelo Leandro 2018-06-26 12:14:43 UTC

(In reply to Ala Hino from comment #14)
> (In reply to Marcelo Leandro from comment #13)
> > (In reply to Marcelo Leandro from comment #12)
> > > (In reply to Marcelo Leandro from comment #10)
> > > > I am have the same problem in a lot of vms.
> > > > 
> > > > attached logs the last vm with problem.
> > > > 
> > > > how a can resolve this problem? i need remove de database?
> > > > 
> > > > Thanks.
> > > 
> > > I think the soluction is similar this :
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > 
> > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > ids.
> > 
> > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > 
> > select snapshot_id,description from snapshots where vm_id =
> > 'c847130e-230d-47ee-a124-54d46e1dd928';
> >              snapshot_id              | description 
> > --------------------------------------+-------------
> >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > 
> > select * from images where vm_snapshot_id =
> > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> >               image_guid              |     creation_date      |     size   
> > |               it_guid                |               parentid             
> > | imagestatus |        lastmodified        |          
> >   vm_snapshot_id            | volume_type | volume_format |           
> > image_group_id            |         _create_date          |        
> > _update_date          | active | volume_classification | qcow_compat 
> > --------------------------------------+------------------------+-------------
> > --+--------------------------------------+-----------------------------------
> > ---+-------------+----------------------------+----------
> > ----------------------------+-------------+---------------+------------------
> > --------------------+-------------------------------+------------------------
> > -------+--------+-----------------------+-------------
> >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.869-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.564-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:07.098-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > 18:03:06.196-03 | a64526ed-
> > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > 
> > 
> > Thanks.
> 
> What's the error that you get when retrying live merge?

Can you help to delete this problem snapshots?

Very Thanks.

Comment 18 Yaniv Lavi 2018-08-13 08:47:47 UTC

Please reply to comment $17.

Comment 19 Ala Hino 2018-08-14 08:34:04 UTC

(In reply to Marcelo Leandro from comment #17)
> (In reply to Ala Hino from comment #14)
> > (In reply to Marcelo Leandro from comment #13)
> > > (In reply to Marcelo Leandro from comment #12)
> > > > (In reply to Marcelo Leandro from comment #10)
> > > > > I am have the same problem in a lot of vms.
> > > > > 
> > > > > attached logs the last vm with problem.
> > > > > 
> > > > > how a can resolve this problem? i need remove de database?
> > > > > 
> > > > > Thanks.
> > > > 
> > > > I think the soluction is similar this :
> > > > 
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > > 
> > > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > > ids.
> > > 
> > > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > > 
> > > select snapshot_id,description from snapshots where vm_id =
> > > 'c847130e-230d-47ee-a124-54d46e1dd928';
> > >              snapshot_id              | description 
> > > --------------------------------------+-------------
> > >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> > >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > > 
> > > select * from images where vm_snapshot_id =
> > > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> > >               image_guid              |     creation_date      |     size   
> > > |               it_guid                |               parentid             
> > > | imagestatus |        lastmodified        |          
> > >   vm_snapshot_id            | volume_type | volume_format |           
> > > image_group_id            |         _create_date          |        
> > > _update_date          | active | volume_classification | qcow_compat 
> > > --------------------------------------+------------------------+-------------
> > > --+--------------------------------------+-----------------------------------
> > > ---+-------------+----------------------------+----------
> > > ----------------------------+-------------+---------------+------------------
> > > --------------------+-------------------------------+------------------------
> > > -------+--------+-----------------------+-------------
> > >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:06.869-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> > >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:06.564-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> > >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:07.098-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> > >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > 18:03:06.196-03 | a64526ed-
> > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > > 
> > > 
> > > Thanks.
> > 
> > What's the error that you get when retrying live merge?
> 
> Can you help to delete this problem snapshots?
> 
> Very Thanks.

Hi,

Do you still need guidelines to delete the snapshot?

Comment 20 Marcelo Leandro 2018-08-14 09:59:44 UTC

(In reply to Ala Hino from comment #19)
> (In reply to Marcelo Leandro from comment #17)
> > (In reply to Ala Hino from comment #14)
> > > (In reply to Marcelo Leandro from comment #13)
> > > > (In reply to Marcelo Leandro from comment #12)
> > > > > (In reply to Marcelo Leandro from comment #10)
> > > > > > I am have the same problem in a lot of vms.
> > > > > > 
> > > > > > attached logs the last vm with problem.
> > > > > > 
> > > > > > how a can resolve this problem? i need remove de database?
> > > > > > 
> > > > > > Thanks.
> > > > > 
> > > > > I think the soluction is similar this :
> > > > > 
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > > > 
> > > > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > > > ids.
> > > > 
> > > > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > > > 
> > > > select snapshot_id,description from snapshots where vm_id =
> > > > 'c847130e-230d-47ee-a124-54d46e1dd928';
> > > >              snapshot_id              | description 
> > > > --------------------------------------+-------------
> > > >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> > > >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > > > 
> > > > select * from images where vm_snapshot_id =
> > > > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> > > >               image_guid              |     creation_date      |     size   
> > > > |               it_guid                |               parentid             
> > > > | imagestatus |        lastmodified        |          
> > > >   vm_snapshot_id            | volume_type | volume_format |           
> > > > image_group_id            |         _create_date          |        
> > > > _update_date          | active | volume_classification | qcow_compat 
> > > > --------------------------------------+------------------------+-------------
> > > > --+--------------------------------------+-----------------------------------
> > > > ---+-------------+----------------------------+----------
> > > > ----------------------------+-------------+---------------+------------------
> > > > --------------------+-------------------------------+------------------------
> > > > -------+--------+-----------------------+-------------
> > > >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > > > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.869-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > > > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> > > >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > > > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.564-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > > > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> > > >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > > > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:07.098-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > > > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> > > >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > > > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.196-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > > > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > > > 
> > > > 
> > > > Thanks.
> > > 
> > > What's the error that you get when retrying live merge?
> > 
> > Can you help to delete this problem snapshots?
> > 
> > Very Thanks.
> 
> Hi,
> 
> Do you still need guidelines to delete the snapshot?

Good morning, 
Yes, I still need.

Very thanks.

Comment 21 Marcelo Leandro 2018-08-18 07:01:59 UTC

(In reply to Ala Hino from comment #19)
> (In reply to Marcelo Leandro from comment #17)
> > (In reply to Ala Hino from comment #14)
> > > (In reply to Marcelo Leandro from comment #13)
> > > > (In reply to Marcelo Leandro from comment #12)
> > > > > (In reply to Marcelo Leandro from comment #10)
> > > > > > I am have the same problem in a lot of vms.
> > > > > > 
> > > > > > attached logs the last vm with problem.
> > > > > > 
> > > > > > how a can resolve this problem? i need remove de database?
> > > > > > 
> > > > > > Thanks.
> > > > > 
> > > > > I think the soluction is similar this :
> > > > > 
> > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1302215#c15
> > > > > 
> > > > > but in this case I need the snapshot id , but in the snapshot tab dont have
> > > > > ids.
> > > > 
> > > > I found the snapshot id, can I apply the solution mentioned in comment 12 ?
> > > > 
> > > > select snapshot_id,description from snapshots where vm_id =
> > > > 'c847130e-230d-47ee-a124-54d46e1dd928';
> > > >              snapshot_id              | description 
> > > > --------------------------------------+-------------
> > > >  9569845f-d961-4e49-8809-1a032ba04b02 | Active VM
> > > >  a64526ed-d32a-4a7d-a6b8-fc2410a371a2 | Backup
> > > > 
> > > > select * from images where vm_snapshot_id =
> > > > 'a64526ed-d32a-4a7d-a6b8-fc2410a371a2'::uuid;
> > > >               image_guid              |     creation_date      |     size   
> > > > |               it_guid                |               parentid             
> > > > | imagestatus |        lastmodified        |          
> > > >   vm_snapshot_id            | volume_type | volume_format |           
> > > > image_group_id            |         _create_date          |        
> > > > _update_date          | active | volume_classification | qcow_compat 
> > > > --------------------------------------+------------------------+-------------
> > > > --+--------------------------------------+-----------------------------------
> > > > ---+-------------+----------------------------+----------
> > > > ----------------------------+-------------+---------------+------------------
> > > > --------------------+-------------------------------+------------------------
> > > > -------+--------+-----------------------+-------------
> > > >  db6f5e35-9c3c-4d60-861a-5c0eea24d848 | 2018-05-21 15:35:10-03 |
> > > > 1209033293824 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.869-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 686fb5d2-ae2f-45c3-9ea2-b5d1a518230f | 2018-05-21 15:35:09.857381-03 |
> > > > 2018-05-22 10:41:13.91491-03  | f      |                     1 |           2
> > > >  a94c614a-7cce-47b5-a23c-af22a888bf26 | 2018-05-21 15:35:09-03 | 
> > > > 107374182400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.564-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 00872bbe-4cc2-4720-878e-492bb69f3a05 | 2018-05-21 15:35:08.927393-03 |
> > > > 2018-05-22 10:41:13.206059-03 | f      |                     1 |           2
> > > >  2578d8f3-1443-4fa6-8297-24c270e99bd9 | 2018-05-21 15:35:09-03 |
> > > > 1717986918400 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:07.098-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 02b51f4c-06af-4c39-a67a-51cee2079cd2 | 2018-05-21 15:35:08.583297-03 |
> > > > 2018-05-22 10:41:14.586563-03 | f      |                     1 |           2
> > > >  85541eb4-52aa-4f98-84c2-275b228c1e34 | 2018-05-21 15:35:10-03 | 
> > > > 536870912000 | 00000000-0000-0000-0000-000000000000 |
> > > > 00000000-0000-0000-0000-000000000000 |           4 | 2018-05-21
> > > > 18:03:06.196-03 | a64526ed-
> > > > d32a-4a7d-a6b8-fc2410a371a2 |           2 |             4 |
> > > > 3f1c8849-9c69-46be-9773-91c803860791 | 2018-05-21 15:35:09.595853-03 |
> > > > 2018-05-22 10:41:17.362994-03 | f      |                     1 |           2
> > > > 
> > > > 
> > > > Thanks.
> > > 
> > > What's the error that you get when retrying live merge?
> > 
> > Can you help to delete this problem snapshots?
> > 
> > Very Thanks.
> 
> Hi,
> 
> Do you still need guidelines to delete the snapshot?
Hi Ala Hino,

Can you help me with guidelines to delete the snapshot ?

Comment 22 Ala Hino 2018-08-28 09:01:54 UTC

Marcelo,

Let's please continue this conversation on bug 1574631, originally reported by you.
Please see https://bugzilla.redhat.com/show_bug.cgi?id=1574631#c45

Comment 23 Alex Kaouris 2018-09-10 15:46:56 UTC

Created attachment 1482199 [details]
engine.log

I might have hit the same case. 
When I try to delete a snapshot to several VMs I get: General command validation failure.

Comment 25 Elad 2019-01-24 09:33:33 UTC

Created attachment 1523044 [details]
user-logs

This was seen also by a user, attaching the logs

Comment 26 Sandro Bonazzola 2019-01-28 09:40:28 UTC

This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

Comment 28 Elad 2019-03-27 14:43:35 UTC

Beny, can you please advise on verification steps?

Comment 29 Benny Zlotnik 2019-03-27 14:47:11 UTC

1. Create multiple snapshots for a disk
2. Break the chain by taking one the images in the chain and set its parent to an empty guid (manually, in the DB)
3. Run live merge
You should see an error in the form of: "The requested snapshot has an invalid parent, either fix the snapshot definition or remove it manually to complete the process."

Comment 31 Evelina Shames 2019-04-07 12:47:09 UTC

Verified on ovirt-engine-4.3.3.2-0.1.el7.noarch

Comment 33 errata-xmlrpc 2019-05-08 12:37:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085

Comment 34 Daniel Gur 2019-08-28 13:14:03 UTC

sync2jira

Comment 35 Daniel Gur 2019-08-28 13:18:18 UTC

sync2jira

Note You need to log in before you can comment on or make changes to this bug.