Bug 1673035

Summary: [cinderlib] Support snapshot operations
Product: [oVirt] ovirt-engine Reporter: Benny Zlotnik <bzlotnik>
Component: BLL.StorageAssignee: Benny Zlotnik <bzlotnik>
Status: CLOSED CURRENTRELEASE QA Contact: Shir Fishbain <sfishbai>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.3.0CC: aefrat, bugs, dfediuck, frolland, sfishbai, tnisan
Target Milestone: ovirt-4.4.2Flags: pm-rhel: ovirt-4.4+
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.3.5.3 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-22 10:22:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1684889, 1705170, 1728255    
Bug Blocks:    
Attachments:
Description Flags
Logs
none
New_Logs
none
new_logs_from_today
none
New_logs
none
New_logs_snapshot_operations_4.3.6
none
New_logs_snapshot_operations_4.3.6_two_disks none

Description Benny Zlotnik 2019-02-06 14:00:32 UTC 
Description of problem:
We should support snapshot related operations on MBS disks:
1. create
2. delete
3. preview
4. undo
5. commit

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 1 Shir Fishbain 2019-03-25 14:35:25 UTC 
Created attachment 1547723 [details]
Logs
Comment 2 Shir Fishbain 2019-03-25 14:39:16 UTC 
The status for each SD:

HP3PAR - couldn't start VM 
Create - Passed without memory
Delete - Passed
Preview - Passed without memory
Undo - Passed without memory
Commit - Passed without memory

NetApp 
Create - Passed without memory
Delete - Passed
Preview - Passed without memory
Undo - Passed without memory
Commit - Passed without memory

Kaminario - 
Create - Passed without memory
Delete - Passed
Preview - Passed without memory
Undo - Passed without memory
Commit - Passed without memory

Native Ceph - 
Create - Passed with memory
Delete - Failed - The disks are locked
Preview - Passed with memory
Undo - Passed with memory
Commit - Passed with memory

When succeed to create a snapshot with memory the disks are locked
logs attached
Comment 3 RHEL Program Management 2019-03-25 14:39:19 UTC 
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 5 Sandro Bonazzola 2019-03-26 07:22:06 UTC 
Re-targeted to 4.3.3 having failed QE on 4.3.2
Comment 7 Shir Fishbain 2019-03-26 14:55:37 UTC 
Created attachment 1548107 [details]
New_Logs
Comment 8 Shir Fishbain 2019-03-27 15:09:41 UTC 
Created attachment 1548615 [details]
new_logs_from_today
Comment 9 Benny Zlotnik 2019-04-01 14:06:58 UTC 
An update on the current status:
NetApp: cannot work at the moment, as we do not have the correct license
The error looks like this:
NaApiError: NetApp API failed. Reason - 14616:FlexClone not licensed

Some of the creation errors are related to the guest agent being unavailable:
The error will look like this
2019-03-27 16:22:12,678+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-313183) [0c2b2589-a29b-4d00-97a2-71e188df69ad] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected

We need to have a guest agent in order to perform live snapshots


HP3PAR:
Snapshot removal will not work if we use the retention options when creating the storage domain, the following options cannot be used: hpe3par_snapshot_retention, hpe3par_snapshot_expiration
 
If an error occurs while deleting a snapshot it will look like this:
AttributeError: 'MetaData' object has no attribute '_bulk_update_tuples'

While the actual error can be seen in the cinderlib logs (/var/log/ovirt-engine/cinderlib/cinderlib.log):
2019-04-01 15:55:44,410 - cinderlib.objects - ERROR - exception occured: volume retention time has not expired

This issue is currently being handled in cinderlib, see[1]


[1] - https://bugs.launchpad.net/cinderlib/+bug/1819708
Comment 10 Tal Nisan 2019-04-30 06:16:20 UTC 
Benny, I see all the bugs have been merged, do we have a reason to keep this bug on POST or can we move it to MODIFIED?
Comment 11 Avihai 2019-04-30 06:53:02 UTC 
(In reply to Tal Nisan from comment #10)
> Benny, I see all the bugs have been merged, do we have a reason to keep this
> bug on POST or can we move it to MODIFIED?

Maybe, as there was an issue with a failed ceph snapshot, deleted operation, see Shir comment#2.
Did not see any follow up on this issue.
Comment 12 Benny Zlotnik 2019-04-30 06:59:27 UTC 
The new ones need to be backported, I'll squash and backport.

I am not sure what issue with ceph is (my guess it's the missing guest agent), if you can reproduce it please add logs specific to ceph.
As for the other backends, see my previous comment.
Comment 13 Avihai 2019-04-30 14:45:05 UTC 
(In reply to Benny Zlotnik from comment #12)
> The new ones need to be backported, I'll squash and backport.
> 
> I am not sure what issue with ceph is (my guess it's the missing guest
> agent), if you can reproduce it please add logs specific to ceph.
Benny , can you please elaborate on what specific ceph logs you require?
  
> As for the other backends, see my previous comment.

Shir , 
Can you please see if the failed ceph snapshot, deleted operation,(mentioned in comment#2) reproduces?
When/if it reproduces please contact Benny for specific ceph logs he requires.
Comment 14 Shir Fishbain 2019-05-01 06:34:18 UTC 
I can't reproduce it because the ceph snapshot doesn't create with memory.
The main issue is that impossible to create a snapshot with memory, just without memory.
When creating VM with regular ISCSI SD disk, the created snapshot succeded,  so the problem is just with Managed block Domain.

Logs attached
Comment 15 Shir Fishbain 2019-05-01 06:35:58 UTC 
Created attachment 1560664 [details]
New_logs
Comment 16 Benny Zlotnik 2019-05-01 07:24:35 UTC 
2019-04-30 18:01:24,853+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-315606) [63a57060-f603-4d22-bc45-84fa93dae0ef] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_1 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
 
This is the error I saw for the ceph-vm, does the VM have a guest agent?
Comment 17 Benny Zlotnik 2019-05-01 17:59:06 UTC 
why does bug 1705170 block this one?
creating a live snapshot and having the guest agent properly freeze the FS is the sensible flow
Comment 18 Avihai 2019-05-02 04:14:18 UTC 
(In reply to Benny Zlotnik from comment #17)
> why does bug 1705170 block this one?
> creating a live snapshot and having the guest agent properly freeze the FS
> is the sensible flow

This bug is about Supporting snapshot operations including live snapshot the same as in non MBS .
As customer expects the same behavior for non MBS and MBS this option is not supported due to bug 1705170 .
Comment 19 Benny Zlotnik 2019-05-02 06:49:12 UTC 
(In reply to Avihai from comment #18)
> (In reply to Benny Zlotnik from comment #17)
> > why does bug 1705170 block this one?
> > creating a live snapshot and having the guest agent properly freeze the FS
> > is the sensible flow
> 
> This bug is about Supporting snapshot operations including live snapshot the
> same as in non MBS .
> As customer expects the same behavior for non MBS and MBS this option is not
> supported due to bug 1705170 .

This is how it always worked for cinder disks, which are non-MBS disks as well, there was no change in the behavior in this regard
Comment 20 Avihai 2019-05-02 07:06:46 UTC 
(In reply to Benny Zlotnik from comment #19)
> (In reply to Avihai from comment #18)
> > (In reply to Benny Zlotnik from comment #17)
> > > why does bug 1705170 block this one?
> > > creating a live snapshot and having the guest agent properly freeze the FS
> > > is the sensible flow
> > 
> > This bug is about Supporting snapshot operations including live snapshot the
> > same as in non MBS .
> > As customer expects the same behavior for non MBS and MBS this option is not
> > supported due to bug 1705170 .
> 
> This is how it always worked for cinder disks, which are non-MBS disks as
> well, there was no change in the behavior in this regard
As discussed in the last scrum by you and Nir it was agreed this should be supported also in MBS and you said to open a bug on this thus bug 1705170.
If this is indeed so then this bug is blocked by this bug.

In the case this limitation is acceptable also with product/PM side we should document it in feature page or document bug and close bug 1705170. 

Let's discuss this F2F with Nir and Tal and see how to proceed here.
Comment 21 Avihai 2019-07-09 06:20:32 UTC 
Hi Shir, 
Can we verify this bug with the recent StartVM issues seen in cinderLib?
Looking at cinderlib status doc[1] we can not start VM's on most storage backends except NetApp, right?

If so we can not test snapshot with memory snapshots meaning we are blocked until the StartVM bug is resolved.
Please block this bug with the Cinderlib StartVM bug, use 'depends on'

[1] https://docs.google.com/document/d/1uIwxRv0k2xSUmfyH8RgMvW4JE6JWuCTvaCpOB43nUvg/edit?usp=sharing
Comment 22 Avihai 2019-07-09 06:22:46 UTC 
(In reply to Avihai from comment #21)
> Hi Shir, 
> Can we verify this bug with the recent StartVM issues seen in cinderLib?
> Looking at cinderlib status doc[1] we can not start VM's on most storage
> backends except NetApp, right?
> 
> If so we can not test snapshot with memory snapshots meaning we are blocked
> until the StartVM bug is resolved.
> Please block this bug with the Cinderlib StartVM bug, use 'depends on'
> 
> [1]
> https://docs.google.com/document/d/
> 1uIwxRv0k2xSUmfyH8RgMvW4JE6JWuCTvaCpOB43nUvg/edit?usp=sharing

Please add all blocking bugs on any snapshot failed operation(Create/Delete/Preview/Undo/Commit) on both regular and memory snapshot to the 'depends on'.
In this way, we will know what is blocking us to verify this bug.
Comment 23 RHEL Program Management 2019-07-21 08:33:20 UTC 
Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.
Comment 24 Avihai 2019-07-21 09:03:03 UTC 
As StartVM operation is blocked due to bugs 1705170 ,1728255 this bug can not be verified fully.
Comment 25 Fred Rolland 2019-07-22 06:40:18 UTC 
1705170 is a doc issue, it is not blocking anything.

I don't understand why we cannot verify the snapshot.
There is a bug for running VMs with a workaround: 1728255 (Which is BTW a specific env issue)

Use the workaround to have the VMs and then verify the snapshot flows.
Comment 26 Avihai 2019-07-22 13:32:25 UTC 
For verifying this bug we need a fix for running VM issue as no customer will do the suggested work around as it will disable his current iscsi SD's.

If we verify now with the suggested work around we can miss an issue when a fix to the running VM is available.

I was not aware this is an environment issue but a bug that can occur and we saw it multiple times.

Shir, please correct me if i'm wrong missed something.
Let's meet tomorrow F2F and discuss this further.
Comment 27 Benny Zlotnik 2019-07-22 13:46:41 UTC 
just to clarify, what referred to as a workaround, is not a workaround for a bug, but for the environment's misconfiguration
Comment 28 Avihai 2019-07-22 15:03:47 UTC 
This is the impact of the blocking bug Bug 1684889:
ISCSI MBS SD's VM's can not be used when Data ISCSI SD's VM's are used.
Meaning ISCSI MBS SD's and ISCSI Data domains can not really work together at all.
  
In order for one to work you need to ISCSI disconnect from all hosts(or unmap all hosts at the storage) the existing SD(Data/MBS) using the current ISCSI connections in order for the brickos/VDSM to succeed connecting to the storage when you VM up.

I was not aware that this is a misconfiguration but a bug that we should fix.

Benny, can you please:

1) Elaborate on what is the misconfiguration exactly?

2) Explain if a customer can make/stumble on this misconfiguration as we did? 
If so we should still handle this issue and not brush it off.
Comment 29 Benny Zlotnik 2019-07-23 08:00:06 UTC 
(In reply to Avihai from comment #28)
> This is the impact of the blocking bug Bug 1684889:
> ISCSI MBS SD's VM's can not be used when Data ISCSI SD's VM's are used.
> Meaning ISCSI MBS SD's and ISCSI Data domains can not really work together
> at all.
>   
> In order for one to work you need to ISCSI disconnect from all hosts(or
> unmap all hosts at the storage) the existing SD(Data/MBS) using the current
> ISCSI connections in order for the brickos/VDSM to succeed connecting to the
> storage when you VM up.
> 
> I was not aware that this is a misconfiguration but a bug that we should fix.
> 
> Benny, can you please:
> 
> 1) Elaborate on what is the misconfiguration exactly?
> 
> 2) Explain if a customer can make/stumble on this misconfiguration as we
> did? 
> If so we should still handle this issue and not brush it off.

1. An iscsi portal can only be used by either a regular data domain or a managed block domain in the same datacenter.
This is because both os-brick and vdsm expect to be the only component managing the connections, and it also does not make much sense to use both simultaneously

2. As stated before, it should not be used together, one should be removed/moved to another DC
Comment 30 Shir Fishbain 2019-08-11 17:10:45 UTC 
The status of each driver:
Netapp :
** The validation of operation snapshot was with OS and the snapshot was created with memory**
1. Start VM - Passed
2. Create a  live snapshot of one MBD disk :
Failed at the first attempt:
2019-08-11 17:31:57,523+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-146345) [dcdfb4de-266a-4730-8913-9dc35192b27e] Failed in 'FreezeVDS' method
2019-08-11 17:31:57,536+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-146345) [dcdfb4de-266a-4730-8913-9dc35192b27e] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-11 17:31:57,537+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-146345) [dcdfb4de-266a-4730-8913-9dc35192b27e] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='61463936-e1a9-40e9-b64b-eab5866484f8'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 17:31:57,554+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-146345) [dcdfb4de-266a-4730-8913-9dc35192b27e] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-11 17:31:57,570+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-146345) [dcdfb4de-266a-4730-8913-9dc35192b27e] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-11 17:31:57,578+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-146345) [dcdfb4de-266a-4730-8913-9dc35192b27e] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot S1 for VM netapp_vm (User: admin@internal-authz).
2019-08-11 17:31:59,324+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [dcdfb4de-266a-4730-8913-9dc35192b27e] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-11 17:31:59,356+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [dcdfb4de-266a-4730-8913-9dc35192b27e] Failed in 'ThawVDS' method
2019-08-11 17:31:59,359+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [dcdfb4de-266a-4730-8913-9dc35192b27e] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='61463936-e1a9-40e9-b64b-eab5866484f8'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 17:31:59,393+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-83) [dcdfb4de-266a-4730-8913-9dc35192b27e] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'S1' creation for VM 'netapp_vm'.

Passed at the second attempt 

3.Preview - Passed
4.Commit - Passed
5.Undo - Passed
6.Delete snapshot - Passed

Ceph :
** The validation of operation snapshot was with OS and the snapshot was created with memory**
1. Start VM - Passed
2. Create a live snapshot of one MBD disk :
Failed at the first attempt:
2019-08-11 19:16:33,240+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-148890) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Failed in 'FreezeVDS' method
2019-08-11 19:16:33,255+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-148890) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-11 19:16:33,255+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-148890) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 19:16:33,264+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-148890) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-11 19:16:33,279+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-148890) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-11 19:16:33,286+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-148890) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snap1 for VM ceph_vm (User: admin@internal-authz).
2019-08-11 19:16:34,776+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-11 19:16:34,832+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Failed in 'ThawVDS' method
2019-08-11 19:16:34,836+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 19:16:34,873+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [f8c96647-ee1f-4fbf-83cf-f2c678f1fa29] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snap1' creation for VM 'ceph_vm'.

Failed at the second attempt:
2019-08-11 19:17:54,269+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-148925) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 19:17:54,285+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-148925) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-11 19:17:54,300+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-148925) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-11 19:17:54,325+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-148925) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot s1 for VM ceph_vm (User: admin@internal-authz).
2019-08-11 19:17:57,418+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-6) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-11 19:17:57,470+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-6) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] Failed in 'ThawVDS' method
2019-08-11 19:17:57,471+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-6) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 19:17:57,507+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-6) [9eb272d7-b8a9-4b1b-94df-2ef2083c8a42] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 's1' creation for VM 'ceph_vm'.

Passed at the third attempt.

3.Preview - Passed
4.Undo - Passed
5.Commit - Passed

6.Delete snapshot- Failed:
2019-08-11 19:25:10,371+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default task-115) [24ddef5f-222d-4f71-8375-1375f1ae9797] cinderlib execution failed: Traceback (most recent call last):
2019-08-11 19:25:19,306+03 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [24ddef5f-222d-4f71-8375-1375f1ae9797] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.
2019-08-11 19:25:19,350+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-43) [24ddef5f-222d-4f71-8375-1375f1ae9797] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'snapshot' for VM 'ceph_vm'.

Kaminario:
** The validation of operation snapshot was with OS and the snapshot was created with memory**
1. Start VM - Passed
2. Create a live snapshot of one MBD disk :
Failed at the first and second attempt:
2019-08-11 18:30:46,347+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-147772) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Failed in 'FreezeVDS' method
2019-08-11 18:30:46,362+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-147772) [b5c0bc50-8165-4952-a238-1f5f085b1a62] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-11 18:30:46,363+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-147772) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 18:30:46,375+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-147772) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-11 18:30:46,388+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-147772) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-11 18:30:46,395+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-147772) [b5c0bc50-8165-4952-a238-1f5f085b1a62] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot s1 for VM kaminario_vm (User: admin@internal-authz).
2019-08-11 18:30:48,227+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-11 18:30:48,280+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Failed in 'ThawVDS' method
2019-08-11 18:30:48,287+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [b5c0bc50-8165-4952-a238-1f5f085b1a62] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 18:30:48,320+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-62) [b5c0bc50-8165-4952-a238-1f5f085b1a62] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 's1' creation for VM 'kaminario_vm'.
2019-08-11 18:31:45,002+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-147795) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Failed in 'FreezeVDS' method
2019-08-11 18:31:45,012+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-147795) [71c897de-46aa-4c25-85e8-7522ac5f83e5] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-11 18:31:45,013+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-147795) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 18:31:45,034+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-147795) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-11 18:31:45,048+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-147795) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-11 18:31:45,057+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-147795) [71c897de-46aa-4c25-85e8-7522ac5f83e5] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot s1 for VM kaminario_vm (User: admin@internal-authz).
2019-08-11 18:31:46,464+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-11 18:31:46,511+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Failed in 'ThawVDS' method
2019-08-11 18:31:46,512+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [71c897de-46aa-4c25-85e8-7522ac5f83e5] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-11 18:31:46,551+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-47) [71c897de-46aa-4c25-85e8-7522ac5f83e5] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 's1' creation for VM 'kaminario_vm'.

Passed at the third attempt

3.Preview - Failed 
1)Include in the previewed VM only the disks that are included in the snapshot.
2019-08-11 18:38:23,759+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default task-78) [d2b9dab7-e172-483b-bfbd-7d97b46fd14d] cinderlib execution failed: Traceback (most recent call last):
2019-08-11 18:38:28,622+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [d2b9dab7-e172-483b-bfbd-7d97b46fd14d] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand' with failure.
2019-08-11 18:38:28,734+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [d2b9dab7-e172-483b-bfbd-7d97b46fd14d] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand' with failure.
2019-08-11 18:38:28,910+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [] EVENT_ID: USER_TRY_BACK_TO_SNAPSHOT_FINISH_FAILURE(99), Failed to complete Snapshot-Preview snapshot1 for VM kaminario_vm.

2)Include in the previewed VM all disks (keep the disks that are not included in the snapshot as-is).
2019-08-11 18:39:33,991+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default task-88) [c985b62d-b3e7-4706-ba5e-81941c1487a4] cinderlib execution failed: Traceback (most recent call last):
2019-08-11 18:39:39,077+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-74) [c985b62d-b3e7-4706-ba5e-81941c1487a4] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand' with failure.
2019-08-11 18:39:39,096+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-74) [c985b62d-b3e7-4706-ba5e-81941c1487a4] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand' with failure.
2019-08-11 18:39:39,153+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-74) [] EVENT_ID: USER_TRY_BACK_TO_SNAPSHOT_FINISH_FAILURE(99), Failed to complete Snapshot-Preview snapshot1 for VM kaminario_vm.

3)None of the above, take me to the 'Custom Preview Snapshot' dialog.
2019-08-11 18:40:40,100+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default task-89) [e15dbd90-6ddf-4aa9-b1f2-9b2a2000fefa] cinderlib execution failed: Traceback (most recent call last):
2019-08-11 18:40:43,371+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [e15dbd90-6ddf-4aa9-b1f2-9b2a2000fefa] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand' with failure.
2019-08-11 18:40:43,419+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [e15dbd90-6ddf-4aa9-b1f2-9b2a2000fefa] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand' with failure.
2019-08-11 18:40:43,729+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-95) [] EVENT_ID: USER_TRY_BACK_TO_SNAPSHOT_FINISH_FAILURE(99), Failed to complete Snapshot-Preview snapshot1 for VM kaminario_vm.

4.Impossible to make commit & undo.
5.Delete snapshot- Passed

3PAR :
Couldn't start VM - Just with the two workarounds -https://bugzilla.redhat.com/show_bug.cgi?id=1730335
Create a cold snapshot with memory (just with one MBD disk) -Passed
Preview - Passed
Undo - Passed
Commit - Passed
Delete - Passed
Comment 31 Shir Fishbain 2019-08-11 17:14:38 UTC 
Created attachment 1602632 [details]
New_logs_snapshot_operations_4.3.6
Comment 32 Shir Fishbain 2019-08-12 15:53:13 UTC 
The status of each driver for snapshots of two disks ( MBD disk and iscsi regular disk):

Netapp :
** The validation of operation snapshot was with OS and the snapshot was created with memory**
1. Create a live snapshot -
At the first and second attempts:
2019-08-12 15:48:47,062+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [8aad5a6c-6ea0-4b1f-98bf-615919082bef] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 15:48:47,090+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [8aad5a6c-6ea0-4b1f-98bf-615919082bef] Failed in 'ThawVDS' method
2019-08-12 15:48:47,093+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [8aad5a6c-6ea0-4b1f-98bf-615919082bef] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='bf08a633-3747-4cf8-8abf-15b7342d1326'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 15:48:47,128+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-28) [8aad5a6c-6ea0-4b1f-98bf-615919082bef] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 's1' creation for VM 'netapp_vm2'.

2019-08-12 15:49:46,449+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-178355) [f12d808d-f094-409d-9931-15aae8ee325c] Failed in 'FreezeVDS' method
2019-08-12 15:49:46,484+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-178355) [f12d808d-f094-409d-9931-15aae8ee325c] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 15:49:46,485+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-178355) [f12d808d-f094-409d-9931-15aae8ee325c] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='bf08a633-3747-4cf8-8abf-15b7342d1326'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 15:49:46,512+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-178355) [f12d808d-f094-409d-9931-15aae8ee325c] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 15:49:46,528+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-178355) [f12d808d-f094-409d-9931-15aae8ee325c] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 15:49:46,537+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-178355) [f12d808d-f094-409d-9931-15aae8ee325c] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snapshot1 for VM netapp_vm2 (User: admin@internal-authz).
2019-08-12 15:49:48,458+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [f12d808d-f094-409d-9931-15aae8ee325c] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 15:49:48,493+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [f12d808d-f094-409d-9931-15aae8ee325c] Failed in 'ThawVDS' method
2019-08-12 15:49:48,494+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [f12d808d-f094-409d-9931-15aae8ee325c] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='bf08a633-3747-4cf8-8abf-15b7342d1326'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 15:49:48,527+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [f12d808d-f094-409d-9931-15aae8ee325c] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snapshot1' creation for VM 'netapp_vm2'.

succeded at the third attempt 

2.Preview- Failed
2019-08-12 16:56:01,169+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (default task-142) [5c3a7680-8c0c-47fa-85a3-7dbb400fa04a] Failed executing creating volume from snapshot: java.lang.IllegalArgumentException: Invalid UUID string: Snapshot '%s' not found
2019-08-12 16:56:05,922+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [5c3a7680-8c0c-47fa-85a3-7dbb400fa04a] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand' with failure.
2019-08-12 16:56:05,991+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [5c3a7680-8c0c-47fa-85a3-7dbb400fa04a] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToSnapshotCommand' with failure.
2019-08-12 16:56:06,324+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [5c3a7680-8c0c-47fa-85a3-7dbb400fa04a] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand' with failure.
2019-08-12 16:56:06,375+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-7) [] EVENT_ID: USER_TRY_BACK_TO_SNAPSHOT_FINISH_FAILURE(99), Failed to complete Snapshot-Preview snap for VM netapp_vm2.

Impossible to make commit & undo

3. Delete snapshot - 
2019-08-12 16:57:53,472+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [] Failed in 'HSMGetAllTasksStatusesVDS' method
2019-08-12 16:57:53,553+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command HSMGetAllTasksStatusesVDS failed: 'e8daab63-57bd-4a87-bad8-4cb11be4e199' is not in list
2019-08-12 16:57:53,584+03 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-81) [] BaseAsyncTask::logEndTaskFailure: Task '1ef18898-a7f1-48a0-adfd-3c162012efee' (Parent Command 'PrepareMerge', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:
2019-08-12 16:57:53,682+03 ERROR [org.ovirt.engine.core.bll.storage.disk.image.PrepareMergeCommand] (EE-ManagedThreadFactory-engine-Thread-179999) [38e01430-80a0-4262-a663-1660bc7738b1] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.PrepareMergeCommand' with failure.
2019-08-12 16:57:54,954+03 ERROR [org.ovirt.engine.core.bll.snapshots.ColdMergeSnapshotSingleDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-20) [38e01430-80a0-4262-a663-1660bc7738b1] Command 'ColdMergeSnapshotSingleDisk' id '8718e1a9-e373-4eb5-bbf8-2f1d7538ec42' failed executing step 'PREPARE_MERGE'
2019-08-12 16:57:55,967+03 ERROR [org.ovirt.engine.core.bll.snapshots.ColdMergeSnapshotSingleDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [38e01430-80a0-4262-a663-1660bc7738b1] Ending command 'org.ovirt.engine.core.bll.snapshots.ColdMergeSnapshotSingleDiskCommand' with failure.
2019-08-12 16:58:04,102+03 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-76) [38e01430-80a0-4262-a663-1660bc7738b1] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.
2019-08-12 16:58:04,138+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-76) [38e01430-80a0-4262-a663-1660bc7738b1] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'snap' for VM 'netapp_vm2'.

Ceph :
** The validation of operation snapshot was with OS and the snapshot was created with memory**
1. Create a live snapshot  - 
Failed at the first, second and third attempts:
2019-08-12 17:02:38,113+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180116) [3ec97a18-29b5-4118-a498-48a82adc3905] Failed in 'FreezeVDS' method
2019-08-12 17:02:38,123+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180116) [3ec97a18-29b5-4118-a498-48a82adc3905] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 17:02:38,124+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180116) [3ec97a18-29b5-4118-a498-48a82adc3905] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:02:38,145+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180116) [3ec97a18-29b5-4118-a498-48a82adc3905] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 17:02:38,173+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180116) [3ec97a18-29b5-4118-a498-48a82adc3905] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 17:02:38,180+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180116) [3ec97a18-29b5-4118-a498-48a82adc3905] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snap123 for VM ceph_vm (User: admin@internal-authz).
2019-08-12 17:02:39,418+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [3ec97a18-29b5-4118-a498-48a82adc3905] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 17:02:39,456+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [3ec97a18-29b5-4118-a498-48a82adc3905] Failed in 'ThawVDS' method
2019-08-12 17:02:39,459+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [3ec97a18-29b5-4118-a498-48a82adc3905] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:02:39,624+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-40) [3ec97a18-29b5-4118-a498-48a82adc3905] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snap123' creation for VM 'ceph_vm'.
2019-08-12 17:03:09,564+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180131) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Failed in 'FreezeVDS' method
2019-08-12 17:03:09,572+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180131) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 17:03:09,572+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180131) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:03:09,578+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180131) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 17:03:09,586+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180131) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 17:03:09,592+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180131) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snap123 for VM ceph_vm (User: admin@internal-authz).
2019-08-12 17:03:10,797+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 17:03:10,842+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Failed in 'ThawVDS' method
2019-08-12 17:03:10,843+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:03:11,001+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-1) [d32b0ac0-15b0-4bbf-bbaa-17f542d32a01] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snap123' creation for VM 'ceph_vm'.

2019-08-12 17:03:42,006+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180146) [3070af91-1672-436e-8976-2ed334dec6eb] Failed in 'FreezeVDS' method
2019-08-12 17:03:42,014+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180146) [3070af91-1672-436e-8976-2ed334dec6eb] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 17:03:42,014+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180146) [3070af91-1672-436e-8976-2ed334dec6eb] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:03:42,020+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180146) [3070af91-1672-436e-8976-2ed334dec6eb] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 17:03:42,033+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180146) [3070af91-1672-436e-8976-2ed334dec6eb] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 17:03:42,038+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180146) [3070af91-1672-436e-8976-2ed334dec6eb] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snap123 for VM ceph_vm (User: admin@internal-authz).
2019-08-12 17:03:43,271+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [3070af91-1672-436e-8976-2ed334dec6eb] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 17:03:43,308+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [3070af91-1672-436e-8976-2ed334dec6eb] Failed in 'ThawVDS' method
2019-08-12 17:03:43,309+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [3070af91-1672-436e-8976-2ed334dec6eb] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='438d24da-aba4-4cb5-ac92-eae4879b7d22'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:03:43,459+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-34) [3070af91-1672-436e-8976-2ed334dec6eb] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snap123' creation for VM 'ceph_vm'.

2.Preview - Failed
2019-08-12 17:06:40,765+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (default task-152) [41557e13-6453-4a16-9201-02a080c53db8] Failed executing creating volume from snapshot: java.lang.IllegalArgumentException: Invalid UUID string: Snapshot '%s' not found
2019-08-12 17:06:47,000+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [41557e13-6453-4a16-9201-02a080c53db8] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand' with failure.
2019-08-12 17:06:47,090+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [41557e13-6453-4a16-9201-02a080c53db8] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToSnapshotCommand' with failure.
2019-08-12 17:06:47,630+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [41557e13-6453-4a16-9201-02a080c53db8] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand' with failure.
2019-08-12 17:06:48,283+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-36) [] EVENT_ID: USER_TRY_BACK_TO_SNAPSHOT_FINISH_FAILURE(99), Failed to complete Snapshot-Preview snapshot1234 for VM ceph_vm.

Impossible to make commit & undo

3. Delete snapshot - Failed
2019-08-12 17:08:04,136+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [] Failed in 'HSMGetAllTasksStatusesVDS' method
2019-08-12 17:08:04,150+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command HSMGetAllTasksStatusesVDS failed: '9f696224-6830-4b95-89ac-18f5d3e54f48' is not in list
2019-08-12 17:08:04,178+03 ERROR [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [] BaseAsyncTask::logEndTaskFailure: Task '01cb0f84-5efd-4485-86e7-ae0e1be7a519' (Parent Command 'PrepareMerge', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure:
2019-08-12 17:08:04,214+03 ERROR [org.ovirt.engine.core.bll.storage.disk.image.PrepareMergeCommand] (EE-ManagedThreadFactory-engine-Thread-180259) [5e55a28a-946e-4bfd-8c98-6167a4ed1064] Ending command 'org.ovirt.engine.core.bll.storage.disk.image.PrepareMergeCommand' with failure.
2019-08-12 17:08:12,263+03 ERROR [org.ovirt.engine.core.bll.snapshots.ColdMergeSnapshotSingleDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-58) [5e55a28a-946e-4bfd-8c98-6167a4ed1064] Command 'ColdMergeSnapshotSingleDisk' id '47371573-4f64-41b4-a059-10e0ac8643ef' failed executing step 'PREPARE_MERGE'
2019-08-12 17:08:13,319+03 ERROR [org.ovirt.engine.core.bll.snapshots.ColdMergeSnapshotSingleDiskCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-97) [5e55a28a-946e-4bfd-8c98-6167a4ed1064] Ending command 'org.ovirt.engine.core.bll.snapshots.ColdMergeSnapshotSingleDiskCommand' with failure.
2019-08-12 17:08:14,526+03 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [5e55a28a-946e-4bfd-8c98-6167a4ed1064] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.
2019-08-12 17:08:14,715+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-70) [5e55a28a-946e-4bfd-8c98-6167a4ed1064] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'snapshot1234' for VM 'ceph_vm'.

Kaminario:
** The validation of operation snapshot was with OS and the snapshot was created with memory**
1. Create a live snapshot -
Failed at the first, second and third attempts:
2019-08-12 17:30:58,012+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180829) [8136f798-32b6-40ec-9c20-a7e9c563f491] Failed in 'FreezeVDS' method
2019-08-12 17:30:58,022+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180829) [8136f798-32b6-40ec-9c20-a7e9c563f491] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 17:30:58,023+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180829) [8136f798-32b6-40ec-9c20-a7e9c563f491] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:30:58,031+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180829) [8136f798-32b6-40ec-9c20-a7e9c563f491] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 17:30:58,046+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180829) [8136f798-32b6-40ec-9c20-a7e9c563f491] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 17:30:58,054+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180829) [8136f798-32b6-40ec-9c20-a7e9c563f491] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snapshot2 for VM kaminario_vm (User: admin@internal-authz).
2019-08-12 17:30:59,896+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [8136f798-32b6-40ec-9c20-a7e9c563f491] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 17:30:59,992+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [8136f798-32b6-40ec-9c20-a7e9c563f491] Failed in 'ThawVDS' method
2019-08-12 17:31:00,002+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [8136f798-32b6-40ec-9c20-a7e9c563f491] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:31:00,129+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [8136f798-32b6-40ec-9c20-a7e9c563f491] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snapshot2' creation for VM 'kaminario_vm'.

2019-08-12 17:31:00,002+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [8136f798-32b6-40ec-9c20-a7e9c563f491] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:31:00,129+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [8136f798-32b6-40ec-9c20-a7e9c563f491] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snapshot2' creation for VM 'kaminario_vm'.
2019-08-12 17:31:47,496+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180850) [da8db3b3-b297-4146-92cf-222c6b459a46] Failed in 'FreezeVDS' method
2019-08-12 17:31:47,513+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180850) [da8db3b3-b297-4146-92cf-222c6b459a46] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 17:31:47,513+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180850) [da8db3b3-b297-4146-92cf-222c6b459a46] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:31:47,539+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180850) [da8db3b3-b297-4146-92cf-222c6b459a46] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 17:31:47,558+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180850) [da8db3b3-b297-4146-92cf-222c6b459a46] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 17:31:47,579+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180850) [da8db3b3-b297-4146-92cf-222c6b459a46] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snapshot123 for VM kaminario_vm (User: admin@internal-authz).
2019-08-12 17:31:49,385+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [da8db3b3-b297-4146-92cf-222c6b459a46] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 17:31:49,466+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [da8db3b3-b297-4146-92cf-222c6b459a46] Failed in 'ThawVDS' method
2019-08-12 17:31:49,467+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [da8db3b3-b297-4146-92cf-222c6b459a46] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:31:49,616+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-64) [da8db3b3-b297-4146-92cf-222c6b459a46] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snapshot123' creation for VM 'kaminario_vm'.

2019-08-12 17:32:22,133+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180867) [ba5a7a22-c939-442c-b631-84dbc9c78381] Failed in 'FreezeVDS' method
2019-08-12 17:32:22,142+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180867) [ba5a7a22-c939-442c-b631-84dbc9c78381] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host_mixed_2 command FreezeVDS failed: Guest agent is not responding: QEMU guest agent is not connected
2019-08-12 17:32:22,142+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.FreezeVDSCommand] (EE-ManagedThreadFactory-engine-Thread-180867) [ba5a7a22-c939-442c-b631-84dbc9c78381] Command 'FreezeVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to FreezeVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:32:22,156+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180867) [ba5a7a22-c939-442c-b631-84dbc9c78381] Command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' failed: EngineException: freezeErr (Failed with error freezeErr and code 76)
2019-08-12 17:32:22,170+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engine-Thread-180867) [ba5a7a22-c939-442c-b631-84dbc9c78381] Transaction rolled-back for command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand'.
2019-08-12 17:32:22,176+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-180867) [ba5a7a22-c939-442c-b631-84dbc9c78381] EVENT_ID: USER_FAILED_CREATE_SNAPSHOT(117), Failed to create Snapshot snapshot1234 for VM kaminario_vm (User: admin@internal-authz).
2019-08-12 17:32:23,785+03 ERROR [org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [ba5a7a22-c939-442c-b631-84dbc9c78381] Ending command 'org.ovirt.engine.core.bll.snapshots.CreateSnapshotForVmCommand' with failure.
2019-08-12 17:32:23,862+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [ba5a7a22-c939-442c-b631-84dbc9c78381] Failed in 'ThawVDS' method
2019-08-12 17:32:23,863+03 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ThawVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [ba5a7a22-c939-442c-b631-84dbc9c78381] Command 'ThawVDSCommand(HostName = host_mixed_2, VdsAndVmIDVDSParametersBase:{hostId='bc1d583a-9914-4d1b-9bae-c737c2bb9e19', vmId='c6e1e640-c38d-4f7d-bcb7-c3d8854e781b'})' execution failed: VDSGenericException: VDSErrorException: Failed to ThawVDS, error = Guest agent is not responding: QEMU guest agent is not connected, code = 19
2019-08-12 17:32:23,987+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [ba5a7a22-c939-442c-b631-84dbc9c78381] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_FAILURE(69), Failed to complete snapshot 'snapshot1234' creation for VM 'kaminario_vm'.
 
2.Preview - Failed
2019-08-12 17:37:00,620+03 ERROR [org.ovirt.engine.core.common.utils.cinderlib.CinderlibExecutor] (default task-163) [73f4d8a8-2a44-4f5c-b580-92519ff641eb] cinderlib execution failed: Traceback (most recent call last):
2019-08-12 17:37:05,725+03 ERROR [org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [73f4d8a8-2a44-4f5c-b580-92519ff641eb] Ending command 'org.ovirt.engine.core.bll.snapshots.TryBackToAllSnapshotsOfVmCommand' with failure.
2019-08-12 17:37:05,809+03 ERROR [org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [73f4d8a8-2a44-4f5c-b580-92519ff641eb] Ending command 'org.ovirt.engine.core.bll.storage.disk.managedblock.TryBackToManagedBlockSnapshotCommand' with failure.
2019-08-12 17:37:05,978+03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-92) [] EVENT_ID: USER_TRY_BACK_TO_SNAPSHOT_FINISH_FAILURE(99), Failed to complete Snapshot-Preview snapshot1 for VM kaminario_vm.

Impossible to make commit & undo 

3. Delete snapshot - passed

3PAR :
1. Create a cold snapshot of two disks (MBD disk and iscsi regular disk) - Passed
2. Preview - Passed
3. Commit - Passed
4. Undo - Passed
5. Delete snapshot -Passed
Comment 33 Shir Fishbain 2019-08-12 15:55:45 UTC 
Created attachment 1602986 [details]
New_logs_snapshot_operations_4.3.6_two_disks
Comment 34 Lukas Svaty 2020-09-22 10:22:12 UTC 
CLOSING low QE severity 4.4.2 issuees with NEXT_RELESE. If you believe these bugs indeed require QE Verification feel free to reopen.