Bug 1492838

Summary: Engine database upgrade: take care of zero 'vm_snapshot_id' fields
Product: Red Hat Enterprise Virtualization Manager Reporter: Gordon Watson <gwatson>
Component: ovirt-engineAssignee: Sharon Gratch <sgratch>
Status: CLOSED ERRATA QA Contact: Liran Rotenberg <lrotenbe>
Severity: high Docs Contact:
Priority: medium    
Version: 4.1.5CC: apinnick, danken, gwatson, ipinto, jbelka, lrotenbe, lsurette, lveyde, mavital, michal.skrivanek, mkalinin, mtessun, rbalakri, Rhev-m-bugs, sgratch, srevivo, tjelinek, tnisan, ykaul, ylavi
Target Milestone: ovirt-4.2.1   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.2.1.3 Doc Type: Bug Fix
Doc Text:
In the past, a bug allowed virtual machines with invalid snapshot IDs to be created in certain scenarios. The virtual machine functioned normally until the user tried to clone or commit the invalid snapshot or to run the virtual machine as 'stateless', resulting in failure or sometimes permanent disk deletion. In the current release, engine-setup checks the database for invalid image IDs. If found, engine-setup displays a warning describing the invalid image ID, the disk alias and virtual machine name assigned to the image, and advises the user to clone the virtual machine to fix the issue.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-05-15 17:45:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gordon Watson 2017-09-18 17:33:49 UTC
Description of problem:

There are several possible scenarios in which a zero 'vm_snapshot_id' field in the images table in the RHV database can be seen.

One such case is where a snapshots table entry exists with a zero 'snapshot_id' too. In this case, a unique uuid can be generated and plugged into both fields. 

This RFE is a request to address this during an engine upgrade.

If the other scenarios can also be addressed, even better.


Version-Release number of selected component (if applicable):

RHEV 4.1


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Dan Kenigsberg 2017-09-18 19:13:09 UTC
This bug adds zero user-facing functionality. It only cleans up some junk from the database during upgrade. Hence, I think it is misleading to call it an RFE.

Gordon, can you add the SQL scripts use in the past to amend customer DBs?

Comment 18 Israel Pinto 2017-12-10 06:50:00 UTC
I have talked with sharon and she managed to reproduce, 
Waiting for the fix.

Comment 30 Liran Rotenberg 2018-02-12 12:17:21 UTC
Verified on:
ovirt-engine-4.2.1.6-0.1.el7.noarch

Steps of verification:

One VM, one snapshot corruption:

1. Create a VM with a disk.
2. Enter the to the engine DB
from the engine: 
# su - postgres
# psql -d engine

3. Take the image_guid (Possible to do from the engine UI in the VM->Snapshots->Disks->Disk Snapshot ID or from the images DB table).

for this example:
# image_guid = 99de0d5a-1f35-4d6d-ad4e-591cd781a971
# vm_snapshot_id = c734531f-11d7-48a2-9daa-b68aa707b31b (from step 4)

4. Take the image vm_snapshot_id ( engine=# select vm_snapshot_id from images where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971'; )

5. Do the three scenarios as mentioned above.
6. After each scenario, activate engine-setup. 

Detailed scenarios:

Scenario 1:
engine=# update images set vm_snapshot_id='00000000-0000-0000-0000-000000000000' where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971';

engine=# update snapshots set snapshot_id = '00000000-0000-0000-0000-000000000000' where snapshot_id = 'c734531f-11d7-48a2-9daa-b68aa707b31b';

Result:

[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]: 
[WARNING] Engine DB is inconsistent due to the existence of invalid 1 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '99de0d5a-1f35-4d6d-ad4e-591cd781a971'}].
         
         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.


Scenario 2:
engine=# update images set vm_snapshot_id='00000000-0000-0000-0000-000000000000' where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971';

Result:

[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]: 
[WARNING] Engine DB is inconsistent due to the existence of invalid 1 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '99de0d5a-1f35-4d6d-ad4e-591cd781a971'}].
         
         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.


Scenario 3:
1) engine=# update images set vm_snapshot_id='00000000-0000-0000-0000-000000000000' where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971';
2) Start the VM.
3) Create a new snapshot.
4) Delete the new snapshot.
5) Power off the VM.
6) Use the DB to verify the snapshot removed from the snapshots table but not from the images table.
The image can be seen from the host:
# ls -al /rhev/data-center/mnt/{nfs_storage}/images/{vm_disk_id}

Result:

[WARNING] Engine DB is inconsistent due to the existence of invalid 1 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '99de0d5a-1f35-4d6d-ad4e-591cd781a971'}].
         
         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.

Another cases made:
Two VMs with snapshot corrupted.

Result:

[WARNING] Engine DB is inconsistent due to the existence of invalid 2 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test_cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '93560be3-1217-44da-ad7c-5e2081bf2212'}, {'vm_name': 'test_cloned_1', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': 'ccbcf333-8ea8-41e5-af41-a8f9ee8007c5'}]. 

         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.

One VM with three corrupted snapshots.

Result:

[WARNING] Engine DB is inconsistent due to the existence of invalid 3 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '7e2cdefe-3fcb-4dfe-b287-640aec89bbe6'}, {'vm_name': 'cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '3cfe0154-5a9a-48ce-8eba-ff89eb4c86e0'}, {'vm_name': 'cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '254a827d-c319-4d89-ada1-1e16abfa8f9a'}].

         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.


Another Information:
Cloning the VM with each problem solved the issue and the warning in engine-setup disappeared, the issue was solved also for the VM with three corrupted snapshots, they were collapsed and fixed.

Comment 34 errata-xmlrpc 2018-05-15 17:45:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

Comment 35 Franta Kust 2019-05-16 13:03:53 UTC
BZ<2>Jira Resync