Bug 1492838 - Engine database upgrade: take care of zero 'vm_snapshot_id' fields
Summary: Engine database upgrade: take care of zero 'vm_snapshot_id' fields
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.1.5
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: ovirt-4.2.1
: ---
Assignee: Sharon Gratch
QA Contact: Liran Rotenberg
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-09-18 17:33 UTC by Gordon Watson
Modified: 2019-05-16 13:06 UTC (History)
20 users (show)

Fixed In Version: ovirt-engine-4.2.1.3
Doc Type: Bug Fix
Doc Text:
In the past, a bug allowed virtual machines with invalid snapshot IDs to be created in certain scenarios. The virtual machine functioned normally until the user tried to clone or commit the invalid snapshot or to run the virtual machine as 'stateless', resulting in failure or sometimes permanent disk deletion. In the current release, engine-setup checks the database for invalid image IDs. If found, engine-setup displays a warning describing the invalid image ID, the disk alias and virtual machine name assigned to the image, and advises the user to clone the virtual machine to fix the issue.
Clone Of:
Environment:
Last Closed: 2018-05-15 17:45:15 UTC
oVirt Team: Virt
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:1488 0 None None None 2018-05-15 17:46:19 UTC
oVirt gerrit 85440 0 master MERGED setup: add a warn to setup if images with zero 'vm_snapshot_id' exist 2021-01-28 07:14:03 UTC

Description Gordon Watson 2017-09-18 17:33:49 UTC
Description of problem:

There are several possible scenarios in which a zero 'vm_snapshot_id' field in the images table in the RHV database can be seen.

One such case is where a snapshots table entry exists with a zero 'snapshot_id' too. In this case, a unique uuid can be generated and plugged into both fields. 

This RFE is a request to address this during an engine upgrade.

If the other scenarios can also be addressed, even better.


Version-Release number of selected component (if applicable):

RHEV 4.1


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Dan Kenigsberg 2017-09-18 19:13:09 UTC
This bug adds zero user-facing functionality. It only cleans up some junk from the database during upgrade. Hence, I think it is misleading to call it an RFE.

Gordon, can you add the SQL scripts use in the past to amend customer DBs?

Comment 18 Israel Pinto 2017-12-10 06:50:00 UTC
I have talked with sharon and she managed to reproduce, 
Waiting for the fix.

Comment 30 Liran Rotenberg 2018-02-12 12:17:21 UTC
Verified on:
ovirt-engine-4.2.1.6-0.1.el7.noarch

Steps of verification:

One VM, one snapshot corruption:

1. Create a VM with a disk.
2. Enter the to the engine DB
from the engine: 
# su - postgres
# psql -d engine

3. Take the image_guid (Possible to do from the engine UI in the VM->Snapshots->Disks->Disk Snapshot ID or from the images DB table).

for this example:
# image_guid = 99de0d5a-1f35-4d6d-ad4e-591cd781a971
# vm_snapshot_id = c734531f-11d7-48a2-9daa-b68aa707b31b (from step 4)

4. Take the image vm_snapshot_id ( engine=# select vm_snapshot_id from images where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971'; )

5. Do the three scenarios as mentioned above.
6. After each scenario, activate engine-setup. 

Detailed scenarios:

Scenario 1:
engine=# update images set vm_snapshot_id='00000000-0000-0000-0000-000000000000' where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971';

engine=# update snapshots set snapshot_id = '00000000-0000-0000-0000-000000000000' where snapshot_id = 'c734531f-11d7-48a2-9daa-b68aa707b31b';

Result:

[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]: 
[WARNING] Engine DB is inconsistent due to the existence of invalid 1 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '99de0d5a-1f35-4d6d-ad4e-591cd781a971'}].
         
         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.


Scenario 2:
engine=# update images set vm_snapshot_id='00000000-0000-0000-0000-000000000000' where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971';

Result:

[ INFO  ] Stage: Setup validation
          During execution engine service will be stopped (OK, Cancel) [OK]: 
[WARNING] Engine DB is inconsistent due to the existence of invalid 1 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '99de0d5a-1f35-4d6d-ad4e-591cd781a971'}].
         
         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.


Scenario 3:
1) engine=# update images set vm_snapshot_id='00000000-0000-0000-0000-000000000000' where image_guid='99de0d5a-1f35-4d6d-ad4e-591cd781a971';
2) Start the VM.
3) Create a new snapshot.
4) Delete the new snapshot.
5) Power off the VM.
6) Use the DB to verify the snapshot removed from the snapshots table but not from the images table.
The image can be seen from the host:
# ls -al /rhev/data-center/mnt/{nfs_storage}/images/{vm_disk_id}

Result:

[WARNING] Engine DB is inconsistent due to the existence of invalid 1 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '99de0d5a-1f35-4d6d-ad4e-591cd781a971'}].
         
         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.

Another cases made:
Two VMs with snapshot corrupted.

Result:

[WARNING] Engine DB is inconsistent due to the existence of invalid 2 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'test_cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '93560be3-1217-44da-ad7c-5e2081bf2212'}, {'vm_name': 'test_cloned_1', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': 'ccbcf333-8ea8-41e5-af41-a8f9ee8007c5'}]. 

         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.

One VM with three corrupted snapshots.

Result:

[WARNING] Engine DB is inconsistent due to the existence of invalid 3 image(s) for virtual machine(s) as follows:
         [{'vm_name': 'cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '7e2cdefe-3fcb-4dfe-b287-640aec89bbe6'}, {'vm_name': 'cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '3cfe0154-5a9a-48ce-8eba-ff89eb4c86e0'}, {'vm_name': 'cloned', 'disk_alias': 'golden_mixed_virtio_template', 'image_guid': '254a827d-c319-4d89-ada1-1e16abfa8f9a'}].

         Please consult support to resolve this issue. Note that the upgrade will be blocked in the subsequent release (4.3) if the issue isn't resolved.
         If you choose to ignore this problem then snapshot operations on the above virtual machine(s) may fail or may corrupt the disk(s).
         To fix this issue, you can clone the virtual machine(s) by starting the engine, searching for the affected
         virtual machine(s) by name (as listed above) and clicking on 'Clone VM' for each virtual machine in the list.
         Warning: If there are snapshots for the cloned virtual machine(s), they will be collapsed.


Another Information:
Cloning the VM with each problem solved the issue and the warning in engine-setup disappeared, the issue was solved also for the VM with three corrupted snapshots, they were collapsed and fixed.

Comment 34 errata-xmlrpc 2018-05-15 17:45:15 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

Comment 35 Franta Kust 2019-05-16 13:03:53 UTC
BZ<2>Jira Resync


Note You need to log in before you can comment on or make changes to this bug.