Red Hat Bugzilla – Bug 1543218
Engine allows importing HE specific volumes and then deleting them.
Last modified: 2018-05-15 13:49:44 EDT
Description of problem: User can completely destroy a HostedEngine install while thinking he is just cleaning up unused disks. In a fresh Hosted-Engine 4.1 install, go to: Administration Portal -> Storage -> hosted_storage -> Disk Import. There are 3 disks showing there, all used by the HostedEngine HA mechanism. - conf_image - metadata_image - lockspace_image There is no description of what they are, all is blank (they should not even be listed there). User thinks it might be some leftover from previous upgrade/deploy and can import and delete them, destroying the environment. See: id | storage_name --------------------------------------+---------------- 1bfd3c51-74f6-45ac-83e7-392c57eb3759 | hosted_storage engine=# select disk_id,size from unregistered_disks where storage_domain_id = '1bfd3c51-74f6-45ac-83e7-392c57eb3759'; disk_id | size --------------------------------------+------------- 8a822870-9629-4215-9b5d-cbd586a8f9a7 | 134217728 <-- metadata_image 863ca357-8f5e-4863-ac5a-db8c1cdd8a52 | 1073741824 <-- lockspace_image fc2edf2c-4f16-426f-bb00-c9562b258be2 | 62277025792 <-- HE Disk 9a5c545d-c0e8-4740-9bb1-22d7c3cb6e27 | 134217728 <-- conf_image HE Disk is in base_disks/images. The other 3 are not and show up for the user to import them. Then the engine allows the user to import them and do whatever he wants, including removing. Version-Release number of selected component (if applicable): 4.1.8.2-0.1.el7 How reproducible: 100% Steps to Reproduce: 1. Deploy HE 2. Administration Portal -> Storage -> hosted_storage -> Disk Import 3. Import the Disks 4. delete/change/move them 5. environment is destroyed, need to follow disaster recovery. Actual results: Hosted-Engine environment destroyed. The only thin Expected results: Do not allow user to shoot himself in the foot. Additional Information: Importing and deleting the 3 disks above: 1. Just to identify which is which: 2018-02-08 12:19:51,932+11 WARN [org.ovirt.engine.core.bll.storage.disk.image.GetUnregisteredDiskQuery] (default task-25) [4286a967-835c-4f81-8b2d-e83e67a6bbf6] Could not parse the description (hosted-engine.metadata) of disk ID '8a822870-9629-4215-9b5d-cbd586a8f9a7'. The description is expected to be in JSON format. 2018-02-08 12:19:55,685+11 WARN [org.ovirt.engine.core.bll.storage.disk.image.GetUnregisteredDiskQuery] (default task-21) [86a606d6-6f58-4169-b71d-773a943bc047] Could not parse the description (hosted-engine.lockspace) of disk ID '9a5c545d-c0e8-4740-9bb1-22d7c3cb6e27'. The description is expected to be in JSON format. 2018-02-08 12:19:56,948+11 WARN [org.ovirt.engine.core.bll.storage.disk.image.GetUnregisteredDiskQuery] (default task-15) [07b6c4b4-f677-4c1d-bad2-ea09023cf7f1] Could not parse the description (HostedEngineConfigurationImage) of disk ID '863ca357-8f5e-4863-ac5a-db8c1cdd8a52'. The description is expected to be in JSON format. 2. All imported 2018-02-08 12:19:53,163+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-14) [4286a967-835c-4f81-8b2d-e83e67a6bbf6] EVENT_ID: USER_REGISTER_DISK_FINISHED_SUCCESS(378), Correlation ID: 4286a967-835c-4f81-8b2d-e83e67a6bbf6, Job ID: eb00e197-89d0-4f9d-b627-a5a713cf2e91, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Disk 'RegisteredDisk_2018-02-08_12-19-52' has been successfully registered as a floating disk. 2018-02-08 12:19:56,145+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-7) [86a606d6-6f58-4169-b71d-773a943bc047] EVENT_ID: USER_REGISTER_DISK_FINISHED_SUCCESS(378), Correlation ID: 86a606d6-6f58-4169-b71d-773a943bc047, Job ID: 73540cfd-62c3-4c8e-be41-57efc6c61be9, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Disk 'RegisteredDisk_2018-02-08_12-19-55' has been successfully registered as a floating disk. 2018-02-08 12:20:00,972+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-12) [07b6c4b4-f677-4c1d-bad2-ea09023cf7f1] EVENT_ID: USER_REGISTER_DISK_FINISHED_SUCCESS(378), Correlation ID: 07b6c4b4-f677-4c1d-bad2-ea09023cf7f1, Job ID: 63973046-15d3-4e80-b596-daffb4c9cebd, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Disk 'RegisteredDisk_2018-02-08_12-20-00' has been successfully registered as a floating disk. 3. All deleted. 2018-02-08 12:20:13,137+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [131b2e7e-5684-43c6-8780-607b5a8d983d] EVENT_ID: USER_FINISHED_REMOVE_DISK(2,014), Correlation ID: 131b2e7e-5684-43c6-8780-607b5a8d983d, Job ID: 47f32f76-0b58-4cea-acca-f05a074a5072, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Disk RegisteredDisk_2018-02-08_12-19-55 was successfully removed from domain hosted_storage (User admin@internal-authz). 2018-02-08 12:20:19,361+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [131b2e7e-5684-43c6-8780-607b5a8d983d] EVENT_ID: USER_FINISHED_REMOVE_DISK(2,014), Correlation ID: 131b2e7e-5684-43c6-8780-607b5a8d983d, Job ID: 5ecb85b7-e549-4085-923e-c46ac6a38283, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Disk RegisteredDisk_2018-02-08_12-20-00 was successfully removed from domain hosted_storage (User admin@internal-authz). 2018-02-08 12:20:23,387+11 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (org.ovirt.thread.pool-6-thread-48) [131b2e7e-5684-43c6-8780-607b5a8d983d] EVENT_ID: USER_FINISHED_REMOVE_DISK(2,014), Correlation ID: 131b2e7e-5684-43c6-8780-607b5a8d983d, Job ID: f66fdb60-b338-46f0-a6e4-e367dfe3c263, Call Stack: null, Custom ID: null, Custom Event ID: -1, Message: Disk RegisteredDisk_2018-02-08_12-19-52 was successfully removed from domain hosted_storage (User admin@internal-authz).
For start, let's add description to the disks...
The description is there but it's just a plain string and not a valid json so I think that the engine simply ignores it.
Created attachment 1393940 [details] Screenshot - no disk alias for HE disks
I think that the description issue comes from here: [root@c74he20180108h1 ~]# source /etc/ovirt-hosted-engine/hosted-engine.conf [root@c74he20180108h1 ~]# vdsm-client Volume getInfo storagepoolID=$spUUID storagedomainID=$sdUUID imageID=$lockspace_image_UUID volumeID=$lockspace_volume_UUID { "status": "OK", "lease": { "owners": [], "version": null }, "domain": "997acd46-4433-4154-b6ea-99ff26da6fe9", "capacity": "134217728", "voltype": "LEAF", "description": "hosted-engine.lockspace", "parent": "00000000-0000-0000-0000-000000000000", "format": "RAW", "generation": 0, "image": "6ed0316a-b777-4679-96a0-835002440109", "uuid": "68b1e121-e47c-41b3-bed9-95fc4440a86b", "disktype": "2", "legality": "LEGAL", "mtime": "0", "apparentsize": "134217728", "truesize": "134217728", "type": "PREALLOCATED", "children": [], "pool": "", "ctime": "1518191441" } [root@c74he20180108h1 ~]# vdsm-client Volume getInfo storagepoolID=$spUUID storagedomainID=$sdUUID imageID=c78da68d-fc54-48a1-9e99-bcd1f9712b21 volumeID=37e76796-21f3-4531-85a1-97e213fb534f { "status": "OK", "lease": { "owners": [], "version": null }, "domain": "997acd46-4433-4154-b6ea-99ff26da6fe9", "capacity": "134217728", "voltype": "LEAF", "description": "{\"Updated\":true,\"Size\":20480,\"Last Updated\":\"Fri Feb 09 19:49:35 CET 2018\",\"Storage Domains\":[{\"uuid\":\"997acd46-4433-4154-b6ea-99ff26da6fe9\"}],\"Disk Description\":\"OVF_STORE\"}", "parent": "00000000-0000-0000-0000-000000000000", "format": "RAW", "generation": 0, "image": "c78da68d-fc54-48a1-9e99-bcd1f9712b21", "uuid": "37e76796-21f3-4531-85a1-97e213fb534f", "disktype": "OVFS", "legality": "LEGAL", "mtime": "0", "apparentsize": "134217728", "truesize": "134217728", "type": "PREALLOCATED", "children": [], "pool": "", "ctime": "1518195252" } On hosted-engine volumes we simply set (from 3.3 I think) the volume description on VDSM side as a plain text label while the volume description on engine generated disks (the OVF_STORE one for example) is a json where the disk description, as shown by the engine, is just "Disk Description" field inside that json. I suppose that the engine simply ignores the plain text volume description since is not a valid json. On node-zero all the disks are created by the engine and so the description is fine by design. We need to find a way to fix the disk descriptions on system deployed in the past and a way to prevent the engine from deleting them (a new special disktype?)
I built ovirt-hosted-engine-setup-2.2.12 half an hour ago
INFO: Bug status (ON_QA) wasn't changed but the folowing should be fixed: [No relevant external trackers attached] For more info please contact: rhv-devops@redhat.com
Works for me on these components on hosts: ovirt-hosted-engine-ha-2.2.7-1.el7ev.noarch ovirt-hosted-engine-setup-2.2.13-1.el7ev.noarch rhvm-appliance-4.2-20180202.0.el7.noarch Linux 3.10.0-861.el7.x86_64 #1 SMP Wed Mar 14 10:21:01 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.5 (Maipo) I saw proper description for all SHE disks, please see the attachment. Moving to verified.
Created attachment 1411027 [details] Screenshot from 2018-03-21 10-14-34.png
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:1488