Bug 1890418
| Summary: | While LSM is being performed, if there is no enough free space on the target domain, Live Merge (as part of LSM) fails | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Evelina Shames <eshames> | ||||||
| Component: | BLL.Storage | Assignee: | Tal Nisan <tnisan> | ||||||
| Status: | CLOSED NOTABUG | QA Contact: | Avihai <aefrat> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 4.4.3.7 | CC: | bugs, bzlotnik | ||||||
| Target Milestone: | --- | Keywords: | Automation, Regression, Reopened | ||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2020-11-24 08:02:06 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
More information: - The VM from a quick look looks fine (tried to log in and it works fine). - If powering off the VM, deleting the big disk (to free some space) and then powering on the VM again - The VM from a quick look looks fine. - There is a snapshot leftover: 'latest-rhel-guest-image-8.2-infra Auto-generated for Live Storage Migration' - The disk moved to the target domain while it was not supposed to. Although, there was an error message on the UI that the LSM failed. As discussed, this shows that the situation is actually improved after our migration to using `qemu-img measure` instead of the estimations we used to do, since now less space is required to create the target disk which previously failed Reopening this bug because Live Merge as part of LSM fails, that means that LSM operation fails. As Benny mentioned, in such case, now there is enough space for the disk movement but there is no enough space for Live Merge. Moreover, the event on the UI is not informative for the customer - it pops as 'Removing snapshot failed' and a general 'Migrating disk from src to dst failed' Attaching screenshot. Created attachment 1725566 [details]
UI_errors
Benny, can we indicate on the "Migrating disk..." task that it succeeded and mark the operation as success and only audit log on the failure in live merge? I think it is considered successful, the message but can be improved to indicate that live merge is the leftover, but as discussed with Shir I think it should be a separate bug because there is no regression here Fine by me, please open a separate bug and close this one again (In reply to Tal Nisan from comment #7) > Fine by me, please open a separate bug and close this one again Done - Bug 1900992. Closing this bug. |
Created attachment 1723429 [details] engine+vdsm-logs Description of problem: While LSM is being performed, if for some reason there is no enough space on the target SD (for example, while LSM is being performed, a disk was created and caused a low space on target domain), LSM should fail and the disk should not be moved. From 4.4.3.6, this scenario ends with a disk that apparently moved to the target domain (on the ui you can see that the disk resides on the target domain), there is an error that LSM failed but not absolutely, looks like there are some leftovers from reverting the operation. Engine log: 2020-10-22 09:23:20,380+03 ERROR [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-83) [8fa969e3-a4cf-46ba-aacd-1a06dcc6033a] Ending command 'org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand' with failure. 2020-10-22 09:23:20,442+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-83) [8fa969e3-a4cf-46ba-aacd-1a06dcc6033a] EVENT_ID: USER_MOVE_IMAGE_GROUP_FAILED_TO_DELETE_SRC_IMAGE(2,025), Possible failure while deleting latest-rhel-guest-image-8.2-infra from the source Storage Domain iscsi_2 during the move operation. The Storage Domain may be manually cleaned-up from possible leftovers (User:admin@internal-authz). VDSM log: 2020-10-22 09:22:22,166+0300 ERROR (mailbox-spm/4) [storage.SPM.Messages.Extend] processRequest: Exception caught while trying to extend volume: c492129b-57f4-4e23-8e7f-22241d4e8421 in domain: 41f61196-2d3e-46c5-90e3-c0a5fc662806 (mailbox:179) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/mailbox.py", line 173, in processRequest pool.extendVolume(volume['domainID'], volume['volumeID'], size) File "/usr/lib/python3.6/site-packages/vdsm/storage/securable.py", line 79, in wrapper return method(self, *args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/storage/sp.py", line 1426, in extendVolume sdCache.produce(sdUUID).extendVolume(volumeUUID, size, isShuttingDown) File "/usr/lib/python3.6/site-packages/vdsm/storage/blockSD.py", line 1573, in extendVolume lvm.extendLV(self.sdUUID, volumeUUID, size) # , isShuttingDown) File "/usr/lib/python3.6/site-packages/vdsm/storage/lvm.py", line 1679, in extendLV % (vgName, lvName, free_extents, needed_extents)) vdsm.storage.exception.VolumeGroupSizeError: Volume Group not big enough: ('Not enough free extents for extending LV 41f61196-2d3e-46c5-90e3-c0a5fc662806/c492129b-57f4-4e23-8e7f-22241d4e8421 (free=9, needed=23)',) 2020-10-22 09:22:23,480+0300 ERROR (mailbox-hsm/0) [storage.TaskManager.Task] (Task='0d35f137-35d4-4c05-aaf2-610d687d0ca6') Unexpect ed error (task:880) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run return fn(*args, **kargs) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 1292, in __afterReplicaExtension self.__verifyVolumeExtension(volInfo) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 1281, in __verifyVolumeExtension (volInfo['name'], volInfo['domainID'], volInfo['volumeID'])) RuntimeError: Volume extension failed for vda (domainID: 41f61196-2d3e-46c5-90e3-c0a5fc662806, volumeID: c492129b-57f4-4e23-8e7f-222 41d4e8421) 2020-10-22 09:22:23,481+0300 ERROR (mailbox-hsm/0) [storage.ThreadPool.WorkerThread] FINISH task e506c221-e37e-4849-a5dc-ef6ae463b63f failed (cmd=<function runTask at 0x7fb179991d08>, args=(<bound method Vm.__afterReplicaExtension of <vdsm.virt.vm.Vm object at 0x7fb1305dbd68>>, {'domainID': '41f61196-2d3e-46c5-90e3-c0a5fc662806', 'imageID': 'e9882cd9-0d0b-42ce-a368-79d293d44d3f', 'name': 'vda', 'newSize': 3221225472, 'poolID': 'd4cd7761-9c57-42a7-89ce-f0fb471600c4', 'volumeID': 'c492129b-57f4-4e23-8e7f-22241d4e8421', 'clock': <Clock(total=2.31*, extend-replica=2.10, refresh-replica=0.20)>})) (threadPool:154) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/storage/threadPool.py", line 150, in _processNextTask cmd(args) File "/usr/lib/python3.6/site-packages/vdsm/storage/mailbox.py", line 88, in runTask ctask.prepare(cmd, *args) File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 110, in wrapper return m(self, *a, **kw) File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 1198, in prepare raise self.error File "/usr/lib/python3.6/site-packages/vdsm/storage/task.py", line 887, in _run return fn(*args, **kargs) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 1292, in __afterReplicaExtension self.__verifyVolumeExtension(volInfo) File "/usr/lib/python3.6/site-packages/vdsm/virt/vm.py", line 1281, in __verifyVolumeExtension (volInfo['name'], volInfo['domainID'], volInfo['volumeID'])) RuntimeError: Volume extension failed for vda (domainID: 41f61196-2d3e-46c5-90e3-c0a5fc662806, volumeID: c492129b-57f4-4e23-8e7f-22241d4e8421) Vm's disk details before LSM: <disk href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f" id="e9882cd9-0d0b-42ce-a368-79d293d44d3f"> <actions> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/reduce" rel="reduce"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/copy" rel="copy"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/export" rel="export"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/move" rel="move"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/refreshlun" rel="refreshlun"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/sparsify" rel="sparsify"/> </actions> <name>latest-rhel-guest-image-8.2-infra</name> <description>latest-rhel-guest-image-8.2-infra (91fc53b)</description> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/permissions" rel="permissions"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/disksnapshots" rel="disksnapshots"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/statistics" rel="statistics"/> <actual_size>2684354560</actual_size> <alias>latest-rhel-guest-image-8.2-infra</alias> <backup>none</backup> <content_type>data</content_type> <format>cow</format> <image_id>b81a4470-d8d8-4f1f-b611-bf749cb6a1f8</image_id> <propagate_errors>false</propagate_errors> <provisioned_size>10737418240</provisioned_size> <qcow_version>qcow2_v3</qcow_version> <shareable>false</shareable> <sparse>true</sparse> <status>ok</status> <storage_type>image</storage_type> <total_size>2684354560</total_size> <wipe_after_delete>false</wipe_after_delete> <disk_profile href="/ovirt-engine/api/diskprofiles/c738b7f3-cc31-4787-a55b-de051fafe6db" id="c738b7f3-cc31-4787-a55b-de051fafe6db"/> <quota href="/ovirt-engine/api/datacenters/d4cd7761-9c57-42a7-89ce-f0fb471600c4/quotas/1a88d395-9160-4bd5-9fc8-274484256bbb" id="1a88d395-9160-4bd5-9fc8-274484256bbb"/> <storage_domains> <storage_domain href="/ovirt-engine/api/storagedomains/48a54342-2a20-4955-9fc0-9e597138a7d1" id="48a54342-2a20-4955-9fc0-9e597138a7d1"/> </storage_domains> </disk> Big disk details: <disk href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3" id="da80b012-3256-4aa6-a26d-81f4a123b8d3"> <actions> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/reduce" rel="reduce"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/copy" rel="copy"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/export" rel="export"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/move" rel="move"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/refreshlun" rel="refreshlun"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/sparsify" rel="sparsify"/> </actions> <name>bug_disk</name> <description/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/permissions" rel="permissions"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/disksnapshots" rel="disksnapshots"/> <link href="/ovirt-engine/api/disks/da80b012-3256-4aa6-a26d-81f4a123b8d3/statistics" rel="statistics"/> <actual_size>141733920768</actual_size> <alias>bug_disk</alias> <backup>none</backup> <content_type>data</content_type> <format>raw</format> <image_id>f457bf89-f282-451f-aa09-2bfbbb437e63</image_id> <propagate_errors>false</propagate_errors> <provisioned_size>141733920768</provisioned_size> <shareable>false</shareable> <sparse>false</sparse> <status>ok</status> <storage_type>image</storage_type> <total_size>141733920768</total_size> <wipe_after_delete>false</wipe_after_delete> <disk_profile href="/ovirt-engine/api/diskprofiles/793f7349-cd0d-48d9-9f33-bd5045dbb652" id="793f7349-cd0d-48d9-9f33-bd5045dbb652"/> <quota href="/ovirt-engine/api/datacenters/d4cd7761-9c57-42a7-89ce-f0fb471600c4/quotas/1a88d395-9160-4bd5-9fc8-274484256bbb" id="1a88d395-9160-4bd5-9fc8-274484256bbb"/> <storage_domains> <storage_domain href="/ovirt-engine/api/storagedomains/41f61196-2d3e-46c5-90e3-c0a5fc662806" id="41f61196-2d3e-46c5-90e3-c0a5fc662806"/> </storage_domains> </disk> VM's disk details after: <disk href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f" id="e9882cd9-0d0b-42ce-a368-79d293d44d3f"> <actions> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/reduce" rel="reduce"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/copy" rel="copy"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/export" rel="export"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/move" rel="move"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/refreshlun" rel="refreshlun"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/sparsify" rel="sparsify"/> </actions> <name>latest-rhel-guest-image-8.2-infra</name> <description>latest-rhel-guest-image-8.2-infra (91fc53b)</description> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/permissions" rel="permissions"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/disksnapshots" rel="disksnapshots"/> <link href="/ovirt-engine/api/disks/e9882cd9-0d0b-42ce-a368-79d293d44d3f/statistics" rel="statistics"/> <actual_size>1207959552</actual_size> <alias>latest-rhel-guest-image-8.2-infra</alias> <backup>none</backup> <content_type>data</content_type> <format>cow</format> <image_id>c492129b-57f4-4e23-8e7f-22241d4e8421</image_id> <propagate_errors>false</propagate_errors> <provisioned_size>10737418240</provisioned_size> <qcow_version>qcow2_v3</qcow_version> <shareable>false</shareable> <sparse>true</sparse> <status>ok</status> <storage_type>image</storage_type> <total_size>3892314112</total_size> <wipe_after_delete>false</wipe_after_delete> <disk_profile href="/ovirt-engine/api/diskprofiles/793f7349-cd0d-48d9-9f33-bd5045dbb652" id="793f7349-cd0d-48d9-9f33-bd5045dbb652"/> <quota href="/ovirt-engine/api/datacenters/d4cd7761-9c57-42a7-89ce-f0fb471600c4/quotas/1a88d395-9160-4bd5-9fc8-274484256bbb" id="1a88d395-9160-4bd5-9fc8-274484256bbb"/> <storage_domains> <storage_domain href="/ovirt-engine/api/storagedomains/41f61196-2d3e-46c5-90e3-c0a5fc662806" id="41f61196-2d3e-46c5-90e3-c0a5fc662806"/> </storage_domains> </disk> Version-Release number of selected component (if applicable): engine-4.4.3-6+ How reproducible: 100% Steps to Reproduce: 1.Create VM from template (rhel 8.2) 2.Run VM 3.Perform LSM to some target domain 4.While LSM is being performed, create a disk on the target domain so there won’t be enough space for the VM's disk. (for example if the free space on the target domain is 150G, create a 149G disk) Actual results: Looks like LSM fails, but VM's disk moved to the target domain. Expected results: LSM should fail and the VM's disk should stay on the source domain. Additional info: Attaching logs with DEBUG level. The environment still exists, let me know if you need it.