Bug 1608790
| Summary: | Storage domain free space is not updated immediately after disk removal | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [oVirt] ovirt-engine | Reporter: | Elad <ebenahar> | ||||
| Component: | BLL.Storage | Assignee: | Tal Nisan <tnisan> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Elad <ebenahar> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 4.2.5 | CC: | bugs, ebenahar, khakimi, ratamir | ||||
| Target Milestone: | --- | Keywords: | Automation, Reopened | ||||
| Target Release: | --- | Flags: | ebenahar:
needinfo-
|
||||
| Hardware: | x86_64 | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-10-22 08:11:02 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | Storage | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Seems logical, it takes a bit before the domain monitor updates the engine with the new statistics, is it a regression? It might be that the ordering of test cases has changed which causes tests that require disk creation on the domain where the large disk was removed to be created. Most likely the test is the cause, it doesn't get updated immediately, I suggest closing as NOTABUG and reworking the test to allow a certain delay or waiting for the update I disagree. I don't see any reason for this delay to refresh the domain free space. If the disk removal task was finished and the disk got deleted, the domain free space should be updated immediately. I don't think we should adjust the test. How? The space is adjusted by the domain monitor and it updates periodically, you have to wait to the next refresh for the size to adjust We didn't have to wait in the test until now so in my opinion, this is a regression. This is an asynchronous operation, by definition you cannot guarantee that both things will happen together, perhaps your test was delayed by another flow going after the delete. Flows in the system does not have to be rewritten to make the tests work, tests should be written from the beginning in a way that takes into consideration the way the system works. The delete operation is occurring and once the tasks is finished there is no guarantee that the domain monitor will send the updated data for Engine to process in the time frame between the delete and the refreshing of the domain data, the test should sleep for the length of the interval between refreshes at least to guarantee a test that will pass in 100% of the times Removing need info as I don't have anything to add here Closing as not a bug due to comment #7 We will adjust our tests based on comment #11 |
Created attachment 1470674 [details] logs Description of problem: After a disk removal operation is being reported to be finished, the storage domain free space is not updated immediately, it takes a few seconds (10-20). Encountered it in RHV automation while testing disks creation after a large disk removal which held most of the domain free space. Disks creation failed on lack of free space even though the large disk removal was reported as finished before. Version-Release number of selected component (if applicable): How reproducible: Timing issue which reproduces in automation Steps to Reproduce: 1. Create an iSCSI domain 2. Create a preallocated disk on the new domain with size that pretty much equals to the domain free space 3. Delete the disk, wait for disk removal to be finished and try to create a disk on the domain. For example, with a domain with free space of 100G, create a disk of 90G, delete it and try to create a 15G disk right after disk removal is done. Actual results: Disk size is 150G which is closed to the domain free space: 2018-07-25 22:09:35,767+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CreateImageVDSCommand] (default task-42) [disks_create_1d62c081-b62c-4684] START, CreateImageVDSCommand( CreateImageVDSCommandParameters:{storagePoolId='ca9906e8-8fec-11e8-83c1-00163e7bd001', ignoreFailoverLimit='false', storageDomainId='7e30b9e8-d51a-41ba-a97c-afae5cad1e70', imageGroupId='ba60813d-d5f6-4145-be75-7fffa512e0a1', imageSizeInBytes='150323855360', volumeFormat='RAW', newImageId='2b4202b6-bf5c-4f88-b3bc-dd9acc7006d3', imageType='Preallocated', newImageDescription='{"DiskAlias":"disk_type_2522085993","DiskDescription":""}', imageInitialSizeInBytes='0'}), log id: 746e492 Disk is being deleted: 2018-07-25 22:09:58,137+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (default task-64) [disks_delete_6b156148-bc62-4bf1] START, DeleteImageGroupVDSCommand( DeleteImageGroupVDSCommandParameters:{storagePoolId='ca9906e8-8fec-11e8-83c1-00163e7bd001', ignoreFailoverLimit='false', storageDomainId='7e30b9e8-d51a-41ba-a97c-afae5cad1e70', imageGroupId='ba60813d-d5f6-4145-be75-7fffa512e0a1', postZeros='false', discard='true', forceDelete='false'}), log id: 1aa01408 Disk removal is finished: 2018-07-25 22:09:58,522+03 INFO [org.ovirt.engine.core.bll.storage.disk.RemoveDiskCommand] (default task-64) [disks_delete_6b156148-bc62-4bf1] Lock freed to object 'EngineLock:{exclusiveLocks='[ba60813d-d5f6-4145-be75-7fffa512e0a1=DISK]', sharedLocks=''}' After that, disk creation on the same domain, 16 seconds after, fails with lack of free space on the domain: 2018-07-25 22:10:14,165+03 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-2) [] Operation Failed: [Cannot add VM. Low disk space on Storage Domain iscsi_1.] 2018-07-25 22:10:15,336+03 WARN [org.ovirt.engine.core.bll.AddVmCommand] (default task-40) [] Validation of action 'AddVm' failed for user admin@internal-authz. Reasons: VAR__ACTION__ADD,VAR__TYPE__VM,ACTION_TY PE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName iscsi_1 Expected results: Storage domain free space should be updated immediately after disk deletion. Additional info: