Bug 1519224 - Disk fails validation in live migration due to quota null
Summary: Disk fails validation in live migration due to quota null
Keywords:
Status: CLOSED DUPLICATE of bug 1518693
Alias: None
Product: ovirt-engine
Classification: oVirt
Component: BLL.Storage
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Allon Mureinik
QA Contact: Raz Tamir
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-30 12:35 UTC by Kevin Alon Goldblatt
Modified: 2017-11-30 12:50 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-30 12:42:13 UTC
oVirt Team: Storage


Attachments (Terms of Use)
vdsm, server, engine logs (1.64 MB, application/x-gzip)
2017-11-30 12:50 UTC, Kevin Alon Goldblatt
no flags Details

Description Kevin Alon Goldblatt 2017-11-30 12:35:33 UTC
Description of problem:
Live storage migration failed on 1 out of 5 disks due to quota null error during validation. This was a glance disk

Checked with the following code:
----------------------------------
ovirt-engine-4.2.0-0.5.master.el7.noarch
vdsm-4.20.8-53.gitc3edfc0.el7.centos.x86_64


How reproducible:


 
Checked with the following scenario:
----------------------------------
1. Create a new cloned vm from template with glance disk
2. Add 4 more disks 2 iscsi and 2 nfs disks
3. Start the vm and write data to all disks
4. Start Live storage migration of all the disks
5. All the non glance disks were migrated but the glance disk validation failed with VAR__ACTION__MOVE,VAR__TYPE__DISK,ACTION_TYPE_FAILED_QUOTA_NULL_NOT_ALLOWED
6. The same operation with 5 non glance disks worked fine

Actual results:
All the non glance disks were migrated but the glance disk validation failed with VAR__ACTION__MOVE,VAR__TYPE__DISK,ACTION_TYPE_FAILED_QUOTA_NULL_NOT_ALLOWED


Expected results:
All disks should have succeeded

Additional info:

Engine.log
--------------------
2017-11-30 12:29:33,881+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetQemuImageInfoVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [7b08071d] FINISH, GetQemuImageInfoVDSCommand, return: 
org.ovirt.engine.core.common.businessentities.storage.QemuImageInfo@4633822f, log id: f113656
2017-11-30 12:29:33,901+02 INFO  [org.ovirt.engine.core.bll.snapshots.CreateAllSnapshotsFromVmCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] Lock freed to object 'EngineLock:{exclusiveLocks='[40
8ed615-1f42-4d9f-b51b-41d35b6b3a34=LIVE_STORAGE_MIGRATION]', sharedLocks=''}'
2017-11-30 12:29:33,962+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-94) [] EVENT_ID: USER_CREATE_SNAPSHOT_FINISHED_SUCCESS(68),
 Snapshot '1494711_Disk4 Auto-generated for Live Storage Migration' creation for VM '1494711' has been completed.
2017-11-30 12:29:34,117+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-22825) [71f70ae9] EVENT_ID: USER_MOVED_DISK_FINISHED_FAILURE(2,011),
 User admin@internal-authz have failed to move disk GlanceDisk-9c3e573 to domain block_3.
2017-11-30 12:29:34,174+02 WARN  [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engine-Thread-22825) [] Validation of action 'LiveMigrateDisk' failed for user admin@inter
nal-authz. Reasons: VAR__ACTION__MOVE,VAR__TYPE__DISK,ACTION_TYPE_FAILED_QUOTA_NULL_NOT_ALLOWED
2017-11-30 12:29:34,176+02 INFO  [org.ovirt.engine.core.bll.storage.lsm.LiveMigrateDiskCommand] (EE-ManagedThreadFactory-engine-Thread-22825) [] Lock freed to object 'EngineLock:{exclusiveLocks='[408ed615-1f42-4
d9f-b51b-41d35b6b3a34=LIVE_STORAGE_MIGRATION]', sharedLocks=''}'
2017-11-30 12:29:34,186+02 INFO  [org.ovirt.engine.core.bll.storage.disk.MoveDiskCommand] (EE-ManagedThreadFactory-engine-Thread-22825) [] Lock freed to object 'EngineLock:{exclusiveLocks='[cf725330-f4a0-4805-95
ac-37a86f81c226=DISK]', sharedLocks=''}'
2017-11-30 12:29:35,349+02 INFO  [org.ovirt.engine.core.bll.storage.lsm.CreateImagePlaceholderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [7b08071d] Running command: CreateImagePlaceholderComman
d internal: true.
2017-11-30 12:29:35,378+02 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.CloneImageGroupStructureVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-85) [7b08071d] START, CloneImageGroupStructureVDSCo
mmand( TargetDomainImageGroupVDSCommandParameters:{storagePoolId='554f5132-0233-44df-8266-1827c26a2731', ignoreFailoverLimit='false', storageDomainId='3af1fd42-ee7f-4ced-bd04-14a695abf210', imageGroupId='4d438e1
8-747b-42fa-8db2-87b152775a2f', dstDomainId='b71445d9-7f47-4706-958f-16b5c7c068f6'}), log id: 4d9521c1



Only 4 merges performed:
-------------------
2017-11-30 12:31:16,119+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [4ad787a5] START, MergeVDSCommand(HostName = green-vdsb.qa.lab.tlv.redhat.com, MergeVDSCommandParameters:{hostId='33578954-26ec-4a07-94ec-e94fbe09b8b3', vmId='408ed615-1f42-4d9f-b51b-41d35b6b3a34', storagePoolId='554f5132-0233-44df-8266-1827c26a2731', storageDomainId='b71445d9-7f47-4706-958f-16b5c7c068f6', imageGroupId='b55b9cf5-e811-4d2c-ba7b-8fde1a5a71c1', imageId='c003dea2-1ba0-4993-8096-452fd9b5d263', baseImageId='c512318b-1ff3-473f-a88a-502976488bac', topImageId='c003dea2-1ba0-4993-8096-452fd9b5d263', bandwidth='0'}), log id: 48e65f88
2017-11-30 12:31:18,472+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-1) [4ad787a5] FINISH, MergeVDSCommand, return: 396b314c-69cb-4c1d-9cb0-33f38b1d8dc7, log id: 48e65f88
2017-11-30 12:32:16,145+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-5) [744c08f2] START, MergeVDSCommand(HostName = green-vdsb.qa.lab.tlv.redhat.com, MergeVDSCommandParameters:{hostId='33578954-26ec-4a07-94ec-e94fbe09b8b3', vmId='408ed615-1f42-4d9f-b51b-41d35b6b3a34', storagePoolId='554f5132-0233-44df-8266-1827c26a2731', storageDomainId='8820b01d-dc2a-4994-93dd-4f0d2c92400a', imageGroupId='5f760559-4d88-449f-8abb-8c71f2abc429', imageId='122c9ebd-e821-4f71-9488-bf8b1175341e', baseImageId='d55c7ac9-a2de-49f4-8a50-44a6c4d59221', topImageId='122c9ebd-e821-4f71-9488-bf8b1175341e', bandwidth='0'}), log id: 773475c2
2017-11-30 12:32:16,861+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-5) [744c08f2] FINISH, MergeVDSCommand, return: 78b59cce-8fc5-421f-9fb9-3d12360f8894, log id: 773475c2
2017-11-30 12:32:21,144+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [7b08071d] START, MergeVDSCommand(HostName = green-vdsb.qa.lab.tlv.redhat.com, MergeVDSCommandParameters:{hostId='33578954-26ec-4a07-94ec-e94fbe09b8b3', vmId='408ed615-1f42-4d9f-b51b-41d35b6b3a34', storagePoolId='554f5132-0233-44df-8266-1827c26a2731', storageDomainId='b71445d9-7f47-4706-958f-16b5c7c068f6', imageGroupId='4d438e18-747b-42fa-8db2-87b152775a2f', imageId='4c3d63c6-9f57-44a4-8712-2470ce0289ec', baseImageId='49d21ec9-2f6e-4193-adcd-e22127b9be0a', topImageId='4c3d63c6-9f57-44a4-8712-2470ce0289ec', bandwidth='0'}), log id: a5d0cad
2017-11-30 12:32:21,678+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-7) [7b08071d] FINISH, MergeVDSCommand, return: 37671372-e551-4df8-84d1-3bacd80410d8, log id: a5d0cad
2017-11-30 12:33:10,195+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [57061e3e] START, MergeVDSCommand(HostName = green-vdsb.qa.lab.tlv.redhat.com, MergeVDSCommandParameters:{hostId='33578954-26ec-4a07-94ec-e94fbe09b8b3', vmId='408ed615-1f42-4d9f-b51b-41d35b6b3a34', storagePoolId='554f5132-0233-44df-8266-1827c26a2731', storageDomainId='8820b01d-dc2a-4994-93dd-4f0d2c92400a', imageGroupId='bee64d5d-4f30-401c-af5d-ba5fdd7b6347', imageId='c8fec057-8ba8-470b-b4f9-e3df1ebe2104', baseImageId='ddbc6075-0984-4b55-a5ff-083e6eb5643d', topImageId='c8fec057-8ba8-470b-b4f9-e3df1ebe2104', bandwidth='0'}), log id: 5f48b00a
2017-11-30 12:33:10,948+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedThreadFactory-commandCoordinator-Thread-8) [57061e3e] FINISH, MergeVDSCommand, return: 31633f93-195f-436c-9370-058d55216624, log id: 5f48b00a


Only 4 snapshots created and deleted:
--------------------------------------
2017-11-30 12:28:01,902+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-22825) [57061e3e] EVENT_ID: USER_CREATE_SNAPSHOT(45), Snapshot '1494711_Disk1 Auto-generated for Live Storage Migration' creation for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:28:20,911+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-22825) [744c08f2] EVENT_ID: USER_CREATE_SNAPSHOT(45), Snapshot '1494711_Disk2 Auto-generated for Live Storage Migration' creation for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:28:53,202+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-22825) [4ad787a5] EVENT_ID: USER_CREATE_SNAPSHOT(45), Snapshot '1494711_Disk3 Auto-generated for Live Storage Migration' creation for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:29:13,503+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-22825) [7b08071d] EVENT_ID: USER_CREATE_SNAPSHOT(45), Snapshot '1494711_Disk4 Auto-generated for Live Storage Migration' creation for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:31:14,959+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-44) [] EVENT_ID: USER_REMOVE_SNAPSHOT(342), Snapshot '1494711_Disk3 Auto-generated for Live Storage Migration' deletion for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:32:13,132+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-93) [] EVENT_ID: USER_REMOVE_SNAPSHOT(342), Snapshot '1494711_Disk2 Auto-generated for Live Storage Migration' deletion for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:32:17,841+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-12) [] EVENT_ID: USER_REMOVE_SNAPSHOT(342), Snapshot '1494711_Disk4 Auto-generated for Live Storage Migration' deletion for VM '1494711' was initiated by admin@internal-authz.
2017-11-30 12:33:07,999+02 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-61) [] EVENT_ID: USER_REMOVE_SNAPSHOT(342), Snapshot '1494711_Disk1 Auto-generated for Live Storage Migration' deletion for VM '1494711' was initiated by admin@internal-authz.

Comment 1 Raz Tamir 2017-11-30 12:42:13 UTC

*** This bug has been marked as a duplicate of bug 1518693 ***

Comment 2 Kevin Alon Goldblatt 2017-11-30 12:50:41 UTC
Created attachment 1360924 [details]
vdsm, server, engine logs

Added logs


Note You need to log in before you can comment on or make changes to this bug.