[engine-backend] [external-provider] engine failure while createVolume task is running in vdsm (as part of importing an image from glance), leaves image in LOCKED state
Created attachment 899500[details]
logs from engine and vdsm
Description of problem:
In a situation which engine crashes during createVolume, as part of import an image from glance, disk gets stuck in LOCKED stete.
Version-Release number of selected component (if applicable):
av9.2
rhevm-3.4.0-0.21.el6ev.noarch
vdsm-4.14.7-3.el6ev.x86_64
How reproducible:
Always
Steps to Reproduce:
On a shared DC with storage domains attached and integrated glance repository with images:
1. Import an image from glance repository.
2. Restart ovirt-engine service during createVolume task in vdsm
Actual results:
In case engine crashed before it got the response of the image guuid from vdsm, it assume that the image guuid is 00000000-0000-0000-0000-000000000000 and sends vdsm to perform the second phase of image import - downloadImage with image guuid of 00000000-0000-0000-0000-000000000000.
Thread-113::DEBUG::2014-05-27 11:30:26,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState
': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}
Thread-113::DEBUG::2014-05-27 11:30:26,000::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-000
0-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}
Thread-113::INFO::2014-05-27 11:30:26,000::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'mess
age': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}}
Thread-113::DEBUG::2014-05-27 11:30:26,000::task::1185::TaskManager.Task::(prepare) Task=`a5899d91-a84c-476e-bfb2-db7bbb316a0e`::finished: {'allTasksStatus': {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100,
'message': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}}
Disk gets stuck in LOCKED.
Expected results:
Not sure what should happen if engine is unable to get the response of the image guuid from vdsm.
Additional info: logs from engine and vdsm
If engine is stopped during createVolume task which is being executed by vdsm, when it comes up, it's being updated about the state of the task and in the DownloadImageVDSCommand it sends the right image guuid and not all zeros uuid:
2014-08-26 13:48:09,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DownloadImageVDSCommand] (org.ovirt.thread.pool-8-thread-25) -- downloadImage parameters:
dstSpUUID=00000002-0002-0002-0002-000000000267
dstSdUUID=68f09ca6-786f-42f3-ace2-2e9e21e5dd33
dstImageGUID=fa679e23-bd87-474c-8d5d-e24500dece75
dstVolUUID=78b1ad35-4f19-41c3-8ab3-a293eb3c54c6
importLocation=HttpLocationInfo [url=http://glance.ovirt.org:9292/v1/images/865bfac3-f9cd-40fc-8d66-293639a9918f, headers={}]
Verified using upstream ovirt-3.5 RC1.1
Created attachment 899500 [details] logs from engine and vdsm Description of problem: In a situation which engine crashes during createVolume, as part of import an image from glance, disk gets stuck in LOCKED stete. Version-Release number of selected component (if applicable): av9.2 rhevm-3.4.0-0.21.el6ev.noarch vdsm-4.14.7-3.el6ev.x86_64 How reproducible: Always Steps to Reproduce: On a shared DC with storage domains attached and integrated glance repository with images: 1. Import an image from glance repository. 2. Restart ovirt-engine service during createVolume task in vdsm Actual results: In case engine crashed before it got the response of the image guuid from vdsm, it assume that the image guuid is 00000000-0000-0000-0000-000000000000 and sends vdsm to perform the second phase of image import - downloadImage with image guuid of 00000000-0000-0000-0000-000000000000. Thread-113::DEBUG::2014-05-27 11:30:26,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState ': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'} Thread-113::DEBUG::2014-05-27 11:30:26,000::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-000 0-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}} Thread-113::INFO::2014-05-27 11:30:26,000::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'mess age': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}} Thread-113::DEBUG::2014-05-27 11:30:26,000::task::1185::TaskManager.Task::(prepare) Task=`a5899d91-a84c-476e-bfb2-db7bbb316a0e`::finished: {'allTasksStatus': {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}} Disk gets stuck in LOCKED. Expected results: Not sure what should happen if engine is unable to get the response of the image guuid from vdsm. Additional info: logs from engine and vdsm