Bug 1101541 - [engine-backend] [external-provider] engine failure while createVolume task is running in vdsm (as part of importing an image from glance), leaves image in LOCKED state
Summary: [engine-backend] [external-provider] engine failure while createVolume task i...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 3.4.0
Hardware: x86_64
OS: Unspecified
unspecified
high
Target Milestone: ---
: 3.5.0
Assignee: Daniel Erez
QA Contact: Elad
URL:
Whiteboard: storage
Depends On:
Blocks: 1102785 rhev3.5beta 1156165
TreeView+ depends on / blocked
 
Reported: 2014-05-27 13:27 UTC by Elad
Modified: 2016-02-10 17:08 UTC (History)
10 users (show)

Fixed In Version: vt1.3
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1102785 (view as bug list)
Environment:
Last Closed: 2015-02-16 19:09:24 UTC
oVirt Team: Storage
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
logs from engine and vdsm (1.05 MB, application/x-gzip)
2014-05-27 13:27 UTC, Elad
no flags Details


Links
System ID Private Priority Status Summary Last Updated
oVirt gerrit 28211 0 master MERGED core: import glance image - image locked on failure Never
oVirt gerrit 28301 0 ovirt-engine-3.4 MERGED core: import glance image - image locked on failure Never
oVirt gerrit 29412 0 master MERGED core: import glance image - update destinationImage status Never
oVirt gerrit 29523 0 ovirt-engine-3.5 ABANDONED core: import glance image - update destinationImage status Never

Description Elad 2014-05-27 13:27:41 UTC
Created attachment 899500 [details]
logs from engine and vdsm

Description of problem:
In a situation which engine crashes during createVolume, as part of import an image from glance, disk gets stuck in LOCKED stete.

Version-Release number of selected component (if applicable):
av9.2
rhevm-3.4.0-0.21.el6ev.noarch
vdsm-4.14.7-3.el6ev.x86_64


How reproducible:
Always

Steps to Reproduce:
On a shared DC with storage domains attached and integrated glance repository with images:
1. Import an image from glance repository.
2. Restart ovirt-engine service during createVolume task in vdsm

Actual results:
In case engine crashed before it got the response of the image guuid from vdsm, it assume that the image guuid is  00000000-0000-0000-0000-000000000000 and sends vdsm to perform the second phase of image import - downloadImage with image guuid of 00000000-0000-0000-0000-000000000000.

Thread-113::DEBUG::2014-05-27 11:30:26,000::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState
': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}
Thread-113::DEBUG::2014-05-27 11:30:26,000::taskManager::113::TaskManager::(getAllTasksStatuses) Return: {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'message': u"Invalid parameter: 'volUUID=00000000-000
0-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}
Thread-113::INFO::2014-05-27 11:30:26,000::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100, 'mess
age': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}}
Thread-113::DEBUG::2014-05-27 11:30:26,000::task::1185::TaskManager.Task::(prepare) Task=`a5899d91-a84c-476e-bfb2-db7bbb316a0e`::finished: {'allTasksStatus': {'ae50a07b-1458-4de2-ad34-2cea5cce5fbd': {'code': 100,
'message': u"Invalid parameter: 'volUUID=00000000-0000-0000-0000-000000000000'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': 'ae50a07b-1458-4de2-ad34-2cea5cce5fbd'}}}


Disk gets stuck in LOCKED.

Expected results:
Not sure what should happen if engine is unable to get the response of the image guuid from vdsm. 

Additional info: logs from engine and vdsm

Comment 2 Daniel Erez 2014-06-30 13:23:33 UTC
As commented in bug 1102785 comment #3. There's another unresolved issue here. Pushed a patch for addressing it.

Comment 3 Elad 2014-08-26 11:06:30 UTC
If engine is stopped during createVolume task which is being executed by vdsm, when it comes up, it's being updated about the state of the task and in the DownloadImageVDSCommand it sends the right image guuid and not all zeros uuid:

2014-08-26 13:48:09,281 INFO  [org.ovirt.engine.core.vdsbroker.irsbroker.DownloadImageVDSCommand] (org.ovirt.thread.pool-8-thread-25) -- downloadImage parameters:
                dstSpUUID=00000002-0002-0002-0002-000000000267
                dstSdUUID=68f09ca6-786f-42f3-ace2-2e9e21e5dd33
                dstImageGUID=fa679e23-bd87-474c-8d5d-e24500dece75
                dstVolUUID=78b1ad35-4f19-41c3-8ab3-a293eb3c54c6
                importLocation=HttpLocationInfo [url=http://glance.ovirt.org:9292/v1/images/865bfac3-f9cd-40fc-8d66-293639a9918f, headers={}]


Verified using upstream ovirt-3.5 RC1.1

Comment 4 Allon Mureinik 2015-02-16 19:09:24 UTC
RHEV-M 3.5.0 has been released, closing this bug.


Note You need to log in before you can comment on or make changes to this bug.