Bug 1242149 – A successful image upload using glance taskflow with ceph as backend ends with "HTTPInternalServerError" response status and errors in the log.

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1242149 - A successful image upload using glance taskflow with ceph as backend ends with "HTTPInternalServerError" response status and errors in the log.

Summary:	A successful image upload using glance taskflow with ceph as backend ends wit...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-glance (Show other bugs)
Sub Component:
Version:	7.0 (Kilo)
Hardware:	Unspecified Unspecified

Priority	unspecified Severity unspecified
Target Milestone:	ga
Target Release:	8.0 (Liberty)
Assigned To:	Flavio Percoco
QA Contact:	Avi Avraham
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2015-07-11 13:25 EDT by shilpa
Modified:	2016-04-26 18:13 EDT (History)
CC List:	7 users (show)

See Also:
Fixed In Version:	openstack-glance-11.0.1-2.el7ost
Doc Type:	Bug Fix
Doc Text:
Story Points:	---
Clone Of:
Environment:
Last Closed:	2016-04-07 17:02:15 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Launchpad	1507564	None	None	None	Never
OpenStack gerrit	241681	None	None	None	Never
OpenStack gerrit	248849	None	None	None	Never
Red Hat Product Errata	RHEA-2016:0603	normal	SHIPPED_LIVE	Red Hat OpenStack Platform 8 Enhancement Advisory	2016-04-07 20:53:53 EDT

Groups: None (edit)

Description shilpa 2015-07-11 13:25:15 EDT

Description of problem:
Using ceph as backend, uploaded a glance image using the taskflow option. The image is converted and uploaded successfully. However the glance task-create command ends with a response "HTTPInternalServerError" with traceback errors in the logs 

Version-Release number of selected component (if applicable):
openstack-glance-2015.1.0-6.el7ost.noarch
python-glance-store-0.4.0-1.el7ost.noarch
python-glance-2015.1.0-6.el7ost.noarch
python-glanceclient-0.17.0-2.el7ost.noarch

ceph version: ceph-0.94.1-13.el7cp.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Configure glance to use rbd. 
2. Configure taskflow option.
[task]
task_executor = taskflow
work_dir=/tmp

[taskflow_executor]
engine_mode = serial
max_workers = 10
conversion_format=raw

3.Upload a local image to glance stored on controller node using task-create command.
#glance --os-image-api-version 2 task-create --type import --input '{"import_from_format": "qcow2", "import_^Com": "http://127.0.0.1:8000/cirros-0.3.0-x86_64-disk.img","image_properties": {"name": "CirrOS-test1", "disk_format": "qcow2", "container_format": "bare"}}'

Used the controller node as a simple HTTP server to fetch a qcow2 image from.

4.Glance image is converted to raw and uploaded successfully.

# glance image-list
| b5a8ab54-087e-46f5-ae27-8a960dd4eef9 | CirrOS-test1  | raw         | bare             | 41126400    | active |

5. On ceph image pool the glance image is created successfully.

"rbd info images/b5a8ab54-087e-46f5-ae27-8a960dd4eef9"
root@10.12.27.33's password: 
rbd image 'b5a8ab54-087e-46f5-ae27-8a960dd4eef9':
	size 40162 kB in 5 objects
	order 23 (8192 kB objects)
	block_name_prefix: rbd_data.37de7042fba6
	format: 2
	features: layering
	flags: 


Actual results:
The response status is HTTPInternalServerError with errors in api.log file. 

Expected results:
A valid response status.

Additional info:

Errors in glance api log file:

2015-07-11 11:39:55.687 6663 INFO glance.domain [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task 
[673eecb5-7864-443b-825e-bbf1df21f28f] status changing from processing to processing
2015-07-11 11:39:57.335 6663 INFO glance.common.scripts.image_import.main [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task None: Got image data uri file:///tmp/b5a8ab54-087e-46f5-ae27-8a960dd4eef9 to be imported
2015-07-11 11:39:57.424 6663 WARNING glance_store._drivers.rbd [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal
2015-07-11 11:40:00.745 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task 'import-DeleteFromFS-673eecb5-7864-443b-825e-bbf1df21f28f' (1d54745c-5b1a-48e1-9903-28d27e517715) transitioned into state 'FAILURE' from state 'RUNNING'
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor Traceback (most recent call last):
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor     result = task.execute(**arguments)
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor   File "/usr/lib/python2.7/site-packages/glance/async/flows/base_import.py", line 175, in execute
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor     store_api.delete_from_backend(file_path)
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 288, in delete_from_backend
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor     loc = location.get_location_from_uri(uri, conf=CONF)
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor   File "/usr/lib/python2.7/site-packages/glance_store/location.py", line 75, in get_location_from_uri
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor     raise exceptions.UnknownScheme(scheme=pieces.scheme)
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor UnknownScheme: Unknown scheme 'file' found in URI
2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor 
2015-07-11 11:40:00.767 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task 'import-DeleteFromFS-673eecb5-7864-443b-825e-bbf1df21f28f' (1d54745c-5b1a-48e1-9903-28d27e517715) transitioned into state 'REVERTED' from state 'REVERTING'

2015-07-11 11:40:00.767 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48
9e233a7 - - -] Task 'import-DeleteFromFS-673eecb5-7864-443b-825e-bbf1df21f28f' (1d54745c-5b1a-48e1-9903-28d27e517715) transitioned into state 'REVERTED' from state 'REV
ERTING'
2015-07-11 11:40:00.777 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48
9e233a7 - - -] Task 'import-ImportToStore-673eecb5-7864-443b-825e-bbf1df21f28f' (bfb1b0df-2d3a-4132-84ed-4e8c87025390) transitioned into state 'REVERTED' from state 'RE
VERTING'
2015-07-11 11:40:00.786 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48
9e233a7 - - -] Task 'import-Introspect-673eecb5-7864-443b-825e-bbf1df21f28f' (7467db20-fce0-4e70-aa79-20189d3a65ba) transitioned into state 'REVERTED' from state 'REVER
TING'
2015-07-11 11:40:00.796 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48
9e233a7 - - -] Task 'import-Convert-673eecb5-7864-443b-825e-bbf1df21f28f' (5467b048-2bf0-495a-acff-9cd35db45063) transitioned into state 'FAILURE' from state 'REVERTING
'
2015-07-11 11:40:00.804 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48
9e233a7 - - -] Flow 'import' (74408f53-ef36-418e-b2cc-4a535f109a70) transitioned into state 'FAILURE' from state 'RUNNING'
2015-07-11 11:40:00.807 6663 ERROR glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e
233a7 - - -] Failed to execute task 673eecb5-7864-443b-825e-bbf1df21f28f: 'module' object has no attribute 'remove'
2015-07-11 11:40:00.813 6663 INFO eventlet.wsgi.server [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -
] 
..
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 454, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 582, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
    return self._call_app(env, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
    return self._app(env, _fake_start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
    return self._call_app(env, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
    return self._app(env, _fake_start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 582, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 873, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 897, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/api/v2/tasks.py", line 70, in create     new_task.run(task_executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
    self.base.run(executor)
  File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 488, in run
    super(TaskProxy, self).run(executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
    self.base.run(executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
    self.base.run(executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/__init__.py", line 440, in run
    executor.begin_processing(self.task_id)
  File "/usr/lib/python2.7/site-packages/glance/async/__init__.py", line 64, in begin_processing
    self._run(task_id, task.type)
  File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 131, in _run
    {'task_id': task_id, 'exc': exc.message})
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 127, in _run
    engine.run()
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 96, in run
    for _state in self.run_iter():
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 146, in run_iter
    self._change_state(states.FAILURE)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 130, in run_iter
    failure.Failure.reraise_if_any(failures)
  File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 244, in reraise_if_any
    failures[0].reraise()
  File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 251, in reraise
    six.reraise(*self._exc_info)
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 86, in _revert_task
    result = task.revert(**arguments)
  File "/usr/lib/python2.7/site-packages/glance/async/flows/convert.py", line 99, in revert
    os.path.remove(fs_path)
AttributeError: 'module' object has no attribute 'remove'
2015-07-11 11:40:00.817 6663 INFO eventlet.wsgi.server [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] 10.12.27.41 - - [11/Jul/2015 11:40:00] "POST /v2/tasks HTTP/1.1" 500 139 5.250361
2015-07-11 11:40:37.660 6664 INFO eventlet.wsgi.server [req-48492be9-d01e-4732-86d4-476c412a534b 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] 10.12.27.41 - - [11/Jul/2015 11:40:37] "GET /v1/images/detail?sort_key=name&sort_dir=asc&limit=20 HTTP/1.1" 200 5447 0.940907

Comment 3 Flavio Percoco 2015-10-19 08:06:06 EDT

I'll report this upstream

Comment 4 shilpa 2015-10-20 07:05:12 EDT

Ran this test again with OSP 7 with ceph 1.3.1 with following packages:
openstack-glance-2015.1.1-1.el7ost.noarch
python-glance-2015.1.1-1.el7ost.noarch
python-glanceclient-0.17.0-2.el7ost.noarch

Now the issue is that the image fails to get created on rbd. It just stays queued.

api.conf:

[task]
task_executor = taskflow
work_dir = /tmp


[taskflow_executor]
engine_mode = serial
max_workers = 10
conversion_format = raw


# glance --os-image-api-version 2 task-create --type import --input '{"import_from_format": "qcow2", "import_from":  "http://127.0.0.1:8000/root/cirros-0.3.4-x86_64-disk.img","image_properties": {"name": "CirrOS", "disk_format": "qcow2", "container_format": "bare"}}'
HTTPInternalServerError (HTTP 500)

 Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 454, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 590, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
    return self._call_app(env, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
    return self._app(env, _fake_start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 590, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 881, in __call__
request, **action_args)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 905, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/api/v2/tasks.py", line 70, in create
    new_task.run(task_executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
    self.base.run(executor)
  File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 488, in run
    super(TaskProxy, self).run(executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
    self.base.run(executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
    self.base.run(executor)
  File "/usr/lib/python2.7/site-packages/glance/domain/__init__.py", line 440, in run
    executor.begin_processing(self.task_id)
  File "/usr/lib/python2.7/site-packages/glance/async/__init__.py", line 64, in begin_processing
    self._run(task_id, task.type)
  File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 131, in _run
    {'task_id': task_id, 'exc': exc.message})
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 127, in _run
    engine.run()
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 96, in run
    for _state in self.run_iter():
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 146, in run_iter
    self._change_state(states.FAILURE)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 130, in run_iter
    failure.Failure.reraise_if_any(failures)
  File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 244, in reraise_if_any
    failures[0].reraise()
  File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 251, in reraise
    six.reraise(*self._exc_info)
  File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executorresult = task.revert(**arguments)
  File "/usr/lib/python2.7/site-packages/glance/async/flows/base_import.py", line 184, in revert
    if os.path.exists(result.split("file://")[-1]):
AttributeError: 'Failure' object has no attribute 'split'

Comment 5 shilpa 2015-10-20 07:38:39 EDT

Some more information from the logs:

2015-10-20 07:35:56.307 26043 WARNING glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878
cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Task 'import-Convert-619da113-3112-4d41-911b-c0ae1cfbe18c' (9ca262c6-e7cc-48
83-8c3d-ebde20e6ecbf) transitioned into state 'FAILURE' from state 'RUNNING'
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor Traceback (most recent call last):
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor   File "/usr/lib/python2.7/site-packages/taskflow/engines/actio
n_engine/executor.py", line 67, in _execute_task
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor     result = task.execute(**arguments)
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor   File "/usr/lib/python2.7/site-packages/glance/async/flows/con
vert.py", line 85, in execute
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor     raise RuntimeError(stderr)
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor RuntimeError: libust[26694/26696]: Error: Error opening shm /lt
tng-ust-wait-5-161 (in get_wait_shm() at lttng-ust-comm.c:886)
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor libust[26694/26695]: Error: Error opening shm /lttng-ust-wait-5
 (in get_wait_shm() at lttng-ust-comm.c:886)
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor libust[26694/26696]: Error: Error opening shm /lttng-ust-wait-5-161 (in get_wait_shm() at lttng-ust-comm.c:886)
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor libust[26694/26695]: Error: Error opening shm /lttng-ust-wait-5 (in get_wait_shm() at lttng-ust-comm.c:886)
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor 
2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor 
2015-10-20 07:35:56.321 26043 WARNING glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Task 'import-Convert-619da113-3112-4d41-911b-c0ae1cfbe18c' (9ca262c6-e7cc-4883-8c3d-ebde20e6ecbf) transitioned into state 'FAILURE' from state 'REVERTING'
2015-10-20 07:35:56.327 26043 WARNING glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Flow 'import' (aa2f1fe6-1505-466c-a8cd-9c64b4d34fda) transitioned into state 'FAILURE' from state 'RUNNING'
2015-10-20 07:35:56.330 26043 ERROR glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Failed to execute task 619da113-3112-4d41-911b-c0ae1cfbe18c: 'Failure' object has no attribute 'split'

Comment 6 Sean Cohen 2015-11-12 08:59:16 EST

Patch proposed upstream under review: https://review.openstack.org/#/c/241681/
Sean

Comment 7 Sergey Gotliv 2015-12-03 08:45:59 EST

This is the Liberty backport which is still under review https://review.openstack.org/#/c/248849/

Comment 8 Flavio Percoco 2015-12-15 10:08:22 EST

Backport landed upstream!

Comment 11 errata-xmlrpc 2016-04-07 17:02:15 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0603.html

Note You need to log in before you can comment on or make changes to this bug.