Bug 1242149
| Summary: | A successful image upload using glance taskflow with ceph as backend ends with "HTTPInternalServerError" response status and errors in the log. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | shilpa <smanjara> |
| Component: | openstack-glance | Assignee: | Flavio Percoco <fpercoco> |
| Status: | CLOSED ERRATA | QA Contact: | Avi Avraham <aavraham> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.0 (Kilo) | CC: | eglynn, fpercoco, jschluet, kimi.zhang, scohen, sgotliv, yeylon |
| Target Milestone: | ga | ||
| Target Release: | 8.0 (Liberty) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-glance-11.0.1-2.el7ost | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-04-07 21:02:15 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I'll report this upstream Ran this test again with OSP 7 with ceph 1.3.1 with following packages:
openstack-glance-2015.1.1-1.el7ost.noarch
python-glance-2015.1.1-1.el7ost.noarch
python-glanceclient-0.17.0-2.el7ost.noarch
Now the issue is that the image fails to get created on rbd. It just stays queued.
api.conf:
[task]
task_executor = taskflow
work_dir = /tmp
[taskflow_executor]
engine_mode = serial
max_workers = 10
conversion_format = raw
# glance --os-image-api-version 2 task-create --type import --input '{"import_from_format": "qcow2", "import_from": "http://127.0.0.1:8000/root/cirros-0.3.4-x86_64-disk.img","image_properties": {"name": "CirrOS", "disk_format": "qcow2", "container_format": "bare"}}'
HTTPInternalServerError (HTTP 500)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 454, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 590, in __call__
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__
return request.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__
return self._call_app(env, start_response)
File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app
return self._app(env, _fake_start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 590, in __call__
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
return app(environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
return resp(environ, start_response)
File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
response = self.app(environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
return resp(environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 881, in __call__
request, **action_args)
File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 905, in dispatch
return method(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/glance/api/v2/tasks.py", line 70, in create
new_task.run(task_executor)
File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
self.base.run(executor)
File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 488, in run
super(TaskProxy, self).run(executor)
File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
self.base.run(executor)
File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run
self.base.run(executor)
File "/usr/lib/python2.7/site-packages/glance/domain/__init__.py", line 440, in run
executor.begin_processing(self.task_id)
File "/usr/lib/python2.7/site-packages/glance/async/__init__.py", line 64, in begin_processing
self._run(task_id, task.type)
File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 131, in _run
{'task_id': task_id, 'exc': exc.message})
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 127, in _run
engine.run()
File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 96, in run
for _state in self.run_iter():
File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 146, in run_iter
self._change_state(states.FAILURE)
File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 130, in run_iter
failure.Failure.reraise_if_any(failures)
File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 244, in reraise_if_any
failures[0].reraise()
File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 251, in reraise
six.reraise(*self._exc_info)
File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executorresult = task.revert(**arguments)
File "/usr/lib/python2.7/site-packages/glance/async/flows/base_import.py", line 184, in revert
if os.path.exists(result.split("file://")[-1]):
AttributeError: 'Failure' object has no attribute 'split'
Some more information from the logs: 2015-10-20 07:35:56.307 26043 WARNING glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878 cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Task 'import-Convert-619da113-3112-4d41-911b-c0ae1cfbe18c' (9ca262c6-e7cc-48 83-8c3d-ebde20e6ecbf) transitioned into state 'FAILURE' from state 'RUNNING' 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor Traceback (most recent call last): 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor File "/usr/lib/python2.7/site-packages/taskflow/engines/actio n_engine/executor.py", line 67, in _execute_task 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor result = task.execute(**arguments) 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor File "/usr/lib/python2.7/site-packages/glance/async/flows/con vert.py", line 85, in execute 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor raise RuntimeError(stderr) 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor RuntimeError: libust[26694/26696]: Error: Error opening shm /lt tng-ust-wait-5-161 (in get_wait_shm() at lttng-ust-comm.c:886) 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor libust[26694/26695]: Error: Error opening shm /lttng-ust-wait-5 (in get_wait_shm() at lttng-ust-comm.c:886) 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor libust[26694/26696]: Error: Error opening shm /lttng-ust-wait-5-161 (in get_wait_shm() at lttng-ust-comm.c:886) 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor libust[26694/26695]: Error: Error opening shm /lttng-ust-wait-5 (in get_wait_shm() at lttng-ust-comm.c:886) 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor 2015-10-20 07:35:56.307 26043 TRACE glance.async.taskflow_executor 2015-10-20 07:35:56.321 26043 WARNING glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Task 'import-Convert-619da113-3112-4d41-911b-c0ae1cfbe18c' (9ca262c6-e7cc-4883-8c3d-ebde20e6ecbf) transitioned into state 'FAILURE' from state 'REVERTING' 2015-10-20 07:35:56.327 26043 WARNING glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Flow 'import' (aa2f1fe6-1505-466c-a8cd-9c64b4d34fda) transitioned into state 'FAILURE' from state 'RUNNING' 2015-10-20 07:35:56.330 26043 ERROR glance.async.taskflow_executor [req-642055c2-ec4d-49ea-936e-f8e77392ec3c b2e365fad2fb49ba878cb92a34e3f68b 2b05210c14954f02a25d190060ee22d3 - - -] Failed to execute task 619da113-3112-4d41-911b-c0ae1cfbe18c: 'Failure' object has no attribute 'split' Patch proposed upstream under review: https://review.openstack.org/#/c/241681/ Sean This is the Liberty backport which is still under review https://review.openstack.org/#/c/248849/ Backport landed upstream! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-0603.html |
Description of problem: Using ceph as backend, uploaded a glance image using the taskflow option. The image is converted and uploaded successfully. However the glance task-create command ends with a response "HTTPInternalServerError" with traceback errors in the logs Version-Release number of selected component (if applicable): openstack-glance-2015.1.0-6.el7ost.noarch python-glance-store-0.4.0-1.el7ost.noarch python-glance-2015.1.0-6.el7ost.noarch python-glanceclient-0.17.0-2.el7ost.noarch ceph version: ceph-0.94.1-13.el7cp.x86_64 How reproducible: Always Steps to Reproduce: 1. Configure glance to use rbd. 2. Configure taskflow option. [task] task_executor = taskflow work_dir=/tmp [taskflow_executor] engine_mode = serial max_workers = 10 conversion_format=raw 3.Upload a local image to glance stored on controller node using task-create command. #glance --os-image-api-version 2 task-create --type import --input '{"import_from_format": "qcow2", "import_^Com": "http://127.0.0.1:8000/cirros-0.3.0-x86_64-disk.img","image_properties": {"name": "CirrOS-test1", "disk_format": "qcow2", "container_format": "bare"}}' Used the controller node as a simple HTTP server to fetch a qcow2 image from. 4.Glance image is converted to raw and uploaded successfully. # glance image-list | b5a8ab54-087e-46f5-ae27-8a960dd4eef9 | CirrOS-test1 | raw | bare | 41126400 | active | 5. On ceph image pool the glance image is created successfully. "rbd info images/b5a8ab54-087e-46f5-ae27-8a960dd4eef9" root.27.33's password: rbd image 'b5a8ab54-087e-46f5-ae27-8a960dd4eef9': size 40162 kB in 5 objects order 23 (8192 kB objects) block_name_prefix: rbd_data.37de7042fba6 format: 2 features: layering flags: Actual results: The response status is HTTPInternalServerError with errors in api.log file. Expected results: A valid response status. Additional info: Errors in glance api log file: 2015-07-11 11:39:55.687 6663 INFO glance.domain [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task [673eecb5-7864-443b-825e-bbf1df21f28f] status changing from processing to processing 2015-07-11 11:39:57.335 6663 INFO glance.common.scripts.image_import.main [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task None: Got image data uri file:///tmp/b5a8ab54-087e-46f5-ae27-8a960dd4eef9 to be imported 2015-07-11 11:39:57.424 6663 WARNING glance_store._drivers.rbd [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] since image size is zero we will be doing resize-before-write for each chunk which will be considerably slower than normal 2015-07-11 11:40:00.745 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task 'import-DeleteFromFS-673eecb5-7864-443b-825e-bbf1df21f28f' (1d54745c-5b1a-48e1-9903-28d27e517715) transitioned into state 'FAILURE' from state 'RUNNING' 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor Traceback (most recent call last): 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor result = task.execute(**arguments) 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor File "/usr/lib/python2.7/site-packages/glance/async/flows/base_import.py", line 175, in execute 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor store_api.delete_from_backend(file_path) 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 288, in delete_from_backend 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor loc = location.get_location_from_uri(uri, conf=CONF) 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor File "/usr/lib/python2.7/site-packages/glance_store/location.py", line 75, in get_location_from_uri 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor raise exceptions.UnknownScheme(scheme=pieces.scheme) 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor UnknownScheme: Unknown scheme 'file' found in URI 2015-07-11 11:40:00.745 6663 ERROR glance.async.taskflow_executor 2015-07-11 11:40:00.767 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] Task 'import-DeleteFromFS-673eecb5-7864-443b-825e-bbf1df21f28f' (1d54745c-5b1a-48e1-9903-28d27e517715) transitioned into state 'REVERTED' from state 'REVERTING' 2015-07-11 11:40:00.767 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48 9e233a7 - - -] Task 'import-DeleteFromFS-673eecb5-7864-443b-825e-bbf1df21f28f' (1d54745c-5b1a-48e1-9903-28d27e517715) transitioned into state 'REVERTED' from state 'REV ERTING' 2015-07-11 11:40:00.777 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48 9e233a7 - - -] Task 'import-ImportToStore-673eecb5-7864-443b-825e-bbf1df21f28f' (bfb1b0df-2d3a-4132-84ed-4e8c87025390) transitioned into state 'REVERTED' from state 'RE VERTING' 2015-07-11 11:40:00.786 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48 9e233a7 - - -] Task 'import-Introspect-673eecb5-7864-443b-825e-bbf1df21f28f' (7467db20-fce0-4e70-aa79-20189d3a65ba) transitioned into state 'REVERTED' from state 'REVER TING' 2015-07-11 11:40:00.796 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48 9e233a7 - - -] Task 'import-Convert-673eecb5-7864-443b-825e-bbf1df21f28f' (5467b048-2bf0-495a-acff-9cd35db45063) transitioned into state 'FAILURE' from state 'REVERTING ' 2015-07-11 11:40:00.804 6663 WARNING glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a48 9e233a7 - - -] Flow 'import' (74408f53-ef36-418e-b2cc-4a535f109a70) transitioned into state 'FAILURE' from state 'RUNNING' 2015-07-11 11:40:00.807 6663 ERROR glance.async.taskflow_executor [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e 233a7 - - -] Failed to execute task 673eecb5-7864-443b-825e-bbf1df21f28f: 'module' object has no attribute 'remove' 2015-07-11 11:40:00.813 6663 INFO eventlet.wsgi.server [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - - ] .. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 454, in handle_one_response result = self.application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 582, in __call__ response = req.get_response(self.application) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send application, catch_exc_info=False) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__ return request.get_response(self.application) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send application, catch_exc_info=False) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__ return self._call_app(env, start_response) File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app return self._app(env, _fake_start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__ return request.get_response(self.application) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send application, catch_exc_info=False) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__ return self._call_app(env, start_response) File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app return self._app(env, _fake_start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 582, in __call__ response = req.get_response(self.application) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send application, catch_exc_info=False) File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__ return app(environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ response = self.app(environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ return resp(environ, start_response) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 873, in __call__ request, **action_args) File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 897, in dispatch return method(*args, **kwargs) File "/usr/lib/python2.7/site-packages/glance/api/v2/tasks.py", line 70, in create new_task.run(task_executor) File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run self.base.run(executor) File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 488, in run super(TaskProxy, self).run(executor) File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run self.base.run(executor) File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 200, in run self.base.run(executor) File "/usr/lib/python2.7/site-packages/glance/domain/__init__.py", line 440, in run executor.begin_processing(self.task_id) File "/usr/lib/python2.7/site-packages/glance/async/__init__.py", line 64, in begin_processing self._run(task_id, task.type) File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 131, in _run {'task_id': task_id, 'exc': exc.message}) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/glance/async/taskflow_executor.py", line 127, in _run engine.run() File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 96, in run for _state in self.run_iter(): File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 146, in run_iter self._change_state(states.FAILURE) File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 119, in __exit__ six.reraise(self.type_, self.value, self.tb) File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 130, in run_iter failure.Failure.reraise_if_any(failures) File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 244, in reraise_if_any failures[0].reraise() File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 251, in reraise six.reraise(*self._exc_info) File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 86, in _revert_task result = task.revert(**arguments) File "/usr/lib/python2.7/site-packages/glance/async/flows/convert.py", line 99, in revert os.path.remove(fs_path) AttributeError: 'module' object has no attribute 'remove' 2015-07-11 11:40:00.817 6663 INFO eventlet.wsgi.server [req-c4c286ec-2248-4d93-b631-a2270de43ec8 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] 10.12.27.41 - - [11/Jul/2015 11:40:00] "POST /v2/tasks HTTP/1.1" 500 139 5.250361 2015-07-11 11:40:37.660 6664 INFO eventlet.wsgi.server [req-48492be9-d01e-4732-86d4-476c412a534b 7a6bd2901f0642b18be91d82782ed1ca 3954f0a8635249eba2eda9a489e233a7 - - -] 10.12.27.41 - - [11/Jul/2015 11:40:37] "GET /v1/images/detail?sort_key=name&sort_dir=asc&limit=20 HTTP/1.1" 200 5447 0.940907