Description of problem: Customer is trying to create snapshots but is getting several errors and permission denied. He is able to create using the command line, but in horizon he is unable to finish the process and receive the following errors: [...] After applied a lot of snapshots test, we discover that when the process started from CLI (command line) all the tests was concluded with success. But, when we did the same process from Horizon we had a problem. Follows the logs in the compute node where the instance was running: Timeout waiting for vif plugging callback for instance 2e69db49-0a61-4b15-952b-25be2f221727 nova-compute.log:2015-07-21 16:52:32.761 27335 ERROR nova.compute.manager [req-381178e9-6a26-47de-a356-4970db6fb857 e7d85171285f4a5193379ef3ff103809 41703e14258949c2a2dac3d5c07eb17c] [instance: 2e69db49-0a61-4b15-952b-25be2f221727] Error while trying to clean up image fca72107-5816-49f4-bba4-78de729c1675 nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] Traceback (most recent call last): nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] image_service.delete(context, image_id) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 397, in delete nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] self._client.call(context, 1, 'delete', image_id) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 216, in call nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] return getattr(client.images, method)(*args, **kwargs) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 256, in delete nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] resp = self._delete("/v1/images/%s" % base.getid(image))[0] nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/glanceclient/openstack/common/apiclient/base.py", line 202, in _delete nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] return self.client.delete(url) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 355, in delete nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] return self.raw_request("DELETE", url, **kwargs) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 328, in raw_request nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] return self._http_request(url, method, **kwargs) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 264, in _http_request nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] raise exc.from_response(resp, body_str) nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] HTTPUnauthorized: 401 Unauthorized nova-compute.log:2015-07-21 16:52:32.761 27335 TRACE nova.compute.manager [instance: 2e69db49-0a61-4b15-952b-25be2f221727] This server could not verify that you are authorized to access the document you requested. Either you supplied the wrong credentials (e.g., bad password), or your browser does not understand how to supply the credentials required. FYI: All the tests were made with admin user (horizon and CLI). [...] I also went into the customer's logs and found: [...] 2015-06-23 09:32:51.427 2262 TRACE oslo.messaging.rpc.dispatcher return self._http_request(url, method, **kwargs) 2015-06-23 09:32:51.427 2262 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 264, in _http_request 2015-06-23 09:32:51.427 2262 TRACE oslo.messaging.rpc.dispatcher raise exc.from_response(resp, body_str) 2015-06-23 09:32:51.427 2262 TRACE oslo.messaging.rpc.dispatcher ImageNotAuthorized: Not authorized for image 66150e0a-1f90-4ca6-ba99-b46ba549c4a1. 2015-06-23 09:32:51.427 2262 TRACE oslo.messaging.rpc.dispatcher 2015-06-23 09:32:51.427 2262 ERROR oslo.messaging._drivers.common [-] Returning exception Not authorized for image 66150e0a-1f90-4ca6-ba99-b46ba549c4a1. to caller ... line 382, in update\n image_id, **image_meta)\n', ' File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 216, in call\n return getattr(client.images, method)(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/glanceclient/v1/images.py", line 331, in update\n \'PUT\', url, headers=hdrs, body=image_data)\n', ' File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 328, in raw_request\n return self._http_request(url, method, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 264, in _http_request\n raise exc.from_response(resp, body_str)\n', 'ImageNotAuthorized: Not authorized for image 66150e0a-1f90-4ca6-ba99-b46ba549c4a1.\n'] 2015-06-23 09:41:07.570 2262 WARNING nova.virt.libvirt.imagecache [-] Unknown base file: /var/lib/nova/instances/_base/e284db4838e5276fd580590cd3a5a0703ebe9f90 [...] How reproducible: Always Steps to Reproduce: 1. Try to create s snapshot using cli and it will finish successfully. 2. Try to reate the snapshot using Horizon and you will receive the erros above. Also, quoting user: When the problem occurs, the message in the horizon is "Image Pending Upload" in the guide "Instances", and in guide "Images" the message is "Queued". During this process, the file is created in the Compute in delta extension, and this file is created with the size of disk. After that, a new file is created and later the file is transfer for the Controller, in the Glance directory (/var/lib/glance/images). The problem occurs in this moment, when the file must be transfer into the Glance. The problem too occurs with admin user. Actual results: Snapsot is not created using Horizon. Expected results: Snapshot should be created without errors. Additional info:
What version are you using? What kind of backend? Which keystone authentification, e.g. pki token, uuid token?
Since you tried to set the target release, I'm assuming you're on OSP5 on RHEL 7? I set up an environment and couldn't find an issue with snapshots here. More questions: - how big is your snapshot (tmp file space sufficient)?
(In reply to Matthias Runge from comment #6) > Since you tried to set the target release, I'm assuming you're on OSP5 on > RHEL 7? > > I set up an environment and couldn't find an issue with snapshots here. > > More questions: > - how big is your snapshot (tmp file space sufficient)? Hi Matthias, I can ask customer but probably it isn't the problem. Unless you tell me that CLI stores the snapshot in a different temporary place than Horizon. According customer the problem is only related with Horizon: <snip> 1. Try to create s snapshot using cli and it will finish successfully. 2. Try to reate the snapshot using Horizon and you will receive the erros above. Also, quoting user: When the problem occurs, the message in the horizon is "Image Pending Upload" in the guide "Instances", and in guide "Images" the message is "Queued". During this process, the file is created in the Compute in delta extension, and this file is created with the size of disk. After that, a new file is created and later the file is transfer for the Controller, in the Glance directory (/var/lib/glance/images). The problem occurs in this moment, when the file must be transfer into the Glance. The problem too occurs with admin user. </snip>
Aligning target release here.