Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 911568

Summary:	[Tracking] Swift+Glance stops working after changing service password
Product:	Red Hat OpenStack	Reporter:	Flavio Percoco <fpercoco>
Component:	openstack-glance	Assignee:	Flavio Percoco <fpercoco>
Status:	CLOSED ERRATA	QA Contact:	Prasanth Anbalagan <panbalag>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	unspecified	CC:	eglynn, fpercoco, ncredi, nlevinki, panbalag, yeylon
Target Milestone:	z1	Keywords:	Triaged, ZStream
Target Release:	6.0 (Juno)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	openstack-glance-2014.2.1-3.el7ost	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-03-05 19:30:32 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Flavio Percoco 2013-02-15 11:34:27 UTC

Description of problem:

Swift stores full (username, password, host) locations URIs inside the database. This turns out to be:

1) A security issue since anyone with access to the database could have the store's username, password and host info.

2) An issue when any of those params change since they'll be changed in the config file but not in the existing images' records

Comment 4 Flavio Percoco 2013-11-22 13:18:53 UTC

There's a plan for doing a bigger change than the one proposed in the review is being planned to address this issue. From yesterday's meeting:

Log: http://eavesdrop.openstack.org/meetings/glance/2013/glance.2013-11-21-14.02.log.html

14:28:56 <flaper87> for example: https://review.openstack.org/#/c/34801/
14:29:23 <flaper87> that was opened on June 27th
14:29:29 <markwash> "Remove user and key from location in Swift"
14:29:36 <markwash> yeah that review is definitely on my mind
14:29:46 <markwash> there has been some recent discussion which I guess has been unfortunately kind of hidden
14:29:54 <markwash> which has kept me from pursuing that
14:30:00 <flaper87> oh
14:30:40 <markwash> rosmaita: would you say the conversations we've been having with smclaren about solving the credentials problem are relevant to that patch?
14:30:47 <markwash> and might change the direction we want to go?
14:31:00 <rosmaita> yes, i would hold off on that patch
14:31:34 <rosmaita> i can post Stuart's doc link if others are interested
14:31:45 <flaper87> rosmaita: yup, please.
14:31:49 <markwash> I might -1 it with a note just to pull it out of the stats
14:31:57 <rosmaita> don't mean to work hidden, but don't want to broadcast our prob

The proposal will address the issue in several stores besides swift. 

Moving to 5.0 since it won't be fixed for 4.0 and backporting the patch doesn't make sense in this case.

Comment 9 Prasanth Anbalagan 2015-02-17 15:01:14 UTC

I configured swift as the backend for glance. Then created 'image2' before changing admin credentials and 'image3' after changing the admin credentials. Now I'm not able to delete images that were created with old credentials. 
But deleting new images work. Please let me know if I can reopen the bug since
the fix may not be working here.

CONSOLE OUTPUT
==============

[root@lynx13 ~(keystone_admin)]# vi /etc/glance/glance-api.conf 
[root@lynx13 ~(keystone_admin)]# !138
service openstack-glance-api restart
Redirecting to /bin/systemctl restart  openstack-glance-api.service
[root@lynx13 ~(keystone_admin)]# !139
service openstack-glance-registry restart
Redirecting to /bin/systemctl restart  openstack-glance-registry.service
[root@lynx13 ~(keystone_admin)]# source keystonerc_admin 

[root@lynx13 ~(keystone_admin)]# glance image-create --name image3 --min-disk 1 --min-ram 768 --file  cirros.qcow2 --is-public True --disk-format=qcow2 --container-format bare --progress
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | d41d8cd98f00b204e9800998ecf8427e     |
| container_format | bare                                 |
| created_at       | 2015-02-17T14:43:11                  |
| deleted          | False                                |
| deleted_at       | None                                 |
| disk_format      | qcow2                                |
| id               | 97af10f6-72ea-476b-a52f-fecb4ba70a09 |
| is_public        | True                                 |
| min_disk         | 1                                    |
| min_ram          | 768                                  |
| name             | image3                               |
| owner            | 1343b6debee64b14966bf6742425aa21     |
| protected        | False                                |
| size             | 0                                    |
| status           | active                               |
| updated_at       | 2015-02-17T14:43:11                  |
| virtual_size     | None                                 |
+------------------+--------------------------------------+

[root@lynx13 ~(keystone_admin)]# glance image-list
+--------------------------------------+--------+-------------+------------------+----------+--------+
| ID                                   | Name   | Disk Format | Container Format | Size     | Status |
+--------------------------------------+--------+-------------+------------------+----------+--------+
| ce70e62d-14b6-43fa-b3b5-2afa6a9fa308 | cirros | qcow2       | bare             | 13200896 | active |
| 4016e592-d8c0-4f30-b898-e40c2d63f911 | image1 | qcow2       | bare             |          | active |
| d9dc2553-7d10-4b9b-a34d-3d60c4fcbe76 | image2 | qcow2       | bare             |          | active |
| 97af10f6-72ea-476b-a52f-fecb4ba70a09 | image3 | qcow2       | bare             |          | active |
+--------------------------------------+--------+-------------+------------------+----------+--------+


DELETING NEW IMAGES PASS 
=========================

[root@lynx13 ~(keystone_admin)]# glance image-delete 97af10f6-72ea-476b-a52f-fecb4ba70a09
[root@lynx13 ~(keystone_admin)]# 


DELETING AN OLD IMAGE FAILS
============================

[root@lynx13 ~(keystone_admin)]# glance image-delete 4016e592-d8c0-4f30-b898-e40c2d63f911
HTTPInternalServerError (HTTP 500): Unable to delete image 4016e592-d8c0-4f30-b898-e40c2d63f911

[root@lynx13 ~(keystone_admin)]# glance image-delete d9dc2553-7d10-4b9b-a34d-3d60c4fcbe76
HTTPInternalServerError (HTTP 500): Unable to delete image d9dc2553-7d10-4b9b-a34d-3d60c4fcbe76


SAMPLE LOGS FOR FAILURE (/var/log/glance-api.log)
===================================================

2015-02-17 16:49:37.055 77137 ERROR swiftclient [08ee6676-515e-4089-87cb-7b1d23e72723 828ad19213794cd290b541d736f8fb6f 1343b6debee64b14966bf6742425aa21 - - -] Unauthorized. Check username, password and tenant name/id.
2015-02-17 16:49:37.055 77137 TRACE swiftclient Traceback (most recent call last):
2015-02-17 16:49:37.055 77137 TRACE swiftclient   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1235, in _retry
2015-02-17 16:49:37.055 77137 TRACE swiftclient     self.url, self.token = self.get_auth()
2015-02-17 16:49:37.055 77137 TRACE swiftclient   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1209, in get_auth
2015-02-17 16:49:37.055 77137 TRACE swiftclient     insecure=self.insecure)
2015-02-17 16:49:37.055 77137 TRACE swiftclient   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 406, in get_auth
2015-02-17 16:49:37.055 77137 TRACE swiftclient     auth_version=auth_version)
2015-02-17 16:49:37.055 77137 TRACE swiftclient   File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 339, in get_auth_keystone
2015-02-17 16:49:37.055 77137 TRACE swiftclient     raise ClientException(msg)
2015-02-17 16:49:37.055 77137 TRACE swiftclient ClientException: Unauthorized. Check username, password and tenant name/id.
2015-02-17 16:49:37.055 77137 TRACE swiftclient 
2015-02-17 16:49:37.103 77137 INFO glance.wsgi.server [08ee6676-515e-4089-87cb-7b1d23e72723 828ad19213794cd290b541d736f8fb6f 1343b6debee64b14966bf6742425aa21 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 433, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/osprofiler/web.py", line 99, in __call__
    return request.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 748, in __call__
    return self._call_app(env, start_response)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token.py", line 684, in _call_app
    return self._app(env, _fake_start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 683, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 707, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 449, in wrapped
    return func(self, req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glance/api/v1/images.py", line 1073, in delete
    {'status': ori_status})
  File "/usr/lib/python2.7/site-packages/glance/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/glance/api/v1/images.py", line 1069, in delete
    upload_utils.initiate_deletion(req, loc_data, id)
  File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 46, in initiate_deletion
    id, location_data)
  File "/usr/lib/python2.7/site-packages/glance/common/store_utils.py", line 124, in delete_image_location_from_backend
    safe_delete_from_backend(context, image_id, location)
  File "/usr/lib/python2.7/site-packages/glance/common/store_utils.py", line 58, in safe_delete_from_backend
    ret = store_api.delete_from_backend(location['url'], context=context)
  File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 280, in delete_from_backend
    return store.delete(loc, context=context)
  File "/usr/lib/python2.7/site-packages/glance_store/_drivers/swift/store.py", line 593, in delete
    location.container, location.obj)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1333, in head_object
    return self._retry(None, head_object, container, obj)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1235, in _retry
    self.url, self.token = self.get_auth()
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1209, in get_auth
    insecure=self.insecure)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 406, in get_auth
    auth_version=auth_version)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 339, in get_auth_keystone
    raise ClientException(msg)
ClientException: Unauthorized. Check username, password and tenant name/id.
2015-02-17 16:49:37.103 77137 INFO glance.wsgi.server [08ee6676-515e-4089-87cb-7b1d23e72723 828ad19213794cd290b541d736f8fb6f 1343b6debee64b14966bf6742425aa21 - - -] 10.35.117.50 - - [17/Feb/2015 16:49:37] "DELETE /v1/images/d9dc2553-7d10-4b9b-a34d-3d60c4fcbe76 HTTP/1.1" 500 139 0.200249

Comment 10 Flavio Percoco 2015-02-17 23:59:31 UTC

@Prashanth

This bug is referring to something quite different from what you've tested. Your tests are based on keystone users credentials, whereas the bug refers to the *store* credentials.

In your test, you're correctly getting an error when you changed the tenant credentials. I say correctly because you're trying to access an image with a different user, AFAICT, which may not be the admin anymore.

What this bug fixes is an issue where the store credentials were kept in the database and therefore, whenever the store credentials were changed, it was not possible to access the image data anymore.

Comment 11 Prasanth Anbalagan 2015-02-18 13:51:30 UTC

Flavio,

Could you let me know how to change the store credentials, so that I can verify this bug?

As for my notes, I dont think it is a correct behavior. It is still the same "admin" user, but only a different password this time. The "admin" user still has admin privileges, so he should be able to delete any image. I will open a new bug for this.

Comment 12 Flavio Percoco 2015-02-20 08:13:45 UTC

@Prasanth,

May I see your config file? Specifically the swift section and the swift config file.

If you changed the password of one of the users listed in the swift config file, you'll need to update the file as well. Did you do this?

Comment 17 Prasanth Anbalagan 2015-02-20 15:20:20 UTC

Flavio,

What is the reference name mentioned in "[ref1]", '[ref2] etc. Is this something I need to configure in glance-api.conf?

Also, is the auth_address same as "swift_store_auth_address" in glance-api.conf?


Thanks
Prasanth

Comment 22 errata-xmlrpc 2015-03-05 19:30:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0644.html