Bug 888545 - glance image-create premature terminate with checksum leaks data on the glance image store
Summary: glance image-create premature terminate with checksum leaks data on the glanc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-glance
Version: 2.0 (Folsom)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: snapshot3
: 2.1
Assignee: Eoghan Glynn
QA Contact: Attila Fazekas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-12-18 21:05 UTC by John Bresnahan
Modified: 2013-09-30 02:04 UTC (History)
2 users (show)

Fixed In Version: openstack-glance-2012.2.3-1.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-05 18:30:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1104924 0 None None None Never
Launchpad 1122299 0 None None None Never
OpenStack gerrit 20481 0 None None None Never
Red Hat Product Errata RHBA-2013:0593 0 normal SHIPPED_LIVE Red Hat OpenStack 2.0 (Folsom) Preview bug fix and enhancement update 2013-03-05 23:28:55 UTC

Description John Bresnahan 2012-12-18 21:05:02 UTC
Description of problem:

A data storage leak occurs in glance when a checksum is provided while uploading an image to glance and the client process uploading the image prematurely dies.  This results in a data on the file system, but no reference to that data in the registry.

Steps to Reproduce:
1. begin an upload of an image with the md5sum
2. terminate the client immediately
3. run 'glance index' and note that a new image has not appeared
4. run ls on the file system which is backing glance and notice that orphaned data is there

Actual results:


$ ls -l /gluster/glance/images/
total 0

$ glance index
ID                                   Name                           Disk Format          Container Format     Size          
------------------------------------ ------------------------------ -------------------- -------------------- --------------

$ glance  image-create  --name testit --container-format bare --disk-format qcow2 --is-public true --file f16-x86_64-openstack-sda.qcow2 --size 213581824 --checksum 755122332caeb9f661d5c978adb8b45f
^CTraceback (most recent call last):
  File "/usr/bin/glance", line 9, in <module>
    load_entry_point('python-glanceclient==0.5.1', 'console_scripts', 'glance')()
  File "/usr/lib/python2.6/site-packages/glanceclient/shell.py", line 432, in main
    OpenStackImagesShell().main(sys.argv[1:])
  File "/usr/lib/python2.6/site-packages/glanceclient/shell.py", line 403, in main
    args.func(client, args)
  File "/usr/lib/python2.6/site-packages/glanceclient/v1/shell.py", line 160, in do_image_create
    image = gc.images.create(**fields)
  File "/usr/lib/python2.6/site-packages/glanceclient/v1/images.py", line 212, in create
    'POST', '/v1/images', headers=hdrs, body=image_data)
  File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 192, in raw_request
    return self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 137, in _http_request
    conn.request(method, url, **kwargs)
  File "/usr/lib64/python2.6/httplib.py", line 914, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.6/httplib.py", line 954, in _send_request
    self.send(body)
  File "/usr/lib64/python2.6/httplib.py", line 756, in send
    self.sock.sendall(data)
  File "<string>", line 1, in sendall
KeyboardInterrupt

$ glance index
ID                                   Name                           Disk Format          Container Format     Size          
------------------------------------ ------------------------------ -------------------- -------------------- --------------

$ ls -l /gluster/glance/images/
total 103208
-rw-r-----. 1 glance glance 105684992 Dec 18 11:02 8d64fb62-67c5-4cbd-b109-123656d70903



Expected results:

The service should either report the image as existing but corrupt, giving the user a chance to clean it.  Or it should automatically deleted when it is detected as a failed upload.

Comment 2 Eoghan Glynn 2013-01-25 10:02:26 UTC
Of the supported backend store types, this issue only impacts on the filesystem store.

(i.e. in the case of swift and s3, I've confirmed that partial uploads are properly cleaned up and do not leave behind partial image fragments).

Comment 3 Eoghan Glynn 2013-01-25 15:26:36 UTC
Fix proposed to master upstream:

  https://review.openstack.org/20472

Comment 4 Eoghan Glynn 2013-01-25 15:27:06 UTC
Fix proposed to stable/folsom upstream:

  https://review.openstack.org/20481

Comment 10 Eoghan Glynn 2013-02-11 16:46:40 UTC
So it looks like the broken upload connection is not reported as an exception from read on RHEL (as was the case when I tested it in devstack on Ubuntu).

However the bug refers specifically to the case where the checksum (and size) is
explicitly specified for the new image. So even in the absence of an exceptional return from the read, we can still ensure deletion on a mismatched checksum or size.

Patch pending shortly ...

Comment 11 Eoghan Glynn 2013-02-11 18:56:31 UTC
Further fix proposed upstream on master:

  https://review.openstack.org/21675

Comment 12 Eoghan Glynn 2013-02-11 18:56:58 UTC
Further fix proposed upstream on stable/folsom:

  https://review.openstack.org/21678

Comment 13 Eoghan Glynn 2013-02-11 19:14:12 UTC
Further fix proposed internally:

  https://code.engineering.redhat.com/gerrit/3103

Comment 14 Eoghan Glynn 2013-02-11 20:45:46 UTC
Merged internally.

Comment 16 Attila Fazekas 2013-02-20 18:26:53 UTC
Working :)

Comment 18 errata-xmlrpc 2013-03-05 18:30:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0593.html


Note You need to log in before you can comment on or make changes to this bug.