Bug 1314264

Summary: glance image download fails with wrong checksum
Product: Red Hat OpenStack Reporter: Martin Schuppert <mschuppe>
Component: python-glanceclientAssignee: Flavio Percoco <fpercoco>
Status: CLOSED NOTABUG QA Contact: nlevinki <nlevinki>
Severity: high Docs Contact:
Priority: high    
Version: 6.0 (Juno)CC: akaris, eglynn, fpercoco, jruzicka, mschuppe, rhosp-bugs-internal, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 6.0 (Juno)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-03-03 11:31:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Martin Schuppert 2016-03-03 10:08:56 UTC 
Description of problem:

When perform glance image download, or start instance from this image, download fails with corrupted checksum. ceph is being used as backend:

glance --debug --os-image-url="http://192.168.0.143:9292" image-download cbbb5670-4c2b-43c9-8c33-664d3df8993b --file rhel72-glb-cloud_x86_64.qcow2.glance-direct-dl

...

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 602, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v1/shell.py", line 152, in do_image_download
    utils.save_image(body, args.file)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 236, in save_image
    for chunk in data:
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 374, in integrity_iter
    (md5sum, checksum))
IOError: [Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b
[Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b

Version-Release number of selected component (if applicable):
# grep glance installed-rpms 
openstack-glance-2014.2.3-3.el7ost.noarch                   Fri Oct 23 08:15:43 2015    1445580943      
python-glance-2014.2.3-3.el7ost.noarch                      Fri Oct 23 08:14:29 2015    1445580869      
python-glance-store-0.1.10-6.el7ost.noarch                  Wed Jan 27 16:36:08 2016    1453908968      
python-glanceclient-0.14.2-3.el7ost.noarch                  Wed Jan 27 16:35:36 2016    1453908936

How reproducible:
always

Steps to Reproduce:
1. glance --debug --os-image-url="http://192.168.0.143:9292" image-download cbbb5670-4c2b-43c9-8c33-664d3df8993b --file rhel72-glb-cloud_x86_64.qcow2.glance-direct-dl

Additional info:

Spawn instance works when use raw image as then creating disk is being handled on ceph side.

To isolate the issue following tests have been performed, see results bellow:
- download/export image from ceph using rbd -> checksum ok
- remove haproxy and load the image direct from glance using --os-image-url= -> same issue, checksum wrong

Details:
*image checksum = d3fb7e67a5c738bd3c97228084483a5b :
[root@cci01-oc03 glance(openstack_admin)]# glance image-show cbbb5670-4c2b-43c9-8c33-664d3df8993b 
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
---> | checksum         | d3fb7e67a5c738bd3c97228084483a5b     |
| container_format | bare                                 |
| created_at       | 2016-02-01T11:15:58                  |
| deleted          | False                                |
| disk_format      | qcow2                                |
| id               | cbbb5670-4c2b-43c9-8c33-664d3df8993b |
| is_public        | True                                 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | rhel72-glb-cloud_x86_64.qcow2        |
| owner            | 787e10cc887141dd9008424d700605c8     |
| protected        | False                                |
| size             | 2163802112                           |
| status           | active                               |
| updated_at       | 2016-02-01T11:16:50                  |
+------------------+--------------------------------------+

* Direct download from glance, no haproxy:
[root@cci01-oc03 test(openstack_admin)]# glance --debug --os-image-url="http://192.168.0.143:9292" image-download cbbb5670-4c2b-43c9-8c33-6
64d3df8993b --file rhel72-glb-cloud_x86_64.qcow2.glance-direct-dl 
curl -i -X HEAD -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X
-Auth-Token: {SHA1}30d0139604ae60e642c0255530cea7c40d12d5ad' -H 'Content-Type: application/octet-stream' http://192.168.0.143:9292/v1/image
s/cbbb5670-4c2b-43c9-8c33-664d3df8993b

HTTP/1.1 200 OK
content-length: 0
x-image-meta-id: cbbb5670-4c2b-43c9-8c33-664d3df8993b
date: Wed, 02 Mar 2016 15:31:59 GMT
x-image-meta-deleted: False
x-image-meta-container_format: bare
connection: keep-alive
---> x-image-meta-checksum: d3fb7e67a5c738bd3c97228084483a5b
x-image-meta-protected: False
x-image-meta-min_disk: 0
x-image-meta-created_at: 2016-02-01T11:15:58
x-image-meta-size: 2163802112
x-image-meta-status: active
etag: d3fb7e67a5c738bd3c97228084483a5b
x-image-meta-is_public: True
x-image-meta-min_ram: 0
x-image-meta-owner: 787e10cc887141dd9008424d700605c8
x-image-meta-updated_at: 2016-02-01T11:16:50
content-type: text/html; charset=UTF-8
x-openstack-request-id: req-6174d705-e6e6-400e-8bc2-6ae82cb98a0d
x-image-meta-disk_format: qcow2
x-image-meta-name: rhel72-glb-cloud_x86_64.qcow2

curl -i -X HEAD -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X
-Auth-Token: {SHA1}30d0139604ae60e642c0255530cea7c40d12d5ad' -H 'Content-Type: application/octet-stream' http://192.168.0.143:9292/v1/image
s/cbbb5670-4c2b-43c9-8c33-664d3df8993b

HTTP/1.1 200 OK
content-length: 0
x-image-meta-id: cbbb5670-4c2b-43c9-8c33-664d3df8993b
date: Wed, 02 Mar 2016 15:31:59 GMT

x-image-meta-deleted: False
x-image-meta-container_format: bare
connection: keep-alive
--> x-image-meta-checksum: d3fb7e67a5c738bd3c97228084483a5b
x-image-meta-protected: False
x-image-meta-min_disk: 0
x-image-meta-created_at: 2016-02-01T11:15:58
x-image-meta-size: 2163802112
x-image-meta-status: active
etag: d3fb7e67a5c738bd3c97228084483a5b
x-image-meta-is_public: True
x-image-meta-min_ram: 0
x-image-meta-owner: 787e10cc887141dd9008424d700605c8
x-image-meta-updated_at: 2016-02-01T11:16:50
content-type: text/html; charset=UTF-8
x-openstack-request-id: req-9b24d23b-2126-47bb-8794-b15692e72907
x-image-meta-disk_format: qcow2
x-image-meta-name: rhel72-glb-cloud_x86_64.qcow2

curl -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}30d0139604ae60e642c0255530cea7c40d12d5ad' -H 'Content-Type: application/octet-stream' http://192.168.0.143:9292/v1/images/cbbb5670-4c2b-43c9-8c33-664d3df8993b

HTTP/1.1 200 OK
content-length: 2163802112
x-image-meta-id: cbbb5670-4c2b-43c9-8c33-664d3df8993b
date: Wed, 02 Mar 2016 15:31:59 GMT
x-image-meta-deleted: False
x-image-meta-container_format: bare
connection: keep-alive
--> x-image-meta-checksum: d3fb7e67a5c738bd3c97228084483a5b
x-image-meta-protected: False
x-image-meta-min_disk: 0
x-image-meta-created_at: 2016-02-01T11:15:58
x-image-meta-size: 2163802112
x-image-meta-status: active
etag: d3fb7e67a5c738bd3c97228084483a5b
x-image-meta-is_public: True
x-image-meta-min_ram: 0
x-image-meta-owner: 787e10cc887141dd9008424d700605c8
x-image-meta-updated_at: 2016-02-01T11:16:50
content-type: application/octet-stream
x-openstack-request-id: req-3f9cfa2b-32bd-4627-9325-094c1cbf2adb
x-image-meta-disk_format: qcow2
x-image-meta-name: rhel72-glb-cloud_x86_64.qcow2

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 602, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v1/shell.py", line 152, in do_image_download
    utils.save_image(body, args.file)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 236, in save_image
    for chunk in data:
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 374, in integrity_iter
    (md5sum, checksum))
IOError: [Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b
[Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b

=> wrong checksum

* From previous tests we see the orig image and downloaded has same size :
[root@cci01-oc01 glancetest(admin_eng)]# ll
total 9915008
-rw-r--r--. 1 root root 2163802112 Feb 23 15:53 rhel72-glb-clou
-rw-r--r--. 1 root root 2163802112 Feb 23 14:29 rhel72-glb-cloud_x86_64.qcow2.out2
-rw-r--r--. 1 root root 2163802112 Feb 23 14:48 rhel72-glb-cloud_x86_64.qcow2.out2.1
-rw-r--r--. 1 root root 2163802112 Feb 23 15:49 rhel72-glb-cloud_x86_64.qcow2-test-download.1

[root@cci01-oc01 glancetest(admin_eng)]# file rhel72-glb-cloud_x86_64.qcow2-test-download.1 
rhel72-glb-cloud_x86_64.qcow2-test-download.1: QEMU QCOW Image (v2), 53687091200 bytes

[root@cci01-oc01 glancetest(admin_eng)]# file rhel72-glb-clou 
rhel72-glb-clou: QEMU QCOW Image (v2), 53687091200 bytes

* image has been created with command like this (from another run)
# glance image-create --container-format=bare --disk-format=qcow2 --is-public=true --name rhel72_glb_1 < rhel72_x86_64.qcow2


[root@cci01-oc03 glance(openstack_admin)]# rbd info images/cbbb5670-4c2b-43c9-8c33-664d3df8993b 
rbd image 'cbbb5670-4c2b-43c9-8c33-664d3df8993b':
        size 2063 MB in 258 objects
        order 23 (8192 kB objects)
        block_name_prefix: rbd_data.c1ccce7c2260f
        format: 2
        features: layering
        flags: 

* export from ceph using rbd:
[root@cci01-oc03 test(openstack_admin)]# rbd export images/cbbb5670-4c2b-43c9-8c33-664d3df8993b ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out

* get md5sum after rbd export:
[root@cci01-oc03 test(openstack_admin)]# md5sum ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out > ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out.md5 

[root@cci01-oc03 test]# cat ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out.md5 
d3fb7e67a5c738bd3c97228084483a5b  ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out

=> correct checksum
Comment 1 Martin Schuppert 2016-03-03 11:31:51 UTC 
Closing this. It turns out that tearing on the images pool was
still active. This resulted in a corrupt download.
Comment 2 bigswitch 2017-07-28 05:25:21 UTC 
how to fix this , tearing on the image pool? , we are experecing same issue
Comment 3 Martin Schuppert 2017-08-21 13:59:39 UTC 
(In reply to bigswitch from comment #2)
> how to fix this , tearing on the image pool? , we are experecing same issue

Sorry that was a typo, it was tiering. Please check [1] on how to remove a cache tier. This is a manual step, so if you have not enabled it the issue might be different.

[1] http://docs.ceph.com/docs/master/rados/operations/cache-tiering/