Description of problem: When perform glance image download, or start instance from this image, download fails with corrupted checksum. ceph is being used as backend: glance --debug --os-image-url="http://192.168.0.143:9292" image-download cbbb5670-4c2b-43c9-8c33-664d3df8993b --file rhel72-glb-cloud_x86_64.qcow2.glance-direct-dl ... Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 602, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/v1/shell.py", line 152, in do_image_download utils.save_image(body, args.file) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 236, in save_image for chunk in data: File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 374, in integrity_iter (md5sum, checksum)) IOError: [Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b [Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b Version-Release number of selected component (if applicable): # grep glance installed-rpms openstack-glance-2014.2.3-3.el7ost.noarch Fri Oct 23 08:15:43 2015 1445580943 python-glance-2014.2.3-3.el7ost.noarch Fri Oct 23 08:14:29 2015 1445580869 python-glance-store-0.1.10-6.el7ost.noarch Wed Jan 27 16:36:08 2016 1453908968 python-glanceclient-0.14.2-3.el7ost.noarch Wed Jan 27 16:35:36 2016 1453908936 How reproducible: always Steps to Reproduce: 1. glance --debug --os-image-url="http://192.168.0.143:9292" image-download cbbb5670-4c2b-43c9-8c33-664d3df8993b --file rhel72-glb-cloud_x86_64.qcow2.glance-direct-dl Additional info: Spawn instance works when use raw image as then creating disk is being handled on ceph side. To isolate the issue following tests have been performed, see results bellow: - download/export image from ceph using rbd -> checksum ok - remove haproxy and load the image direct from glance using --os-image-url= -> same issue, checksum wrong Details: *image checksum = d3fb7e67a5c738bd3c97228084483a5b : [root@cci01-oc03 glance(openstack_admin)]# glance image-show cbbb5670-4c2b-43c9-8c33-664d3df8993b +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ ---> | checksum | d3fb7e67a5c738bd3c97228084483a5b | | container_format | bare | | created_at | 2016-02-01T11:15:58 | | deleted | False | | disk_format | qcow2 | | id | cbbb5670-4c2b-43c9-8c33-664d3df8993b | | is_public | True | | min_disk | 0 | | min_ram | 0 | | name | rhel72-glb-cloud_x86_64.qcow2 | | owner | 787e10cc887141dd9008424d700605c8 | | protected | False | | size | 2163802112 | | status | active | | updated_at | 2016-02-01T11:16:50 | +------------------+--------------------------------------+ * Direct download from glance, no haproxy: [root@cci01-oc03 test(openstack_admin)]# glance --debug --os-image-url="http://192.168.0.143:9292" image-download cbbb5670-4c2b-43c9-8c33-6 64d3df8993b --file rhel72-glb-cloud_x86_64.qcow2.glance-direct-dl curl -i -X HEAD -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X -Auth-Token: {SHA1}30d0139604ae60e642c0255530cea7c40d12d5ad' -H 'Content-Type: application/octet-stream' http://192.168.0.143:9292/v1/image s/cbbb5670-4c2b-43c9-8c33-664d3df8993b HTTP/1.1 200 OK content-length: 0 x-image-meta-id: cbbb5670-4c2b-43c9-8c33-664d3df8993b date: Wed, 02 Mar 2016 15:31:59 GMT x-image-meta-deleted: False x-image-meta-container_format: bare connection: keep-alive ---> x-image-meta-checksum: d3fb7e67a5c738bd3c97228084483a5b x-image-meta-protected: False x-image-meta-min_disk: 0 x-image-meta-created_at: 2016-02-01T11:15:58 x-image-meta-size: 2163802112 x-image-meta-status: active etag: d3fb7e67a5c738bd3c97228084483a5b x-image-meta-is_public: True x-image-meta-min_ram: 0 x-image-meta-owner: 787e10cc887141dd9008424d700605c8 x-image-meta-updated_at: 2016-02-01T11:16:50 content-type: text/html; charset=UTF-8 x-openstack-request-id: req-6174d705-e6e6-400e-8bc2-6ae82cb98a0d x-image-meta-disk_format: qcow2 x-image-meta-name: rhel72-glb-cloud_x86_64.qcow2 curl -i -X HEAD -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X -Auth-Token: {SHA1}30d0139604ae60e642c0255530cea7c40d12d5ad' -H 'Content-Type: application/octet-stream' http://192.168.0.143:9292/v1/image s/cbbb5670-4c2b-43c9-8c33-664d3df8993b HTTP/1.1 200 OK content-length: 0 x-image-meta-id: cbbb5670-4c2b-43c9-8c33-664d3df8993b date: Wed, 02 Mar 2016 15:31:59 GMT x-image-meta-deleted: False x-image-meta-container_format: bare connection: keep-alive --> x-image-meta-checksum: d3fb7e67a5c738bd3c97228084483a5b x-image-meta-protected: False x-image-meta-min_disk: 0 x-image-meta-created_at: 2016-02-01T11:15:58 x-image-meta-size: 2163802112 x-image-meta-status: active etag: d3fb7e67a5c738bd3c97228084483a5b x-image-meta-is_public: True x-image-meta-min_ram: 0 x-image-meta-owner: 787e10cc887141dd9008424d700605c8 x-image-meta-updated_at: 2016-02-01T11:16:50 content-type: text/html; charset=UTF-8 x-openstack-request-id: req-9b24d23b-2126-47bb-8794-b15692e72907 x-image-meta-disk_format: qcow2 x-image-meta-name: rhel72-glb-cloud_x86_64.qcow2 curl -i -X GET -H 'Accept-Encoding: gzip, deflate' -H 'Accept: */*' -H 'User-Agent: python-glanceclient' -H 'Connection: keep-alive' -H 'X-Auth-Token: {SHA1}30d0139604ae60e642c0255530cea7c40d12d5ad' -H 'Content-Type: application/octet-stream' http://192.168.0.143:9292/v1/images/cbbb5670-4c2b-43c9-8c33-664d3df8993b HTTP/1.1 200 OK content-length: 2163802112 x-image-meta-id: cbbb5670-4c2b-43c9-8c33-664d3df8993b date: Wed, 02 Mar 2016 15:31:59 GMT x-image-meta-deleted: False x-image-meta-container_format: bare connection: keep-alive --> x-image-meta-checksum: d3fb7e67a5c738bd3c97228084483a5b x-image-meta-protected: False x-image-meta-min_disk: 0 x-image-meta-created_at: 2016-02-01T11:15:58 x-image-meta-size: 2163802112 x-image-meta-status: active etag: d3fb7e67a5c738bd3c97228084483a5b x-image-meta-is_public: True x-image-meta-min_ram: 0 x-image-meta-owner: 787e10cc887141dd9008424d700605c8 x-image-meta-updated_at: 2016-02-01T11:16:50 content-type: application/octet-stream x-openstack-request-id: req-3f9cfa2b-32bd-4627-9325-094c1cbf2adb x-image-meta-disk_format: qcow2 x-image-meta-name: rhel72-glb-cloud_x86_64.qcow2 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 602, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/v1/shell.py", line 152, in do_image_download utils.save_image(body, args.file) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 236, in save_image for chunk in data: File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 374, in integrity_iter (md5sum, checksum)) IOError: [Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b [Errno 32] Corrupt image download. Checksum was 229ff1e86f5b4171f9917dceb3194e95 expected d3fb7e67a5c738bd3c97228084483a5b => wrong checksum * From previous tests we see the orig image and downloaded has same size : [root@cci01-oc01 glancetest(admin_eng)]# ll total 9915008 -rw-r--r--. 1 root root 2163802112 Feb 23 15:53 rhel72-glb-clou -rw-r--r--. 1 root root 2163802112 Feb 23 14:29 rhel72-glb-cloud_x86_64.qcow2.out2 -rw-r--r--. 1 root root 2163802112 Feb 23 14:48 rhel72-glb-cloud_x86_64.qcow2.out2.1 -rw-r--r--. 1 root root 2163802112 Feb 23 15:49 rhel72-glb-cloud_x86_64.qcow2-test-download.1 [root@cci01-oc01 glancetest(admin_eng)]# file rhel72-glb-cloud_x86_64.qcow2-test-download.1 rhel72-glb-cloud_x86_64.qcow2-test-download.1: QEMU QCOW Image (v2), 53687091200 bytes [root@cci01-oc01 glancetest(admin_eng)]# file rhel72-glb-clou rhel72-glb-clou: QEMU QCOW Image (v2), 53687091200 bytes * image has been created with command like this (from another run) # glance image-create --container-format=bare --disk-format=qcow2 --is-public=true --name rhel72_glb_1 < rhel72_x86_64.qcow2 [root@cci01-oc03 glance(openstack_admin)]# rbd info images/cbbb5670-4c2b-43c9-8c33-664d3df8993b rbd image 'cbbb5670-4c2b-43c9-8c33-664d3df8993b': size 2063 MB in 258 objects order 23 (8192 kB objects) block_name_prefix: rbd_data.c1ccce7c2260f format: 2 features: layering flags: * export from ceph using rbd: [root@cci01-oc03 test(openstack_admin)]# rbd export images/cbbb5670-4c2b-43c9-8c33-664d3df8993b ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out * get md5sum after rbd export: [root@cci01-oc03 test(openstack_admin)]# md5sum ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out > ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out.md5 [root@cci01-oc03 test]# cat ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out.md5 d3fb7e67a5c738bd3c97228084483a5b ceph-export-cbbb5670-4c2b-43c9-8c33-664d3df8993b.out => correct checksum
Closing this. It turns out that tearing on the images pool was still active. This resulted in a corrupt download.
how to fix this , tearing on the image pool? , we are experecing same issue
(In reply to bigswitch from comment #2) > how to fix this , tearing on the image pool? , we are experecing same issue Sorry that was a typo, it was tiering. Please check [1] on how to remove a cache tier. This is a manual step, so if you have not enabled it the issue might be different. [1] http://docs.ceph.com/docs/master/rados/operations/cache-tiering/