Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1139959

Summary: [CISCO RHEL-OSP] image-create from --file fails with ceph backed glance
Product: Red Hat OpenStack Reporter: Britt Houser <bhouser>
Component: openstack-glanceAssignee: Flavio Percoco <fpercoco>
Status: CLOSED NOTABUG QA Contact: nlevinki <nlevinki>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 5.0 (RHEL 7)CC: eglynn, fpercoco, jdurgin, scohen, yeylon
Target Milestone: ---Keywords: ZStream
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-15 07:27:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1154145, 1154159, 1154162    
Attachments:
Description Flags
Glance Config File none

Description Britt Houser 2014-09-10 05:16:53 UTC
Description of problem:
After packstack installation completed successfully, I setup a ceph cluster, set selinux to permissive mode, and converted glance to use the ceph cluster.  Once this change was made, I was no longer able to create images using the --file paramater.  However if I specify a URL using --location, the image is created successfully.  Both options worked correctly before switching to ceph backend.

Version-Release number of selected component (if applicable):


How reproducible:
Everytime I switch to ceph backend

Steps to Reproduce:
1. Install OSP5 via packstack.
2. Setup ceph cluster
3. change selinux to permissive mode on controller node
4. change glance backed to use ceph cluster

Actual results:
500 error

Expected results:
image created successfully

Additional info:

Here are CLI logs and glance-api.logs from trying --file and then copying the into /var/www/html and trying --location.


##############################
#Create image from filesystem#
##############################
# wget http://download.cirros-cloud.net/0.3.2/cirros-0.3.2-x86_64-disk.img
# qemu-img convert -f qcow2 -O raw cirros-0.3.2-x86_64-disk.img cirros-0.3.2-x86_64-disk.raw
# glance image-create --name "cirros-0.3.2-x86_64" --is-public True \
  --disk-format raw --container-format=bare \
  --file cirros-0.3.2-x86_64-disk.raw
500 Internal Server Error
Failed to upload image a3b60572-b36d-4fba-91dc-408e83170a9f
    (HTTP 500)
#

################
#glance-api.log#
################
2014-09-10 00:40:26.780 29318 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.131.1
2014-09-10 00:40:26.872 29318 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.131.1
2014-09-10 00:40:26.974 29318 INFO oslo.messaging._drivers.impl_rabbit [0fb5ab21-1d74-4a74-b5bc-860b8e1829b8 f028f2d3eaf345638c9199449fd2fe9b 93cc3e39fd3c4fe882576856704e38b5 - - -] Connected to AMQP server on 192.168.131.1:5672
2014-09-10 00:40:27.033 29318 ERROR glance.api.v1.upload_utils [0fb5ab21-1d74-4a74-b5bc-860b8e1829b8 f028f2d3eaf345638c9199449fd2fe9b 93cc3e39fd3c4fe882576856704e38b5 - - -] Failed to upload image a3b60572-b36d-4fba-91dc-408e83170a9f
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance/api/v1/upload_utils.py", line 99, in upload_data_to_store
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils     store)
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance/store/__init__.py", line 380, in store_add_to_backend
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils     (location, size, checksum, metadata) = store.add(image_id, data, size)
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/glance/store/rbd.py", line 319, in add
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils     with rados.Rados(conffile=self.conf_file, rados_id=self.user) as conn:
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/rados.py", line 256, in __enter__
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils     self.connect()
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils   File "/usr/lib/python2.7/site-packages/rados.py", line 419, in connect
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils     raise make_ex(ret, "error calling connect")
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils ObjectNotFound: error calling connect
2014-09-10 00:40:27.033 29318 TRACE glance.api.v1.upload_utils
2014-09-10 00:40:27.139 29318 INFO glance.wsgi.server [0fb5ab21-1d74-4a74-b5bc-860b8e1829b8 f028f2d3eaf345638c9199449fd2fe9b 93cc3e39fd3c4fe882576856704e38b5 - - -] 192.168.131.1 - - [10/Sep/2014 00:40:27] "POST /v1/images HTTP/1.1" 500 293 0.369025


#######################
#Create image from URL#
#######################
# cp cirros-0.3.2-x86_64-disk.raw /var/www/html/
# glance image-create --name "cirros-0.3.2-x86_64" --is-public True \
>   --disk-format raw --container-format=bare \
>   --location http://cvf13-server-1/cirros-0.3.2-x86_64-disk.raw
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | None                                 |
| container_format | bare                                 |
| created_at       | 2014-09-10T04:46:41                  |
| deleted          | False                                |
| deleted_at       | None                                 |
| disk_format      | raw                                  |
| id               | 6cf0d2ce-8f2e-4437-b5a7-db5ab41eac04 |
| is_public        | True                                 |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | cirros-0.3.2-x86_64                  |
| owner            | 93cc3e39fd3c4fe882576856704e38b5     |
| protected        | False                                |
| size             | 41126400                             |
| status           | active                               |
| updated_at       | 2014-09-10T04:46:41                  |
| virtual_size     | None                                 |
+------------------+--------------------------------------+

################
#glance-api.log#
################
2014-09-10 00:46:40.914 29305 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.131.1
2014-09-10 00:46:41.008 29305 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 192.168.131.1
2014-09-10 00:46:41.118 29305 INFO oslo.messaging._drivers.impl_rabbit [5f029a99-0dfe-4d06-b984-5744def81705 f028f2d3eaf345638c9199449fd2fe9b 93cc3e39fd3c4fe882576856704e38b5 - - -] Connected to AMQP server on 192.168.131.1:5672
2014-09-10 00:46:41.183 29305 INFO glance.wsgi.server [5f029a99-0dfe-4d06-b984-5744def81705 f028f2d3eaf345638c9199449fd2fe9b 93cc3e39fd3c4fe882576856704e38b5 - - -] 192.168.131.1 - - [10/Sep/2014 00:46:41] "POST /v1/images HTTP/1.1" 201 723 0.279931

Comment 2 Flavio Percoco 2014-09-10 13:52:30 UTC
Hi Britt,

Based on the traceback, it looks like Glance is not able to connect to ceph correctly. Is it possible for you to share your glance config file? I'm interested in how the rbd store was configured.

Comment 3 Britt Houser 2014-09-10 14:04:59 UTC
Created attachment 936178 [details]
Glance Config File

Comment 4 Flavio Percoco 2014-09-11 14:06:31 UTC
Britt,

Glance's config looks ok. Based on the error message returned by rados, there's probably something wrong in Ceph's configs/setup. Is it possible to have your ceph config file as well?

@Josh, could you please help debugging this issue?

Comment 5 Josh Durgin 2014-09-11 18:39:57 UTC
That is a failure to connect to the ceph cluster, most likely due to missing/unreadable auth info. Are /etc/ceph/ceph.client.glance.keyring and /etc/ceph/ceph.conf on the node running glance-api, and are they readable by the user glance-api is running as?

Comment 6 Britt Houser 2014-09-12 12:09:56 UTC
Yes and Yes.  I figurd out the problem.  When I created the auth user I issued the command with line wraps like this:

--
ceph auth get-or-create client.glance mon 'allow r' \
  osd 'allow class-read object_prefix rbd_children, \ 
  allow rwx pool=images' -o /etc/ceph/client.glance.keyring
--

the line wrap in the middle of the OSD parameter didn't get unwrapped correctly and I ended up with:

--
client.glance
	key: AQDebhJUyI0JNxAA8IeaonS/g/WvBM1oQhp/0A==
	caps: [mon] allow r
	caps: [osd] allow class-read object_prefix rbd_children, \
  allow rwx pool=images
--

When it should have looked like:

--
client.glance
	key: AQDO1BJUmNAgNRAAV1Nve2AruiCvGg7EITA60w==
	caps: [mon] allow r
	caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=images
--

So it looks like the issue as I have reported it was user error.  Only question remaining in my mind is why did --location work?  Was glance storing files from --location in a different backend?

Comment 7 Josh Durgin 2014-09-12 18:18:38 UTC
IIRC with --location glance does not check that the location is actually accessible, it just checks that it's a valid url for one of its stores, so
it would only have a problem once you tried to download the image.

Comment 8 Britt Houser 2014-09-12 18:20:44 UTC
Ok that is good to know.  When we specified --location, however, the image successfully made it into glance and we booted VMs from it, even though ceph authentication was misconfigured...

Comment 9 Josh Durgin 2014-09-12 18:26:17 UTC
If you were using a version with rbd cloning enabled for nova, or creating a volume from an image onto ceph, glance would never actually access ceph.

Comment 10 Britt Houser 2014-09-12 19:15:10 UTC
Ok that is good to know.   This was for initial image creation.  When Ceph was mis-configured, then creating a image from --file failed, whereas from --location worked.  And the image created from --location could boot up just fine.

Comment 11 Flavio Percoco 2014-09-15 07:27:54 UTC
FWIW, the `--location` behavior described by Josh is correct.

I'll close this report! Thanks for bringing this up, Britt.