Bug 1570026

Summary:

Error connecting to ceph cluster (BackendException, rados.Rados.connect)

Product:

Red Hat OpenStack

Reporter:

Filip Hubík <fhubik>

Component:

openstack-glance

Assignee:

Cyril Roelandt <cyril>

Status:

CLOSED DUPLICATE

QA Contact:

Mike Abrams <mabrams>

Severity:

unspecified

Docs Contact:

Kim Nylander <knylande>

Priority:

unspecified

Version:

10.0 (Newton)

CC:

eglynn, fpercoco, srevivo

Target Milestone:

---

Keywords:

Automation, AutomationBlocker

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2018-04-20 13:12:34 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
partial logfile glance/api.log	none

Description Filip Hubík 2018-04-20 13:04:14 UTC

Created attachment 1424504 [details]
partial logfile glance/api.log

Description of problem:

Glance can not upload image to registry, glance.log reports following exception:

...
DEBUG glance_store.capabilities [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python2.7/site-packages/glance_store/capabilities.py:97
DEBUG glance_store.driver [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Late loading location class glance_store._drivers.rbd.StoreLocation get_store_location_class /usr/lib/python2.7/site-packages/glance_store/driver.py:89
DEBUG glance_store.location [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Registering scheme rbd with {'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store': <glance_store._drivers.rbd.Store object at 0x7fad5a0c0290>, 'store_entry': 'glance.store.rbd.Store'} register_scheme_map /usr/lib/python2.7/site-packages/glance_store/location.py:88
ERROR glance_store._drivers.rbd [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Error connecting to ceph cluster.
ERROR glance_store._drivers.rbd Traceback (most recent call last):
ERROR glance_store._drivers.rbd   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 273, in get_connection
ERROR glance_store._drivers.rbd     client.connect(timeout=self.connect_timeout)
ERROR glance_store._drivers.rbd   File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969)
ERROR glance_store._drivers.rbd ObjectNotFound: error connecting to the cluster
ERROR glance_store._drivers.rbd 
DEBUG oslo_messaging._drivers.amqpdriver [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] CAST unique_id: 91a51cc277804e4fbf0c8655 f854e11d NOTIFY exchange 'glance' topic 'notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:552
ERROR glance.api.v2.image_data [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Failed to upload image data due to internal error
ERROR glance.api.v2.image_data Traceback (most recent call last):
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
ERROR glance.api.v2.image_data     image.set_data(data, size)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
ERROR glance.api.v2.image_data     self.base.set_data(data, size)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
ERROR glance.api.v2.image_data     _send_notification(notify_error, 'image.upload', msg)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR glance.api.v2.image_data     self.force_reraise()
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR glance.api.v2.image_data     six.reraise(self.type_, self.value, self.tb)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
ERROR glance.api.v2.image_data     self.repo.set_data(data, size)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 187, in set_data
ERROR glance.api.v2.image_data     return self.image.set_data(*args, **kwargs)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
ERROR glance.api.v2.image_data     self.image.set_data(data, size=size)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
ERROR glance.api.v2.image_data     verifier=verifier)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 452, in add_to_backend
ERROR glance.api.v2.image_data     verifier)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 425, in store_add_to_backend
ERROR glance.api.v2.image_data     verifier=verifier)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
ERROR glance.api.v2.image_data     return store_op_fun(store, *args, **kwargs)
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 447, in add
ERROR glance.api.v2.image_data     rados_id=self.user) as conn:
ERROR glance.api.v2.image_data   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
ERROR glance.api.v2.image_data     return self.gen.next()
ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 277, in get_connection
ERROR glance.api.v2.image_data     raise exceptions.BackendException()
ERROR glance.api.v2.image_data BackendException
ERROR glance.api.v2.image_data
...

(full exception attached)

Version-Release number of selected component (if applicable):
Started since puddle 2018-04-19.1, OSP10.

How reproducible:
always

Steps to Reproduce:
1. Deploy OSP10 topology virthost-1cont_1comp_1ceph on RHEL 7.5
2. Tempest won't run since it can not even upload cirros image to glance

Additional info:
*ceph's health and status reports OK on ceph node
*this sounds like issue with glance <-> rbd comminication
*manual attempt to upload to glance with debug enabled on client side:
PUT call to glance-api for http://10.0.0.110:9292/v2/images/1d85055a-5a8e-45ce-a577-f304c66b7de3/file used request id req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea
DEBUG:glanceclient.common.http:PUT call to glance-api for http://10.0.0.110:9292/v2/images/1d85055a-5a8e-45ce-a577-f304c66b7de3/file used request id req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea
Request returned failure status 500.
DEBUG:glanceclient.common.http:Request returned failure status 500.
+------------------+--------------------------------------+
| Property         | Value                                |
+------------------+--------------------------------------+
| checksum         | None                                 |
| container_format | bare                                 |
| created_at       | 2018-04-20T12:45:44Z                 |
| disk_format      | qcow2                                |
| id               | 1d85055a-5a8e-45ce-a577-f304c66b7de3 |
| locations        | []                                   |
| min_disk         | 0                                    |
| min_ram          | 0                                    |
| name             | cirros                               |
| owner            | c1a39a4ee28c4f3184fd6384b09d6ee3     |
| protected        | False                                |
| size             | None                                 |
| status           | queued                               |
| tags             | []                                   |
| updated_at       | 2018-04-20T12:45:44Z                 |
| virtual_size     | None                                 |
| visibility       | private                              |
+------------------+--------------------------------------+
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 668, in main
    OpenStackImagesShell().main(argv)
  File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 599, in main
    args.func(client, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 98, in func_wrapper
    return func(gc, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 86, in do_image_create
    do_image_upload(gc, args)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 324, in do_image_upload
    gc.images.upload(args.id, image_data, args.size)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 217, in upload
    self.http_client.put(url, headers=hdrs, data=body)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 185, in put
    return self.request(url, 'PUT', **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 346, in request
    return self._handle_response(resp)
  File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response
    raise exc.from_response(resp, resp.content)
HTTPInternalServerError: 500 Internal Server Error

# and image stays in "queued" state

Comment 2 Filip Hubík 2018-04-20 13:12:34 UTC


*** This bug has been marked as a duplicate of bug 1570007 ***