Created attachment 1424504 [details] partial logfile glance/api.log Description of problem: Glance can not upload image to registry, glance.log reports following exception: ... DEBUG glance_store.capabilities [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Store glance_store._drivers.rbd.Store doesn't support updating dynamic storage capabilities. Please overwrite 'update_capabilities' method of the store to implement updating logics if needed. update_capabilities /usr/lib/python2.7/site-packages/glance_store/capabilities.py:97 DEBUG glance_store.driver [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Late loading location class glance_store._drivers.rbd.StoreLocation get_store_location_class /usr/lib/python2.7/site-packages/glance_store/driver.py:89 DEBUG glance_store.location [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Registering scheme rbd with {'location_class': <class 'glance_store._drivers.rbd.StoreLocation'>, 'store': <glance_store._drivers.rbd.Store object at 0x7fad5a0c0290>, 'store_entry': 'glance.store.rbd.Store'} register_scheme_map /usr/lib/python2.7/site-packages/glance_store/location.py:88 ERROR glance_store._drivers.rbd [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Error connecting to ceph cluster. ERROR glance_store._drivers.rbd Traceback (most recent call last): ERROR glance_store._drivers.rbd File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 273, in get_connection ERROR glance_store._drivers.rbd client.connect(timeout=self.connect_timeout) ERROR glance_store._drivers.rbd File "rados.pyx", line 785, in rados.Rados.connect (rados.c:8969) ERROR glance_store._drivers.rbd ObjectNotFound: error connecting to the cluster ERROR glance_store._drivers.rbd DEBUG oslo_messaging._drivers.amqpdriver [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] CAST unique_id: 91a51cc277804e4fbf0c8655 f854e11d NOTIFY exchange 'glance' topic 'notifications.error' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:552 ERROR glance.api.v2.image_data [req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea 3a941b5a446a4b4a83ab0006bb19931c c1a39a4ee28c4f3184fd6384b09d6ee3 - default default] Failed to upload image data due to internal error ERROR glance.api.v2.image_data Traceback (most recent call last): ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload ERROR glance.api.v2.image_data image.set_data(data, size) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data ERROR glance.api.v2.image_data self.base.set_data(data, size) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data ERROR glance.api.v2.image_data _send_notification(notify_error, 'image.upload', msg) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ ERROR glance.api.v2.image_data self.force_reraise() ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise ERROR glance.api.v2.image_data six.reraise(self.type_, self.value, self.tb) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data ERROR glance.api.v2.image_data self.repo.set_data(data, size) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 187, in set_data ERROR glance.api.v2.image_data return self.image.set_data(*args, **kwargs) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data ERROR glance.api.v2.image_data self.image.set_data(data, size=size) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data ERROR glance.api.v2.image_data verifier=verifier) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 452, in add_to_backend ERROR glance.api.v2.image_data verifier) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 425, in store_add_to_backend ERROR glance.api.v2.image_data verifier=verifier) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker ERROR glance.api.v2.image_data return store_op_fun(store, *args, **kwargs) ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 447, in add ERROR glance.api.v2.image_data rados_id=self.user) as conn: ERROR glance.api.v2.image_data File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ ERROR glance.api.v2.image_data return self.gen.next() ERROR glance.api.v2.image_data File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 277, in get_connection ERROR glance.api.v2.image_data raise exceptions.BackendException() ERROR glance.api.v2.image_data BackendException ERROR glance.api.v2.image_data ... (full exception attached) Version-Release number of selected component (if applicable): Started since puddle 2018-04-19.1, OSP10. How reproducible: always Steps to Reproduce: 1. Deploy OSP10 topology virthost-1cont_1comp_1ceph on RHEL 7.5 2. Tempest won't run since it can not even upload cirros image to glance Additional info: *ceph's health and status reports OK on ceph node *this sounds like issue with glance <-> rbd comminication *manual attempt to upload to glance with debug enabled on client side: PUT call to glance-api for http://10.0.0.110:9292/v2/images/1d85055a-5a8e-45ce-a577-f304c66b7de3/file used request id req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea DEBUG:glanceclient.common.http:PUT call to glance-api for http://10.0.0.110:9292/v2/images/1d85055a-5a8e-45ce-a577-f304c66b7de3/file used request id req-bac55fb6-a4a0-4394-aca1-eb3a5e299aea Request returned failure status 500. DEBUG:glanceclient.common.http:Request returned failure status 500. +------------------+--------------------------------------+ | Property | Value | +------------------+--------------------------------------+ | checksum | None | | container_format | bare | | created_at | 2018-04-20T12:45:44Z | | disk_format | qcow2 | | id | 1d85055a-5a8e-45ce-a577-f304c66b7de3 | | locations | [] | | min_disk | 0 | | min_ram | 0 | | name | cirros | | owner | c1a39a4ee28c4f3184fd6384b09d6ee3 | | protected | False | | size | None | | status | queued | | tags | [] | | updated_at | 2018-04-20T12:45:44Z | | virtual_size | None | | visibility | private | +------------------+--------------------------------------+ Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 668, in main OpenStackImagesShell().main(argv) File "/usr/lib/python2.7/site-packages/glanceclient/shell.py", line 599, in main args.func(client, args) File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 98, in func_wrapper return func(gc, args) File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 86, in do_image_create do_image_upload(gc, args) File "/usr/lib/python2.7/site-packages/glanceclient/v2/shell.py", line 324, in do_image_upload gc.images.upload(args.id, image_data, args.size) File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 217, in upload self.http_client.put(url, headers=hdrs, data=body) File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 185, in put return self.request(url, 'PUT', **kwargs) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 346, in request return self._handle_response(resp) File "/usr/lib/python2.7/site-packages/glanceclient/common/http.py", line 107, in _handle_response raise exc.from_response(resp, resp.content) HTTPInternalServerError: 500 Internal Server Error # and image stays in "queued" state
*** This bug has been marked as a duplicate of bug 1570007 ***