Description of problem: OSP17 [RHEL8] Phase1 fails due to Glance failing to handle an image update. It's possible this is a deployment issue. The Compose ID is RHOS-17.0-RHEL-8-20210707.n.0 2021-07-08 17:12:33.663 30 DEBUG glance_store.location [req-6ce45a37-1ab9-44de-9567-0151f7a74ce3 0821654bdf0943e2a871433906070eb7 bd91378862544c44b8ca71978ed44e7c - default default] Registering scheme swift+config with {'default_backend': {'store': <glance_store._drivers.swift.store.SingleTenantStore object at 0x7f7f8806c8d0>, 'location_class': <class 'glance_store._drivers.swift.store.StoreLocation'>, 'store_entry': 'swift'}} register_scheme_backend_map /usr/lib/python3.6/site-packages/glance_store/location.py:132 2021-07-08 17:12:33.962 30 ERROR glance.api.v2.image_data [req-6ce45a37-1ab9-44de-9567-0151f7a74ce3 0821654bdf0943e2a871433906070eb7 bd91378862544c44b8ca71978ed44e7c - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Cannot find swift service endpoint : internalURL endpoint for object-store service not found 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi [req-6ce45a37-1ab9-44de-9567-0151f7a74ce3 0821654bdf0943e2a871433906070eb7 bd91378862544c44b8ca71978ed44e7c - default default] Caught error: Cannot find swift service endpoint : internalURL endpoint for object-store service not found: glance_store.exceptions.BackendException: Cannot find swift service endpoint : internalURL endpoint for object-store service not found 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi Traceback (most recent call last): 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/connection_manager.py", line 156, in _get_storage_url 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi region_name=self.store.region 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1243, in get_endpoint 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi return auth.get_endpoint(self, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi allow_version_hack=allow_version_hack, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 279, in get_endpoint_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi service_name=service_name) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/keystoneauth1/access/service_catalog.py", line 462, in endpoint_data_for 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi raise exceptions.EndpointNotFound(msg) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi keystoneauth1.exceptions.catalog.EndpointNotFound: internalURL endpoint for object-store service not found 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi Traceback (most recent call last): 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1354, in __call__ 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi request, **action_args) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/wsgi.py", line 1397, in dispatch 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/common/utils.py", line 416, in wrapped 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 300, in upload 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi self.force_reraise() 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi raise self.value 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/v2/image_data.py", line 165, in upload 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 501, in set_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi self.force_reraise() 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi raise self.value 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/notifier.py", line 448, in set_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi set_active=set_active) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/api/policy.py", line 273, in set_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/quota/__init__.py", line 323, in set_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi set_active=set_active) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 567, in set_data 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance/location.py", line 467, in _upload_to_store 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi verifier=verifier) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 399, in add_with_multihash 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi image_id, data, size, hashing_algo, store, context, verifier) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/multi_backend.py", line 481, in store_add_to_backend_with_multihash 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi image_id, data, size, hashing_algo, context=context, verifier=verifier) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 940, in add 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi allow_reauth=need_chunks) as manager: 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/store.py", line 1469, in get_manager 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi allow_reauth) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/connection_manager.py", line 62, in __init__ 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi self.storage_url = self._get_storage_url() 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi File "/usr/lib/python3.6/site-packages/glance_store/_drivers/swift/connection_manager.py", line 163, in _get_storage_url 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi raise exceptions.BackendException(msg) 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Cannot find swift service endpoint : internalURL endpoint for object-store service not found 2021-07-08 17:12:33.988 30 ERROR glance.common.wsgi 2021-07-08 17:12:34.000 30 INFO eventlet.wsgi.server [req-6ce45a37-1ab9-44de-9567-0151f7a74ce3 0821654bdf0943e2a871433906070eb7 bd91378862544c44b8ca71978ed44e7c - default default] 192.168.24.1 - - [08/Jul/2021 17:12:34] "PUT /v2/images/902c1d1b-4276-4f86-b497-7f577a260d80/file HTTP/1.1" 500 317 0.386453 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Correction: compose ID is RHOS-17.0-RHEL-8-20210708.n.0
> glance_store.exceptions.BackendException: Cannot find swift service endpoint : internalURL endpoint for object-store service not found Can we see the output of "openstack endpoint list"?
As mentioned in my email, the job is configured with enable_nova set to true. It shouldn't be. This is like a job config issue, however if the metalsmith items are the only supported method for provisioning in OSP17, we likely should fail or remove if enable_nova is set to true.
Compose was built and tested w/ http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp17/api/civotes_agg_detail.html?ref_hash=79a78b4bc12b9093f400f5351d125f79
This failure is because we've 'also' removed swift from the undercloud in wallaby upstream and glance (enabled with nova) uses swift backend as default in undercloud. So there are 2 options here.. 1. Convert all jobs to use metalsmith based provisioning and remove enable_nova flag 1. Enable swift like nova and make the jobs green till the jobs are converted. Please keep in mind nova was enabled long back when we removed it by default from upstream. Swift was removed later on. It was clearly a stop-gap arrangement till jobs were converted to use metalsmith.
This change added the enable_nova : https://review.gerrithub.io/c/redhat-openstack/infrared/+/502081 Is reverting the above change would make the "openstack overcloud image upload" working like it was described in the spec ? https://specs.openstack.org/openstack/tripleo-specs/specs/stein/nova-less-deploy.html Is 17 supposed to support only the new way or both should be supported ?
> Is reverting the above change would make the "openstack overcloud image upload" working like it was described in the spec ? Yes it would, as images would be uploaded locally without glance being available. > Is 17 supposed to support only the new way or both should be supported ? Old way is not the default, but not deprecated or removed yet upstream. I don't know if it would be deprecated in OSP17.
I would switch it to the metalsmith method and let it fail until things are fixed. We don't test the old way anymore and AFAIK it won't be a supported method for OSP17.
Adding a needsinfo for Steve to get his input, but ultimately it seems like infrared just needs to be updated.
If this option really not needed, not even for any kind upgrade just remove it or make sure the doc clearly states it is not supported. If the docs already makes the situation clear , just set this BZ to WONT FIX. For the time being I just insta revert the change, it will need some time to propane trough some ci protection layers. In case the old stuff needs to be supported for whatever reason please make an infrared ticket for making it an option.