Description of problem: After updating to lates release (z5) glance commands exit with this error: "Internal Server Error" in glance log on controller-2 we found this error: 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi [req-0c73043f-94eb-49fa-90c8-9b2005723365 8c3cbf366ef54c40b292de07f1aa161d 5e62927ce5b14e7b9cc5a80ad2e25d49 - default default] Caught error: 'location_order' 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi Traceback (most recent call last): 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1116, in __call__ 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi request, **action_args) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1155, in dispatch 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi return method(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/v2/images.py", line 111, in index 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi member_status=member_status) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/authorization.py", line 111, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi images = self.image_repo.list(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 114, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi return super(ImageRepoProxy, self).list(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 89, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi items = self.base.list(*args, **kwargs) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/db/__init__.py", line 184, in list 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi image = self._format_image_from_db(db_image, db_image['tags']) 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/db/__init__.py", line 210, in _format_image_from_db 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi locations=location_strategy.get_ordered_locations(locations), 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi File "/usr/lib/python2.7/site-packages/glance/common/location_strategy/__init__.py", line 117, in get_ordered_locations 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi strategy_module = _available_strategies[CONF.location_strategy] 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi KeyError: 'location_order' 2018-05-22 11:47:05.580 747087 ERROR glance.common.wsgi Version-Release number of selected component (if applicable): python-glance-14.0.1-1.el7ost.noarch python-glanceclient-2.6.0-1.el7ost.noarch openstack-glance-14.0.1-1.el7ost.noarch python-glance-store-0.20.0-1.el7ost.noarch puppet-glance-10.4.0-1.el7ost.noarch How reproducible: Steps to Reproduce: 1. install z4 cloud 2. update cloud to z5 3. try to list or load image Actual results: see attached file. Expected results: Additional info:
Created attachment 1440246 [details] controller 2 logs
Does this happen on a machine we can tinker with? I'd love to add debug statements in _load_strategies and see what happens there.
I did try to reproduce this bug and failed, The glance error could be caused by tlv lab network problems.