Description of problem: If an OpenStack provider has more than 25 images, only the first 25 are inventoried. Version-Release number of selected component (if applicable): 5.3.1.8.20141030214429 How reproducible:100% Steps to Reproduce: 1. Add more than 25 images to an OpenStack instance (RHOS4 tested) 2. Iventory/Refresh provider 3. Actual results: Only the first 25 images are present in CFME Expected results: All images should be present Additional info: I couldn't really understand what was happening here, so I took a little time to dig as it was discovered during an Automation run. I used wireshark to track the API calls that CFME was making. and got the following: GET /v1.1/images/detail HTTP/1.1 User-Agent: fog/1.19.0 The response gave 25 entries, so I tried it using pure "requests" in python using this snippet and got the same, data_json = {"auth": {"tenantName": "xxxxxxx", "passwordCredentials": {"username": "xxxxxxxx", "password": "xxxxxxxx"}}} l = requests.post("http://xx.xx.xx.xx:yyyy/v2.0/tokens", headers={'Content-Type': 'application/json'}, data=json.dumps(data_json)) token = l.json()['access']['token']['id'] headers = {'Content-Type': 'application/json', 'X-Auth-Token':token} r = requests.get("http://xx.xx.xx.xx:xxxx/v1.1/images", headers=headers) print r.json() print len(r.json()['images']) >>> 25 However, when doing the call with the native python client I see that it first does: GET /v2/images?limit=20 HTTP/1.1 before following it up with: GET /v2/images?marker=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&limit=20 HTTP/1.1 Where xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx is the GUID of the last image from the previous list. This is apparently the OpenStack APIs way of doing pagination and I get the correct number of images listed. I believe CFME uses fog to do this work, but I could not see the images.all() method from Fog:Compute:OpenStack:Images being invoked when doing a search of the code base, apart from here https://github.com/ManageIQ/manageiq/blob/d9f336ede36d929cfab6c5ce2b15df9542df2538/lib/openstack/test/image_openstack_handle_example.rb which is apparently just a test case. So maybe the fog call is not being used in this way. It appears that Fog:Image:OpenStack _is_ being used here https://github.com/ManageIQ/manageiq/blob/d9f336ede36d929cfab6c5ce2b15df9542df2538/lib/openstack/openstack_handle/image_delegate.rb I am unsure, but it may be that this http://www.rubydoc.info/gems/fog/1.19.0/Fog/Compute/OpenStack/Images#all-instance_method is the right call instead? I'm no Ruby expert, just what I found from digging a little.
Seems like FOG is not defining any limit. Default request is clean https://github.com/fog/fog/blob/f1500ba0003ac232d581ac8f1fdcf98f717d92ec/lib/fog/openstack/requests/image/list_public_images_detailed.rb#L9 Any chance you have defined limit in your /etc/glance/.. 'limit_param_default' or 'api_limit_max'? It's being used here https://github.com/openstack/glance/blob/12f12fb82ea78e23a276b2f0fb9faa3efa374ce3/glance/api/v2/images.py#L98 Even if you didn't set it, default should be 25, as stated here: http://docs.openstack.org/juno/config-reference/content/section_glance-registry.conf.html Seems like the fix for this bug is, that we need to work with pagination correctly, since we can't be sure pagination is disabled. This is also the way python-glanceclient does it. But, seems like FOG has a bug when sending parameter, e.g: svc.send(:list_public_images_detailed, 'limit', '20') URI::InvalidComponentError Exception: bad component(expected absolute path component): /v1.1/images/detail?limit=20 Plus it allows to send only one parametere now, which probable won't be enough. I suspect we also need to specify limit, because without that I don't see body['next'] which should have the next marker.
Yes, that is defined in my environment I believe.
btw. seems like this is the FOG bug I am seeing, the url with ? doesn't pass URI::parser.regexp[:ABS_PATH] (byebug) v='/v1.1/images/detail?limit=20' "/v1.1/images/detail?limit=20" (byebug) URI::parser.regexp[:ABS_PATH] !~ v true (byebug) v='/v1.1/images/detail' "/v1.1/images/detail" (byebug) URI::parser.regexp[:ABS_PATH] !~ v false Any idea how to define the URL correctly?
I've pushed fix to FOG, the followup CFME fix will depend on this https://github.com/fog/fog/pull/3321
Another FOG patch dependency. We need to be able to pass the query to the images model https://github.com/fog/fog/pull/3322
Fix is here: https://github.com/ManageIQ/manageiq/pull/1198 Anybody has experience with the FOG CI? Seems like the gate is failing on some internal stuff, can I just recheck it somehow?
Ladislav, I have a little experience with FOG CI. I also have seen PRs merged in FOG with failing tests. :/
New commit detected on manageiq/master: https://github.com/ManageIQ/manageiq/commit/0ac11e0b57fb077cffabbc101c813ea3eef7cc8d commit 0ac11e0b57fb077cffabbc101c813ea3eef7cc8d Author: Ladislav Smola <lsmola> AuthorDate: Tue Dec 9 13:07:42 2014 +0100 Commit: Ladislav Smola <lsmola> CommitDate: Tue Dec 16 15:41:02 2014 +0100 Fix OpenStack images pagination Pagination is always forced by Glance conf file. So we need to always obtain all pages of images in cycle. Depends-On: https://github.com/fog/fog/pull/3321 Depends-On: https://github.com/fog/fog/pull/3322 https://bugzilla.redhat.com/show_bug.cgi?id=1164764 lib/openstack/openstack_handle/image_delegate.rb | 17 +- .../openstack_refresher_rhos_havana_spec.rb | 26 +- .../openstack_refresher_rhos_grizzly.yml | 1286 +++++++++++--------- .../refreshers/openstack_refresher_rhos_havana.yml | 879 +++++++------ 4 files changed, 1226 insertions(+), 982 deletions(-)
Just a quick note on verification There are three values that control the glance API limit: 1) the limit parameter passed in via the API call We need to patch Fog (being tracked in the upstream clone of this bug) to allow CFME to pass this value to Glance via the API. Once we have this ability, we can control how we gather images with pagination enabled in Glance. 2. the limit_param_default glance-registry.conf file setting If no parameter is passed in the API call, the limit_param_default is used. 3. the api_limit_max glance-registry.conf file setting The api_limit_max setting is used as an upper bound on the number of images returned via the Glance API call. The logic for setting the image count limit looks like: limit = value_from_api_call || limit_param_default limit = min(api_limit_max, limit) When attempting to verify this fix, make sure that there are enough images to cause pagination based on the above settings.
Verified in 5.4.0.0.11
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-1100.html