Bug 1223536 - CF ems refresh doesn't find all instances in OSP !>1000
Summary: CF ems refresh doesn't find all instances in OSP !>1000
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.3.0
Hardware: All
OS: All
high
high
Target Milestone: GA
: 5.5.0
Assignee: Ladislav Smola
QA Contact: Pete Savage
URL:
Whiteboard:
Depends On:
Blocks: 1227909 1227917 1232423
TreeView+ depends on / blocked
 
Reported: 2015-05-20 19:32 UTC by Josh Carter
Modified: 2019-08-15 04:37 UTC (History)
10 users (show)

Fixed In Version: 5.5.0.1
Doc Type: Bug Fix
Doc Text:
The previous version of CloudForms Management Engine had a bug in the OpenStack inventory gathering code, which limited its ability to find OpenStack server instances to a maximum of 1000 instances. The bug was a result of the inventory code not supporting paginated queries for OpenStack server instances. This bug was fixed by adding pagination support to the query for OpenStack server instances. The new version of CloudForms Management Engine is able to list all OpenStack server instances for an OpenStack provider.
Clone Of:
: 1227909 1227917 (view as bug list)
Environment:
Last Closed: 2015-12-08 13:11:01 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:2551 0 normal SHIPPED_LIVE Moderate: CFME 5.5.0 bug fixes and enhancement update 2015-12-08 17:58:09 UTC

Description Josh Carter 2015-05-20 19:32:06 UTC
Description of problem:

CloudForms inventory does not get all the instances from OpenStack.  Seems it doesn't support pagination.  Our OSP has exceeded 1000 instances.  To  confirm and workaround we have adjusted the osapi_max_limit to 2000.  We need a permanent fix as we expect we will have 10,000 or so instances and cannot tolerate the response times from nova and cinder when we adjust osapi_max_limit upwards.  We expect to return this osapi_max_limit setting back to 1000 as soon as we can.

Version-Release number of selected component (if applicable):
5.3.2

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Greg Blomquist 2015-05-20 19:36:58 UTC
Looking at the openstack source, I'm seeing "limit" and "marker" API parameters in nova's API for getting servers[1] and flavors[2].

We should fix both while we're at it.  I'll look through the rest of the service APIs to find other pageable API calls.  We'll need to fix this once and do it right.

[1] https://github.com/openstack/nova/blob/0fce05b27c8fc80042c0afd65aa38ebe3334f72e/nova/api/openstack/compute/servers.py#L217

[2] https://github.com/openstack/nova/blob/0fce05b27c8fc80042c0afd65aa38ebe3334f72e/nova/api/openstack/compute/flavors.py#L75

Comment 3 Greg Blomquist 2015-05-27 12:46:34 UTC
https://github.com/ManageIQ/manageiq/pull/2967

Comment 7 CFME Bot 2015-06-03 19:31:18 UTC
New commit detected on manageiq/master:
https://github.com/ManageIQ/manageiq/commit/44ce2f935cf94022fc92d502e21e3c8c5e539ad9

commit 44ce2f935cf94022fc92d502e21e3c8c5e539ad9
Author:     Ladislav Smola <lsmola>
AuthorDate: Wed Jun 3 11:06:54 2015 +0200
Commit:     Ladislav Smola <lsmola>
CommitDate: Wed Jun 3 13:04:01 2015 +0200

    Fix Openstack servers pagination issue
    
    Fix Openstack servers pagination issue similar to how we do it
    for images. This is small patch for backport. Followup PRs needs
    to unify this in Fog, then in ManageIQ
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1223536

 lib/openstack/openstack_handle/compute_delegate.rb |   18 +-
 lib/openstack/openstack_handle/image_delegate.rb   |    8 +-
 .../openstack_refresher_rhos_havana_spec.rb        |   30 +-
 .../openstack_infra_refresher_rhos_juno.yml        | 1097 ++++----
 .../openstack_refresher_rhos_grizzly.yml           |   33 +
 .../refreshers/openstack_refresher_rhos_havana.yml | 2954 +++++++++++++++-----
 6 files changed, 2866 insertions(+), 1274 deletions(-)

Comment 10 CFME Bot 2015-06-22 15:29:51 UTC
New commit detected on cfme/5.4.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=53ac99295c7073c0b9f1c1afc4a0d5478ecf6418

commit 53ac99295c7073c0b9f1c1afc4a0d5478ecf6418
Author:     Ladislav Smola <lsmola>
AuthorDate: Wed Jun 3 11:06:54 2015 +0200
Commit:     Joe VLcek <jvlcek>
CommitDate: Fri Jun 5 13:59:57 2015 -0400

    Fix Openstack servers pagination issue
    
    Fix Openstack servers pagination issue similar to how we do it
    for images. This is small patch for backport. Followup PRs needs
    to unify this in Fog, then in ManageIQ
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1223536

 lib/openstack/openstack_handle/compute_delegate.rb |   18 +-
 lib/openstack/openstack_handle/image_delegate.rb   |    8 +-
 .../openstack_refresher_rhos_havana_spec.rb        |   30 +-
 .../openstack_infra_refresher_rhos_juno.yml        | 1097 ++++----
 .../openstack_refresher_rhos_grizzly.yml           |   33 +
 .../refreshers/openstack_refresher_rhos_havana.yml | 2954 +++++++++++++++-----
 6 files changed, 2866 insertions(+), 1274 deletions(-)

Comment 12 Ladislav Smola 2015-07-29 08:53:42 UTC
@psav nevermind, it was my error in Fog, so we should be good with this fix

Comment 13 Pete Savage 2015-10-30 08:50:31 UTC
Verified in 5.5.0.7

Comment 15 errata-xmlrpc 2015-12-08 13:11:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551


Note You need to log in before you can comment on or make changes to this bug.