Bug 1497169

Summary:	ems_refresh time increases when tries to update inventory for OpenStack10 provider with multiple tenants
Product:	Red Hat CloudForms Management Engine	Reporter:	Sachin <sacpatil>
Component:	Providers	Assignee:	Marek Aufart <maufart>
Status:	CLOSED NOTABUG	QA Contact:	Ola Pavlenko <opavlenk>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	5.7.0	CC:	gblomqui, igortiunov, jfrey, jhardy, obarenbo, sacpatil, tzumainn
Target Milestone:	GA
Target Release:	cfme-future
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-11-02 03:31:40 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	Bug
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	Openstack	Target Upstream Version:
Embargoed:

Description Sachin 2017-09-29 11:05:22 UTC

Description of problem:

Recently customer added new tenants in OpenStack10 provider from 5 to 140 and observed that the total ems_refresh time has increases ten times(from 300 to 3000 seconds).

Please find below data extracted from current logs from one appliance having 'ems_inventory' role in zone enabled. The collect_inventory_for_target and save_inventory is under limit but parse_legacy_inventory & parse_targeted_inventory is much higher.


| MessageID          | collect_inventory_for_target | parse_legacy_inventory | parse_targeted_inventory | save_inventory | ems_refresh |
|--------------------+------------------------------+------------------------+--------------------------+----------------+-------------|
| 16450095(full)     |                     0.000004 |            2471.296330 |              2471.296341 |       8.327366 | 2479.623937 |
| 16452364(full)     |                     0.000004 |            3032.477864 |              3032.477921 |       7.695983 | 3040.174192 |
| 16454435(full)     |                     0.000003 |            2988.104470 |              2988.104479 |       6.870958 | 2994.975635 |
| 16457320(targeted) |                     0.000003 |            2551.129330 |              2551.129341 |       8.136193 | 2559.265796 |
| 16449276(full)     |                     0.000004 |            5437.989589 |              5437.989606 |       7.208448 | 5445.198315 |
| 16450297(full)     |                     0.000005 |            5753.051083 |              5753.051096 |       3.397968 | 5756.449312 |


Customer has also provided list of methods spent most time,

- get_tenants
- get_flavors
- get_vailability_zones
- get_host_aggregates
- get_quotas
- get_key_pairs
- load_orchestration_stacks
- get_vnfs
- get_vnfds
- get_images
- get_servers
- get_cloud_services


Version-Release number of selected component (if applicable):
5.7.3.2

How reproducible:
Always

Steps to Reproduce:
1. 
2.
3.

Actual results:


Expected results:


Additional info:
- Newly added projects(135) do not have any VMs or other object. Basically they are empty.

Comment 4 Tzu-Mainn Chen 2017-10-02 12:37:26 UTC

Ah, I see! I think this may be related to this other BZ then:
https://bugzilla.redhat.com/show_bug.cgi?id=1460977. The fix for that BZ creates a config option that allows the refresh code to query all tenants at once for each query. This only works if the user is the 'admin' user. Would that fix your issue?

Comment 5 ITD27M01 2017-10-02 13:18:47 UTC

(In reply to Tzu-Mainn Chen from comment #4)
> Ah, I see! I think this may be related to this other BZ then:
> https://bugzilla.redhat.com/show_bug.cgi?id=1460977. The fix for that BZ
> creates a config option that allows the refresh code to query all tenants at
> once for each query. This only works if the user is the 'admin' user. Would
> that fix your issue?

Currently these ems settings is available only in CFME 5.8 and work if user is domain admin. I decided to upgrade 5.7 to 5.8 because I have no time to wait of backport for this bug.

For 'admin' user from default domain the collection of Cinder cloud volumes from another domains does not work. Do not forget that customers can use not only default domain but the domain with external authentication (like Active Directory). 

For this reason I create admin project in my custom domain and configure related advanced settings:

~~~~~~~~~~~~~~~~~~~~
:ems:
  :ems_openstack:
    :excon:
      :omit_default_port: true
      :read_timeout: 600
    :refresh:
      :is_admin: true  <----- Need to support fast refresh  by one api call
      :heat:
        :is_global_admin: true <----- Need to support fast refresh by one api call
~~~~~~~~~~~~~~~~~~~~

Currently the refresh timing in acceptable range - 1,5 to 3 minutes for CFME 5.8.1

Comment 8 Sachin 2017-11-02 03:31:40 UTC

Yes. Closing BZ as the fix is available in 5.8