Bug 1607368

Summary:	Inventory Refresh taking 10-14 mins per migrated vm
Product:	Red Hat CloudForms Management Engine	Reporter:	mlehrer
Component:	Providers	Assignee:	Fabien Dupont <fdupont>
Status:	CLOSED CURRENTRELEASE	QA Contact:	mlehrer
Severity:	medium	Docs Contact:
Priority:	high
Version:	5.9.4	CC:	bthurber, dmetzger, fdupont, gblomqui, istein, jfrey, jhardy, kkulkarn, mlehrer, obarenbo, simaishi, smallamp
Target Milestone:	GA	Keywords:	TestOnly, ZStream
Target Release:	5.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	5.10.0.11	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1610550 (view as bug list)		Environment:
Last Closed:	2019-02-11 14:18:27 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1668049, 1668791
Bug Blocks:	1564236, 1610550

Description mlehrer 2018-07-23 11:30:23 UTC

Description of problem:

In this test we migrated 10 vms in 1 plan from CFME to a heavily populated rhv 4.2 enviroment.
In the logs we saw that refresh inventory per each of the 10 vms was executed and had a duration of 10-14 mins

Per individual migration we see steps like:

Assess Migration
Acquire Transformation Host
Power off
Collapse Snapshots
Transform VM
Convert disks
Task state started
Refresh inventory   < === taking 10-14 mins
Update description of VM
Power-on VM
Restore VM Attributes
Mark source as migrated
Virtual machine migrated


The rhv which is being used as the target for this migration contains 4,000 vms and 400 hosts.


Version-Release number of selected component (if applicable):
cfme 5.9.4.0.20180711215406_53ad123

How reproducible:
reproduces

Steps to Reproduce:
1. Connect CFME with RHV with large inventory of 4k vms and 400 hosts
2. Build migration plan of N vms
3. Check logs to see inventory refresh occur N times taking 10-14 mins on average

Actual results:
Full inventory refresh

Expected results:
Not a full inventory refresh per vm migrated

Additional info:

- evm & automation logs can be found at this link inside 'engine_logs' folder [1]
- Break down of migration steps per each vm can be found here [2]



[1] https://drive.google.com/open?id=1q9NWVUMlpuCctH4XNMKGcchqKiCeObBN

[2] https://docs.google.com/spreadsheets/d/1td8JfUhzCpzHRNmx3FChx57-7m5gjx6cYY-HPdsTATg/edit?usp=sharing

Comment 2 Dave Johnson 2018-07-23 11:45:17 UTC

Please assess the impact of this issue and update the severity accordingly.  Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity for a reminder on each severity's definition.

If it's something like a tracker bug where it doesn't matter, please set the severity to Low.

Comment 6 Fabien Dupont 2018-07-30 18:06:42 UTC

Associated PR: https://github.com/ManageIQ/manageiq-content/pull/381

Comment 8 Fabien Dupont 2018-07-31 09:29:39 UTC

@mlehrer if you want to test the fix before it's added to a build, you can use this automate domain: https://github.com/fdupont-redhat/manageiq-content-381. Instructions to install are in the README.

Comment 10 Fabien Dupont 2018-08-09 13:27:54 UTC

Extra PR - Remove check on number of runners when converting VM
https://github.com/ManageIQ/manageiq-content/pull/395

Comment 11 Fabien Dupont 2018-08-20 06:20:27 UTC

The previously mentioned PR was incorrectly linked and does not belong to this BZ.

Comment 12 Ilanit Stein 2019-01-30 13:22:47 UTC

Verified on CFME-5.10.0.32/RHV-4.2.8-4.
Analyzing evm.log, after migration of 20 VMs, 100G disk each, to RHV with:
10 Clusters
430 Hosts
50 Datastores
4365 VMs
22 Templates

Here's the target refresh, triggered for 1 VM ("100_VM_TEST_15"), instead of a full refresh:
(This target refresh took several seconds)
  

[----] I, [2019-01-23T14:56:47.425985 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#refresh) EMS: [RHV], id: [4]   InventoryRefresh::TargetCollection [Collection of 1 targets] id [Collection of targets with id: ["100_VM_TEST_15"]]

[----] I, [2019-01-23T14:56:47.546882 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#collect_inventory_for_targets) Filtering inventory for InventoryRefresh::TargetCollection [Collection of 1 targets] id: [Collection of targets with id: ["100_VM_TEST_15"]]...

[----] I, [2019-01-23T14:56:50.781780 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#refresh_targets_for_ems) EMS: [RHV], id: [4] Refreshing target InventoryRefresh::TargetCollection [Collection of 19 targets] id [Collection of targets with id: ["100_VM_TEST_15", {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/1bba0366-1769-11e9-ad7c-b8ca3a638904"}, {:ems_ref=>"/api/clusters/9181171d-7e3a-4239-b74a-6b034b4ce040"}, {:ems_ref=>"/api/clusters/dd0a6157-a4fa-47ad-9b92-d3aee918944c"}, {:ems_ref=>"/api/clusters/140d40a0-1f54-4bbe-85a4-2e091e3a8899"}, {:ems_ref=>"/api/clusters/5f0d81fd-a551-42cf-a76e-06c9ddcfbc19"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/clusters/e4928991-caa0-4f22-93ae-d1fced8f8071"}, {:ems_ref=>"/api/clusters/cbc9c8ed-428e-4fb3-ac0b-397c4b59bfc3"}, {:ems_ref=>"/api/clusters/10a2397a-9e59-47da-a1fc-dafee9c39152"}, {:ems_ref=>"/api/clusters/88a91e70-4127-4b5c-ae49-b5531a2643df"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}]]...

----] I, [2019-01-23T14:56:51.866684 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#refresh_targets_for_ems) EMS: [RHV], id: [4] Refreshing target InventoryRefresh::TargetCollection [Collection of 19 targets] id [Collection of targets with id: ["100_VM_TEST_15", {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/1bba0366-1769-11e9-ad7c-b8ca3a638904"}, {:ems_ref=>"/api/clusters/9181171d-7e3a-4239-b74a-6b034b4ce040"}, {:ems_ref=>"/api/clusters/dd0a6157-a4fa-47ad-9b92-d3aee918944c"}, {:ems_ref=>"/api/clusters/140d40a0-1f54-4bbe-85a4-2e091e3a8899"}, {:ems_ref=>"/api/clusters/5f0d81fd-a551-42cf-a76e-06c9ddcfbc19"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/clusters/e4928991-caa0-4f22-93ae-d1fced8f8071"}, {:ems_ref=>"/api/clusters/cbc9c8ed-428e-4fb3-ac0b-397c4b59bfc3"}, {:ems_ref=>"/api/clusters/10a2397a-9e59-47da-a1fc-dafee9c39152"}, {:ems_ref=>"/api/clusters/88a91e70-4127-4b5c-ae49-b5531a2643df"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}]]...Complete