Bug 1607368 - Inventory Refresh taking 10-14 mins per migrated vm
Summary: Inventory Refresh taking 10-14 mins per migrated vm
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.9.4
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: GA
: 5.10.0
Assignee: Fabien Dupont
QA Contact: mlehrer
URL:
Whiteboard:
Depends On: 1668049 1668791
Blocks: 1564236 1610550
TreeView+ depends on / blocked
 
Reported: 2018-07-23 11:30 UTC by mlehrer
Modified: 2019-08-22 00:54 UTC (History)
12 users (show)

Fixed In Version: 5.10.0.11
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1610550 (view as bug list)
Environment:
Last Closed: 2019-02-11 14:18:27 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github patternfly patternfly issues 1096 0 None None None 2018-07-25 16:16:29 UTC

Description mlehrer 2018-07-23 11:30:23 UTC
Description of problem:

In this test we migrated 10 vms in 1 plan from CFME to a heavily populated rhv 4.2 enviroment.
In the logs we saw that refresh inventory per each of the 10 vms was executed and had a duration of 10-14 mins

Per individual migration we see steps like:

Assess Migration
Acquire Transformation Host
Power off
Collapse Snapshots
Transform VM
Convert disks
Task state started
Refresh inventory   < === taking 10-14 mins
Update description of VM
Power-on VM
Restore VM Attributes
Mark source as migrated
Virtual machine migrated


The rhv which is being used as the target for this migration contains 4,000 vms and 400 hosts.


Version-Release number of selected component (if applicable):
cfme 5.9.4.0.20180711215406_53ad123

How reproducible:
reproduces

Steps to Reproduce:
1. Connect CFME with RHV with large inventory of 4k vms and 400 hosts
2. Build migration plan of N vms
3. Check logs to see inventory refresh occur N times taking 10-14 mins on average

Actual results:
Full inventory refresh

Expected results:
Not a full inventory refresh per vm migrated

Additional info:

- evm & automation logs can be found at this link inside 'engine_logs' folder [1]
- Break down of migration steps per each vm can be found here [2]



[1] https://drive.google.com/open?id=1q9NWVUMlpuCctH4XNMKGcchqKiCeObBN

[2] https://docs.google.com/spreadsheets/d/1td8JfUhzCpzHRNmx3FChx57-7m5gjx6cYY-HPdsTATg/edit?usp=sharing

Comment 2 Dave Johnson 2018-07-23 11:45:17 UTC
Please assess the impact of this issue and update the severity accordingly.  Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity for a reminder on each severity's definition.

If it's something like a tracker bug where it doesn't matter, please set the severity to Low.

Comment 6 Fabien Dupont 2018-07-30 18:06:42 UTC
Associated PR: https://github.com/ManageIQ/manageiq-content/pull/381

Comment 8 Fabien Dupont 2018-07-31 09:29:39 UTC
@mlehrer if you want to test the fix before it's added to a build, you can use this automate domain: https://github.com/fdupont-redhat/manageiq-content-381. Instructions to install are in the README.

Comment 10 Fabien Dupont 2018-08-09 13:27:54 UTC
Extra PR - Remove check on number of runners when converting VM
https://github.com/ManageIQ/manageiq-content/pull/395

Comment 11 Fabien Dupont 2018-08-20 06:20:27 UTC
The previously mentioned PR was incorrectly linked and does not belong to this BZ.

Comment 12 Ilanit Stein 2019-01-30 13:22:47 UTC
Verified on CFME-5.10.0.32/RHV-4.2.8-4.
Analyzing evm.log, after migration of 20 VMs, 100G disk each, to RHV with:
10 Clusters
430 Hosts
50 Datastores
4365 VMs
22 Templates

Here's the target refresh, triggered for 1 VM ("100_VM_TEST_15"), instead of a full refresh:
(This target refresh took several seconds)
  

[----] I, [2019-01-23T14:56:47.425985 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#refresh) EMS: [RHV], id: [4]   InventoryRefresh::TargetCollection [Collection of 1 targets] id [Collection of targets with id: ["100_VM_TEST_15"]]

[----] I, [2019-01-23T14:56:47.546882 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#collect_inventory_for_targets) Filtering inventory for InventoryRefresh::TargetCollection [Collection of 1 targets] id: [Collection of targets with id: ["100_VM_TEST_15"]]...

[----] I, [2019-01-23T14:56:50.781780 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#refresh_targets_for_ems) EMS: [RHV], id: [4] Refreshing target InventoryRefresh::TargetCollection [Collection of 19 targets] id [Collection of targets with id: ["100_VM_TEST_15", {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/1bba0366-1769-11e9-ad7c-b8ca3a638904"}, {:ems_ref=>"/api/clusters/9181171d-7e3a-4239-b74a-6b034b4ce040"}, {:ems_ref=>"/api/clusters/dd0a6157-a4fa-47ad-9b92-d3aee918944c"}, {:ems_ref=>"/api/clusters/140d40a0-1f54-4bbe-85a4-2e091e3a8899"}, {:ems_ref=>"/api/clusters/5f0d81fd-a551-42cf-a76e-06c9ddcfbc19"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/clusters/e4928991-caa0-4f22-93ae-d1fced8f8071"}, {:ems_ref=>"/api/clusters/cbc9c8ed-428e-4fb3-ac0b-397c4b59bfc3"}, {:ems_ref=>"/api/clusters/10a2397a-9e59-47da-a1fc-dafee9c39152"}, {:ems_ref=>"/api/clusters/88a91e70-4127-4b5c-ae49-b5531a2643df"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}]]...

----] I, [2019-01-23T14:56:51.866684 #48583:f3af48]  INFO -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Refresh::Strategies::Api4#refresh_targets_for_ems) EMS: [RHV], id: [4] Refreshing target InventoryRefresh::TargetCollection [Collection of 19 targets] id [Collection of targets with id: ["100_VM_TEST_15", {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}, {:ems_ref=>"/api/clusters/1bba0366-1769-11e9-ad7c-b8ca3a638904"}, {:ems_ref=>"/api/clusters/9181171d-7e3a-4239-b74a-6b034b4ce040"}, {:ems_ref=>"/api/clusters/dd0a6157-a4fa-47ad-9b92-d3aee918944c"}, {:ems_ref=>"/api/clusters/140d40a0-1f54-4bbe-85a4-2e091e3a8899"}, {:ems_ref=>"/api/clusters/5f0d81fd-a551-42cf-a76e-06c9ddcfbc19"}, {:ems_ref=>"/api/clusters/0005b216-b980-450c-8f98-94e1d2f34aef"}, {:ems_ref=>"/api/datacenters/a729cda4-ff0f-4bf6-b55e-dcef47752c7d"}, {:ems_ref=>"/api/clusters/e4928991-caa0-4f22-93ae-d1fced8f8071"}, {:ems_ref=>"/api/clusters/cbc9c8ed-428e-4fb3-ac0b-397c4b59bfc3"}, {:ems_ref=>"/api/clusters/10a2397a-9e59-47da-a1fc-dafee9c39152"}, {:ems_ref=>"/api/clusters/88a91e70-4127-4b5c-ae49-b5531a2643df"}, {:ems_ref=>"/api/storagedomains/a6e6b684-8fc7-4d58-917b-000cb9e108c6"}, {:ems_ref=>"/api/vms/8d88d337-a8ee-4857-9478-b84a91f2db94"}]]...Complete


Note You need to log in before you can comment on or make changes to this bug.