1610054 – [v2v] Migration balance is not working as expected

Bug 1610054 - [v2v] Migration balance is not working as expected

Summary: [v2v] Migration balance is not working as expected

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat CloudForms Management Engine
Classification:	Red Hat
Component:	UI - OPS
Sub Component:
Version:	5.9.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	urgent
Target Milestone:	GA
Target Release:	5.9.4
Assignee:	Fabien Dupont
QA Contact:	Jad Haj Yahya
Docs Contact:
URL:
Whiteboard:	v2v
Depends On:	1600152
Blocks:
TreeView+	depends on / blocked

Reported:	2018-07-30 23:36 UTC by Satoe Imaishi
Modified:	2018-09-05 12:56 UTC (History)
CC List:	13 users (show)
Fixed In Version:	5.9.4.4
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:	1600152
Environment:
Last Closed:	2018-09-05 12:56:29 UTC
Category:	---
Cloudforms Team:	CFME Core
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
evm and automation logs and time of wrapper calling virt-v2v tool (3.49 MB, application/octet-stream) 2018-08-08 15:00 UTC, Mor	no flags	Details
View All

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	ManageIQ manageiq-content pull 358	0	None	closed	Fix computation of currently running conversions by host	2020-08-25 11:24:11 UTC

Comment 3 CFME Bot 2018-07-31 01:01:48 UTC

New commit detected on ManageIQ/manageiq-content/gaprindashvili:

https://github.com/ManageIQ/manageiq-content/commit/99fff66d837394b3eee4bdf639c0bb44b71a7464
commit 99fff66d837394b3eee4bdf639c0bb44b71a7464
Author:     Greg McCullough <gmccullo>
AuthorDate: Mon Jul 30 13:41:19 2018 -0400
Commit:     Greg McCullough <gmccullo>
CommitDate: Mon Jul 30 13:41:19 2018 -0400

    Merge pull request #379 from fdupont-redhat/v2v_fix_transformation_host_selection

    Fix transformation host selection
    (cherry picked from commit b2982c6bc856b5b6ad4af762dfa028cf9df9bb11)

    https://bugzilla.redhat.com/show_bug.cgi?id=1610054

 content/automate/ManageIQ/Transformation/TransformationHosts/Common.class/__methods__/utils.rb | 44 +-
 content/automate/ManageIQ/Transformation/TransformationHosts/Common.class/__methods__/vmtransform.rb | 2 +-
 2 files changed, 31 insertions(+), 15 deletions(-)

Comment 5 Mor 2018-08-08 14:54:38 UTC

I'm failing it on the following environment:
CFME 5.9.4.2
RHV 4.2.5.2-0.1.el7ev

Fabien also did a check on the evm.log file, and he found that the distribution went wrong on the run. 95% of the task end up on the same host: b02-h23-r620.rhev.openstack.engineering.redhat.com, while they should be equally split between the two hosts:
$ grep service_template_transformation_plan_task_ automation.log | awk '{ print $8; }' | sort -u | grep -v automation | cut -d "[" -f2 | cut -d "]" -f1 | while read task ; do grep "${task}" automation.log | grep 'acquiretransformationhost> Transformation Host: ' ; done
[----] I, [2018-08-07T16:07:41.589414 #2187:4fef494]  INFO -- : Q-task_id([service_template_transformation_plan_task_187]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:07:40.687186 #2179:32f32dc]  INFO -- : Q-task_id([service_template_transformation_plan_task_188]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:09.753008 #2179:45a6410]  INFO -- : Q-task_id([service_template_transformation_plan_task_189]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:14.161519 #2187:13d20c4]  INFO -- : Q-task_id([service_template_transformation_plan_task_190]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:36.205294 #2179:4de9bb8]  INFO -- : Q-task_id([service_template_transformation_plan_task_191]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:41.128391 #2187:451bd60]  INFO -- : Q-task_id([service_template_transformation_plan_task_192]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:03.026338 #2179:448f630]  INFO -- : Q-task_id([service_template_transformation_plan_task_193]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:07.558536 #2187:4fc679c]  INFO -- : Q-task_id([service_template_transformation_plan_task_194]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:29.818315 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_195]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:34.514493 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_196]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:56.853518 #2179:106ea2c]  INFO -- : Q-task_id([service_template_transformation_plan_task_197]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:01.765281 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_198]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:23.999313 #2179:106ea2c]  INFO -- : Q-task_id([service_template_transformation_plan_task_199]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:28.929544 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_200]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:50.888203 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_201]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:55.828331 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_202]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:17.774265 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_203]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:22.857803 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_204]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:44.905029 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_205]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:50.065942 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_206]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:28:41.082774 #2187:507600c]  INFO -- : Q-task_id([service_template_transformation_plan_task_227]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:28:41.816563 #2179:5680a74]  INFO -- : Q-task_id([service_template_transformation_plan_task_228]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:09.849669 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_229]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:14.911800 #2179:106efcc]  INFO -- : Q-task_id([service_template_transformation_plan_task_230]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:36.875056 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_231]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:42.857836 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_232]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:05.631546 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_233]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:11.390289 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_234]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:33.515944 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_235]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:39.461398 #2179:106ee8c]  INFO -- : Q-task_id([service_template_transformation_plan_task_236]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:01.534685 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_237]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:07.317003 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_238]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:29.746206 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_239]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:35.981348 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_240]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:58.043836 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_241]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:03.995541 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_242]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:25.947143 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_243]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:32.062706 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_244]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:53.725072 #2187:454e97c]  INFO -- : Q-task_id([service_template_transformation_plan_task_245]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:59.953511 #2179:106ee8c]  INFO -- : Q-task_id([service_template_transformation_plan_task_246]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:40:41.989470 #2187:4fd4a18]  INFO -- : Q-task_id([service_template_transformation_plan_task_267]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:40:42.402620 #2179:5ca5ddc]  INFO -- : Q-task_id([service_template_transformation_plan_task_268]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:15.132479 #2187:421a56c]  INFO -- : Q-task_id([service_template_transformation_plan_task_269]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:15.632362 #2179:4fa018c]  INFO -- : Q-task_id([service_template_transformation_plan_task_270]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:48.415274 #2187:3b0c568]  INFO -- : Q-task_id([service_template_transformation_plan_task_271]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:48.777358 #2179:4f95958]  INFO -- : Q-task_id([service_template_transformation_plan_task_272]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:21.237464 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_273]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:21.752712 #2187:45cdb00]  INFO -- : Q-task_id([service_template_transformation_plan_task_274]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:54.182562 #2179:509d224]  INFO -- : Q-task_id([service_template_transformation_plan_task_275]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:54.918697 #2187:4503d8c]  INFO -- : Q-task_id([service_template_transformation_plan_task_276]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:18:37.563996 #2179:59e1718]  INFO -- : Q-task_id([service_template_transformation_plan_task_277]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:18:05.650024 #2179:5beed80]  INFO -- : Q-task_id([service_template_transformation_plan_task_278]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:20:28.677262 #56714:5e7b6d4]  INFO -- : Q-task_id([service_template_transformation_plan_task_279]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:15:47.404147 #2187:5befb40]  INFO -- : Q-task_id([service_template_transformation_plan_task_280]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:18:54.840479 #56714:5704c5c]  INFO -- : Q-task_id([service_template_transformation_plan_task_281]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:20:56.158912 #2179:475d538]  INFO -- : Q-task_id([service_template_transformation_plan_task_282]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:50:23.023113 #2179:3ad4960]  INFO -- : Q-task_id([service_template_transformation_plan_task_283]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:16:48.977591 #2179:45171fc]  INFO -- : Q-task_id([service_template_transformation_plan_task_284]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:17:10.107579 #56714:56dacf4]  INFO -- : Q-task_id([service_template_transformation_plan_task_285]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:50:53.983058 #56714:5463084]  INFO -- : Q-task_id([service_template_transformation_plan_task_286]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.

Comment 6 Mor 2018-08-08 15:00:15 UTC

Created attachment 1474374 [details]
evm and automation logs and time of wrapper calling virt-v2v tool

Also included a CSV file with running times of virt-v2v tool on the conversion hosts, taken from the wrapper logs.

Comment 7 Brett Thurber 2018-08-09 03:39:34 UTC

@Fabien, any updates on this?

Comment 8 Brett Thurber 2018-08-10 04:39:52 UTC

PR https://github.com/ManageIQ/manageiq-content/pull/395

Comment 9 Mor 2018-08-20 05:48:29 UTC

Seems to be working as expected on:
CFME 5.9.4.4
RHV 4.2.5.3-0.1.el7ev
when running a plan with 40 VMs on 4 migration hosts.

Before verifying - I have other two options that I didn't checked yet, @Fabien, maybe you can answer this, and I wonder if those be included in this bug, or in a new RFE/bug.

1. Does the logic takes into consideration CPU or memory usage of the target conversion hosts?

2. I know this one is quite complex and depends on various things, but, does the logic takes into consideration VM disk size? (which can affect conversion time). For example: if you want to convert 4 VMs of 10GB and 1 VM of 1TB, it makes sense to run 3 on one host, and 1 on the second host, assuming both hosts have more or less the same specs and network bandwidth.

Comment 10 Fabien Dupont 2018-08-20 06:13:02 UTC

@Mor, I don't think we should add these to this bug, as the behavior is as expected. The current balancing algorithm is pretty naive, but has proven sufficient in previous migration projects. More advanced use cases are under design at the moment, and the main item we consider is I/O and network bandwidth, both probably addressed using cgroups. Checking memory and CPU is a good idea too and we will consider it.

Comment 11 Mor 2018-08-20 06:14:57 UTC

Verified:
CFME 5.9.4.4
RHV 4.2.5.3-0.1.el7ev

Note You need to log in before you can comment on or make changes to this bug.