Bug 1610054

Summary:

[v2v] Migration balance is not working as expected

Product:

Red Hat CloudForms Management Engine

Reporter:

Satoe Imaishi <simaishi>

Component:

UI - OPS

Assignee:

Fabien Dupont <fdupont>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Jad Haj Yahya <jhajyahy>

Severity:

urgent

Docs Contact:

Priority:

high

Version:

5.9.0

CC:

bperkins, bthurber, cpelland, dagur, fdupont, hkataria, jprause, kkulkarn, lavenel, mlehrer, mpovolny, obarenbo, smallamp

Target Milestone:

Keywords:

AutomationBlocker, TestBlocker, ZStream

Target Release:

5.9.4

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

v2v

Fixed In Version:

5.9.4.4

Doc Type:

If docs needed, set a value

Doc Text:

Story Points:

---

Clone Of:

1600152

Environment:

Last Closed:

2018-09-05 12:56:29 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

CFME Core

Target Upstream Version:

Embargoed:

Bug Depends On:

1600152

Bug Blocks:

Attachments:

Description	Flags
evm and automation logs and time of wrapper calling virt-v2v tool	none

Comment 3 CFME Bot 2018-07-31 01:01:48 UTC

New commit detected on ManageIQ/manageiq-content/gaprindashvili:

https://github.com/ManageIQ/manageiq-content/commit/99fff66d837394b3eee4bdf639c0bb44b71a7464
commit 99fff66d837394b3eee4bdf639c0bb44b71a7464
Author:     Greg McCullough <gmccullo>
AuthorDate: Mon Jul 30 13:41:19 2018 -0400
Commit:     Greg McCullough <gmccullo>
CommitDate: Mon Jul 30 13:41:19 2018 -0400

    Merge pull request #379 from fdupont-redhat/v2v_fix_transformation_host_selection

    Fix transformation host selection
    (cherry picked from commit b2982c6bc856b5b6ad4af762dfa028cf9df9bb11)

    https://bugzilla.redhat.com/show_bug.cgi?id=1610054

 content/automate/ManageIQ/Transformation/TransformationHosts/Common.class/__methods__/utils.rb | 44 +-
 content/automate/ManageIQ/Transformation/TransformationHosts/Common.class/__methods__/vmtransform.rb | 2 +-
 2 files changed, 31 insertions(+), 15 deletions(-)

Comment 5 Mor 2018-08-08 14:54:38 UTC

I'm failing it on the following environment:
CFME 5.9.4.2
RHV 4.2.5.2-0.1.el7ev

Fabien also did a check on the evm.log file, and he found that the distribution went wrong on the run. 95% of the task end up on the same host: b02-h23-r620.rhev.openstack.engineering.redhat.com, while they should be equally split between the two hosts:
$ grep service_template_transformation_plan_task_ automation.log | awk '{ print $8; }' | sort -u | grep -v automation | cut -d "[" -f2 | cut -d "]" -f1 | while read task ; do grep "${task}" automation.log | grep 'acquiretransformationhost> Transformation Host: ' ; done
[----] I, [2018-08-07T16:07:41.589414 #2187:4fef494]  INFO -- : Q-task_id([service_template_transformation_plan_task_187]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:07:40.687186 #2179:32f32dc]  INFO -- : Q-task_id([service_template_transformation_plan_task_188]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:09.753008 #2179:45a6410]  INFO -- : Q-task_id([service_template_transformation_plan_task_189]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:14.161519 #2187:13d20c4]  INFO -- : Q-task_id([service_template_transformation_plan_task_190]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:36.205294 #2179:4de9bb8]  INFO -- : Q-task_id([service_template_transformation_plan_task_191]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:08:41.128391 #2187:451bd60]  INFO -- : Q-task_id([service_template_transformation_plan_task_192]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:03.026338 #2179:448f630]  INFO -- : Q-task_id([service_template_transformation_plan_task_193]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:07.558536 #2187:4fc679c]  INFO -- : Q-task_id([service_template_transformation_plan_task_194]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:29.818315 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_195]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:34.514493 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_196]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:09:56.853518 #2179:106ea2c]  INFO -- : Q-task_id([service_template_transformation_plan_task_197]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:01.765281 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_198]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:23.999313 #2179:106ea2c]  INFO -- : Q-task_id([service_template_transformation_plan_task_199]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:28.929544 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_200]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:50.888203 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_201]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:10:55.828331 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_202]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:17.774265 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_203]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:22.857803 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_204]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:44.905029 #2179:106eb1c]  INFO -- : Q-task_id([service_template_transformation_plan_task_205]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:11:50.065942 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_206]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:28:41.082774 #2187:507600c]  INFO -- : Q-task_id([service_template_transformation_plan_task_227]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:28:41.816563 #2179:5680a74]  INFO -- : Q-task_id([service_template_transformation_plan_task_228]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:09.849669 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_229]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:14.911800 #2179:106efcc]  INFO -- : Q-task_id([service_template_transformation_plan_task_230]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:36.875056 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_231]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:29:42.857836 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_232]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:05.631546 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_233]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:11.390289 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_234]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:33.515944 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_235]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:30:39.461398 #2179:106ee8c]  INFO -- : Q-task_id([service_template_transformation_plan_task_236]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:01.534685 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_237]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:07.317003 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_238]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:29.746206 #2187:10712e0]  INFO -- : Q-task_id([service_template_transformation_plan_task_239]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:35.981348 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_240]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:31:58.043836 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_241]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:03.995541 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_242]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:25.947143 #2187:10713f8]  INFO -- : Q-task_id([service_template_transformation_plan_task_243]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:32.062706 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_244]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:53.725072 #2187:454e97c]  INFO -- : Q-task_id([service_template_transformation_plan_task_245]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:32:59.953511 #2179:106ee8c]  INFO -- : Q-task_id([service_template_transformation_plan_task_246]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:40:41.989470 #2187:4fd4a18]  INFO -- : Q-task_id([service_template_transformation_plan_task_267]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:40:42.402620 #2179:5ca5ddc]  INFO -- : Q-task_id([service_template_transformation_plan_task_268]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:15.132479 #2187:421a56c]  INFO -- : Q-task_id([service_template_transformation_plan_task_269]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:15.632362 #2179:4fa018c]  INFO -- : Q-task_id([service_template_transformation_plan_task_270]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:48.415274 #2187:3b0c568]  INFO -- : Q-task_id([service_template_transformation_plan_task_271]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:41:48.777358 #2179:4f95958]  INFO -- : Q-task_id([service_template_transformation_plan_task_272]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:21.237464 #2179:106efe0]  INFO -- : Q-task_id([service_template_transformation_plan_task_273]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:21.752712 #2187:45cdb00]  INFO -- : Q-task_id([service_template_transformation_plan_task_274]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:54.182562 #2179:509d224]  INFO -- : Q-task_id([service_template_transformation_plan_task_275]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T16:42:54.918697 #2187:4503d8c]  INFO -- : Q-task_id([service_template_transformation_plan_task_276]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:18:37.563996 #2179:59e1718]  INFO -- : Q-task_id([service_template_transformation_plan_task_277]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:18:05.650024 #2179:5beed80]  INFO -- : Q-task_id([service_template_transformation_plan_task_278]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:20:28.677262 #56714:5e7b6d4]  INFO -- : Q-task_id([service_template_transformation_plan_task_279]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:15:47.404147 #2187:5befb40]  INFO -- : Q-task_id([service_template_transformation_plan_task_280]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:18:54.840479 #56714:5704c5c]  INFO -- : Q-task_id([service_template_transformation_plan_task_281]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:20:56.158912 #2179:475d538]  INFO -- : Q-task_id([service_template_transformation_plan_task_282]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:50:23.023113 #2179:3ad4960]  INFO -- : Q-task_id([service_template_transformation_plan_task_283]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:16:48.977591 #2179:45171fc]  INFO -- : Q-task_id([service_template_transformation_plan_task_284]) <AEMethod acquiretransformationhost> Transformation Host: b02-h23-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:17:10.107579 #56714:56dacf4]  INFO -- : Q-task_id([service_template_transformation_plan_task_285]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.
[----] I, [2018-08-07T17:50:53.983058 #56714:5463084]  INFO -- : Q-task_id([service_template_transformation_plan_task_286]) <AEMethod acquiretransformationhost> Transformation Host: b01-h17-r620.rhev.openstack.engineering.redhat.com.

Comment 6 Mor 2018-08-08 15:00:15 UTC

Created attachment 1474374 [details]
evm and automation logs and time of wrapper calling virt-v2v tool

Also included a CSV file with running times of virt-v2v tool on the conversion hosts, taken from the wrapper logs.

Comment 7 Brett Thurber 2018-08-09 03:39:34 UTC

@Fabien, any updates on this?

Comment 8 Brett Thurber 2018-08-10 04:39:52 UTC

PR https://github.com/ManageIQ/manageiq-content/pull/395

Comment 9 Mor 2018-08-20 05:48:29 UTC

Seems to be working as expected on:
CFME 5.9.4.4
RHV 4.2.5.3-0.1.el7ev
when running a plan with 40 VMs on 4 migration hosts.

Before verifying - I have other two options that I didn't checked yet, @Fabien, maybe you can answer this, and I wonder if those be included in this bug, or in a new RFE/bug.

1. Does the logic takes into consideration CPU or memory usage of the target conversion hosts?

2. I know this one is quite complex and depends on various things, but, does the logic takes into consideration VM disk size? (which can affect conversion time). For example: if you want to convert 4 VMs of 10GB and 1 VM of 1TB, it makes sense to run 3 on one host, and 1 on the second host, assuming both hosts have more or less the same specs and network bandwidth.

Comment 10 Fabien Dupont 2018-08-20 06:13:02 UTC

@Mor, I don't think we should add these to this bug, as the behavior is as expected. The current balancing algorithm is pretty naive, but has proven sufficient in previous migration projects. More advanced use cases are under design at the moment, and the main item we consider is I/O and network bandwidth, both probably addressed using cgroups. Checking memory and CPU is a good idea too and we will consider it.

Comment 11 Mor 2018-08-20 06:14:57 UTC

Verified:
CFME 5.9.4.4
RHV 4.2.5.3-0.1.el7ev