Bug 1574438

Summary: [upstream][v2v] migration from VMware to RHV fails
Product: Red Hat CloudForms Management Engine Reporter: Mor <mkalfon>
Component: UI - OPSAssignee: Brett Thurber <bthurber>
Status: CLOSED DEFERRED QA Contact: Mor <mkalfon>
Severity: high Docs Contact:
Priority: urgent    
Version: 5.4.0CC: bthurber, cpelland, dagur, hkataria, jprause, kkulkarn, lavenel, michal.skrivanek, mkalfon, mpovolny, obarenbo, rjones, smallamp
Target Milestone: GAKeywords: TestBlocker, TestOnly, ZStream
Target Release: 5.10.0   
Hardware: x86_64   
OS: Linux   
Whiteboard: v2v
Fixed In Version: 5.10.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1588576 (view as bug list) Environment:
Last Closed: 2018-06-21 02:35:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Openstack Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1491182, 1588576    
Attachments:
Description Flags
automation and import logs
none
evm and automation logs none

Description Mor 2018-05-03 10:40:11 UTC
Description of problem:
Migration of VM from VMware VM to RHV using the setup and migration steps described in the link [1] fails to work. 

Migration task fails with 'VM Transformations failed'. In addition, the percentage in the progress bar is not updated in the UI, and stays at 0%.

Automation.log logs multiple errors see [2].

virt-v2v logs [3] on the conversion host keeps filling up (indicating a progress of v2v process) while MIQ reports a task failure.

Source environment is VMware vSphere v6.5, source VM is in power-off state. Target environment is RHV 4.2.3 with currently one selected RHV host as migration host. Both environments makes use of iSCSI data-stores.
 
Version-Release number of selected component (if applicable):
RHV 4.2.3.2-0.1.el7
VMware vSphere 6.5

How reproducible:
100%

Steps to Reproduce:
From compute->migration:
1. Create infrastructure mapping between VMware to RHV:
Select source/target clusters, source/target data-store, source/target networks.
2. Create a local CSV file with the following columns: <vm_name>, <migration_plan_name>, <id>
3. Run migration plan, choose the infrastructure mapping from step 1 and CSV from step 2.

Actual results:
1. 'VM Transformations failed'. 
2. Progress is not updated in the UI and stays at 0%.

Expected results:
Should succeed, and progress bar should be updated. 

Additional info:
[1] - https://github.com/RedHatDemos/RHS-Optimize_IT-Infrastructure_Migration/blob/master/docs/00-redhat_solutions-insfrastructure_migration_v2-deployment.adoc

[2] Automation.log
[----] E, [2018-05-03T12:06:21.918450 #36140:cb24628] ERROR -- : <AEMethod vmchecktransformed_vmwarews2rhevm_vddk> The following error occurred during method evaluation:
[----] E, [2018-05-03T12:06:21.920016 #36140:cb24628] ERROR -- : <AEMethod vmchecktransformed_vmwarews2rhevm_vddk>   NoMethodError: undefined method `each' for nil:NilClass
[----] E, [2018-05-03T12:06:21.925603 #36140:cb24628] ERROR -- : <AEMethod vmchecktransformed_vmwarews2rhevm_vddk>   /V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwarews2rhevm_vddk:40:i
n `main'
[----] E, [2018-05-03T12:06:21.937422 #36140:cb24628] ERROR -- : Method STDERR: /V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwarews2rhevm_vddk:40:in `main': undefined method `each' for
 nil:NilClass (NoMethodError)
[----] E, [2018-05-03T12:06:21.938896 #36140:cb24628] ERROR -- : Method STDERR:         from /V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwarews2rhevm_vddk:86:in `<main>'
[----] I, [2018-05-03T12:06:22.015798 #36140:b1114c]  INFO -- : Q-task_id([r16_service_template_transformation_plan_task_16]) <AEMethod [/V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwa
rews2rhevm_vddk]> Ending
[----] E, [2018-05-03T12:06:22.016976 #36140:b1114c] ERROR -- : Q-task_id([r16_service_template_transformation_plan_task_16]) Aborting instantiation (unknown method return code) because [Method exited with rc=Un
known RC: [1]]
[----] E, [2018-05-03T12:06:22.018069 #36140:b1114c] ERROR -- : Q-task_id([r16_service_template_transformation_plan_task_16]) State=<State2> running  raised exception: <Method exited with rc=Unknown RC: [1]>

[3] v2v-import-date-number-wrapper.log
--- tailed log lines --
2018-05-03 09:58:56,283:DEBUG: Updated progress: 100.00 (virt-v2v-wrapper:142)
2018-05-03 10:25:28,154:INFO: virt-v2v terminated with return code 0 (virt-v2v-wrapper:323)
2018-05-03 10:25:28,158:INFO: Removing password files (virt-v2v-wrapper:460)
2018-05-03 10:25:28,159:INFO: Finished (virt-v2v-wrapper:479)

Comment 2 Mor 2018-05-03 10:50:51 UTC
MIQ version: master.20180417121918_2dd64f8.

Comment 5 Richard W.M. Jones 2018-05-03 14:45:26 UTC
I see two v2v logs:

v2v-import-20180430T085329-43940.log
v2v-import-20180503T090617-41661.log

Both are successful conversions.

Comment 6 Sudhir Mallamprabhakara 2018-05-03 14:48:04 UTC
E-mail conversation from KK:

Hey Mor,

Thanks for filing the BZ. I would like to request you to provide me access to your environment . And I would also like to request you to try migration with latest nightly appliance as there have been a lot of changes in the latest nightly. 

Please let me know the info of your appliance for further investigation.


Thanks,
KK.

Comment 7 Mor 2018-05-03 15:16:27 UTC
(In reply to Richard W.M. Jones from comment #5)
> I see two v2v logs:
> 
> v2v-import-20180430T085329-43940.log
> v2v-import-20180503T090617-41661.log
> 
> Both are successful conversions.

The logs shows success, but no VM is created on RHV, and MIQ reports migration failure.

Comment 8 Mor 2018-05-03 16:21:38 UTC
Quick update on this ticket:

I had a great session with KK regarding the v2v workflow, and the new nightly version which contains significant improvements. It has decided not to waste time on my current version, so I will try upgrade my environment to the latest nightly MIQ next week and update you with results.

Meanwhile - lowering the severity to medium.

Comment 9 Mor 2018-05-07 08:19:09 UTC
Update on latest versions:
RHV 4.2.3.5-0.1.el7
MIQ master.20180504232106_a937a31 

I tried to run the migration plan again, to migrate a single VM, but I still get the error: "VM Transformations failed".

I'm changing the severity back to HIGH and attaching new logs.

Comment 10 Mor 2018-05-07 08:24:31 UTC
Created attachment 1432525 [details]
automation and import logs

Comment 11 Michal Skrivanek 2018-05-07 09:24:30 UTC
this is still using -o rhv, not the new rhv-upload. What are you testing?

Comment 17 Daniel Gur 2018-05-10 07:33:58 UTC
relevant bug to this one
Bug 1574734 - raw + sparse isn't working with iSCSI storage domain for upload (edit)  https://bugzilla.redhat.com/show_bug.cgi?id=1574734

Comment 18 Richard W.M. Jones 2018-05-15 07:56:53 UTC
https://people.redhat.com/~rjones/virt-v2v-RHEL-7.5-rhv-preview/

virt-v2v >= -6.10.rhvpreview from the repository above removes the
restriction on sparse.  You can now use ‘-oa preallocated’ on the
command line which should work for iSCSI & FC transfers.

It doesn't (because it cannot) solve the problem that different
storage domains have unknowable requirements so you have to somehow
just know whether or not to pass ‘-oa preallocated’.

Comment 19 Mor 2018-05-15 08:15:04 UTC
(In reply to Richard W.M. Jones from comment #18)
> https://people.redhat.com/~rjones/virt-v2v-RHEL-7.5-rhv-preview/
> 
> virt-v2v >= -6.10.rhvpreview from the repository above removes the
> restriction on sparse.  You can now use ‘-oa preallocated’ on the
> command line which should work for iSCSI & FC transfers.
> 
> It doesn't (because it cannot) solve the problem that different
> storage domains have unknowable requirements so you have to somehow
> just know whether or not to pass ‘-oa preallocated’.

Richard, the workaround works manually from the command line, not if triggered virt-v2v >= -6.10.rhvpreview from MIQ, right?

Comment 20 Richard W.M. Jones 2018-05-15 08:37:04 UTC
I believe that is correct, yes.

Comment 31 Brett Thurber 2018-05-29 16:10:32 UTC
Testing last week was successful in migrating to iSCSI storage domains using -oa preallocated with RHV compose.

Comment 37 Mor 2018-06-18 20:54:46 UTC
I'm failing this bug on CFME version 5.10.0.0.20180613200131_887cc81

When trying to migrate a VM from vSphere 6.5 I get "VM Transformations failed" with the following error:

[----] E, [2018-06-18T23:34:23.638528 #11725:aaf5820] ERROR -- : <AEMethod vmtransform_vmwarews2rhevm_vddk> The following error occurred during method evaluation:
[----] E, [2018-06-18T23:34:23.640039 #11725:aaf5820] ERROR -- : <AEMethod vmtransform_vmwarews2rhevm_vddk>   Errno::ENOTTY: Inappropriate ioctl for device
[----] E, [2018-06-18T23:34:23.644992 #11725:aaf5820] ERROR -- : <AEMethod vmtransform_vmwarews2rhevm_vddk>   /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/prompt.rb:44:in `noecho'
[----] E, [2018-06-18T23:34:23.686174 #11725:aaf5820] ERROR -- : Method STDERR: /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/prompt.rb:44:in `noecho': Inappropriate ioctl for device (Errno::ENOTTY)
[----] E, [2018-06-18T23:34:23.690540 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/prompt.rb:44:in `ask'
[----] E, [2018-06-18T23:34:23.692198 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/password.rb:68:in `ask_password'
[----] E, [2018-06-18T23:34:23.693836 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/password.rb:21:in `authenticate'
[----] E, [2018-06-18T23:34:23.695531 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:80:in `block in authenticate'
[----] E, [2018-06-18T23:34:23.697077 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `each'
[----] E, [2018-06-18T23:34:23.698736 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `authenticate'
[----] E, [2018-06-18T23:34:23.700160 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh.rb:241:in `start'
[----] E, [2018-06-18T23:34:23.701780 #11725:aaf5820] ERROR -- : Method STDERR:         from ManageIQ/Transformation/TransformationHosts/ovirt_host/Utils:15:in `remote_command'
[----] E, [2018-06-18T23:34:23.703321 #11725:aaf5820] ERROR -- : Method STDERR:         from /ManageIQ/Transformation/TransformationHosts/ovirt_host/VMTransform_vmwarews2rhevm_vddk:74:in `main'
[----] E, [2018-06-18T23:34:23.704753 #11725:aaf5820] ERROR -- : Method STDERR:         from /ManageIQ/Transformation/TransformationHosts/ovirt_host/VMTransform_vmwarews2rhevm_vddk:103:in `<main>'
[----] E, [2018-06-18T23:34:23.854573 #11725:128ef78] ERROR -- : Q-task_id([r1000000000010_service_template_transformation_plan_task_1000000000022]) Aborting instantiation (unknown method return code) because [Method exited with rc=Unknown RC: [1]]
[----] E, [2018-06-18T23:34:23.855223 #11725:128ef78] ERROR -- : Q-task_id([r1000000000010_service_template_transformation_plan_task_1000000000022]) State=<State1> running  raised exception: <Method exited with rc=Unknown RC: [1]>

Logs will be attached.

Comment 39 Mor 2018-06-19 08:28:35 UTC
There are no import logs generated. It fails when executing the virt-v2v wrapper. 

PLEASE NOTE: 
evm.log contains another unrelated keystone error. It's on a totally different RHV/VMWare registered environment. We are still investigating it. If you aware of this issue, please let us know.

[----] E, [2018-06-19T11:00:31.733875 #23546:358f7c] ERROR -- : MIQ(ManageIQ::Providers::Redhat::NetworkManager#verify_api_credentials) Error Class=Excon::Error::Unauthorized, Message=Expected([200, 204]) <=> Actual(401 Unauthorized)
excon.error.response
  :body          => "{\n  \"error\": {\n    \"message\": \"Error during SSO authentication invalid_request : Missing parameter: 'client_secret'\",\n    \"code\": 401,\n    \"title\": \"Unauthorized\"\n  }\n}\n"
  :cookies       => [
  ]
  :headers       => {
    "Connection"   => "close"
    "Content-Type" => "application/json"
    "Date"         => "Tue, 19 Jun 2018 08:00:31 GMT"
    "Server"       => "BaseHTTP/0.3 Python/2.7.5"
  }
  :host          => "nsimsolo41.scl.lab.tlv.redhat.com"
  :local_address => "10.12.69.26"
  :local_port    => 35058
  :path          => "/v2.0/tokens"
  :port          => 35357
  :reason_phrase => "Unauthorized"
  :remote_ip     => "10.35.161.173"
  :status        => 401
  :status_line   => "HTTP/1.0 401 Unauthorized\r\n"

[----] W, [2018-06-19T11:00:31.734057 #23546:358f7c]  WARN -- : MIQ(ManageIQ::Providers::Redhat::NetworkManager::EventCatcher::Runner#start_event_monitor) EMS [nsimsolo41.scl.lab.tlv.redhat.com] as [admin@internal] Login failed due to a bad username or password.

Comment 40 Mor 2018-06-19 08:29:52 UTC
Created attachment 1452842 [details]
evm and automation logs

Comment 44 Mor 2018-06-21 02:35:56 UTC
MIQ nightly: master.20180619230249_84e9fa9 found to be working. I will downgrade the CFME from 5.10 to 5.9 and reopen the bug in case it fails.