Bug 1574438 - [upstream][v2v] migration from VMware to RHV fails
Summary: [upstream][v2v] migration from VMware to RHV fails
Keywords:
Status: CLOSED DEFERRED
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: UI - OPS
Version: 5.4.0
Hardware: x86_64
OS: Linux
urgent
high
Target Milestone: GA
: 5.10.0
Assignee: Brett Thurber
QA Contact: Mor
URL:
Whiteboard: v2v
Depends On:
Blocks: 1491182 1588576
TreeView+ depends on / blocked
 
Reported: 2018-05-03 10:40 UTC by Mor
Modified: 2019-02-26 09:28 UTC (History)
13 users (show)

Fixed In Version: 5.10.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1588576 (view as bug list)
Environment:
Last Closed: 2018-06-21 02:35:56 UTC
Category: ---
Cloudforms Team: Openstack
Target Upstream Version:


Attachments (Terms of Use)
automation and import logs (3.12 MB, application/zip)
2018-05-07 08:24 UTC, Mor
no flags Details
evm and automation logs (190.03 KB, application/x-bzip)
2018-06-19 08:29 UTC, Mor
no flags Details

Description Mor 2018-05-03 10:40:11 UTC
Description of problem:
Migration of VM from VMware VM to RHV using the setup and migration steps described in the link [1] fails to work. 

Migration task fails with 'VM Transformations failed'. In addition, the percentage in the progress bar is not updated in the UI, and stays at 0%.

Automation.log logs multiple errors see [2].

virt-v2v logs [3] on the conversion host keeps filling up (indicating a progress of v2v process) while MIQ reports a task failure.

Source environment is VMware vSphere v6.5, source VM is in power-off state. Target environment is RHV 4.2.3 with currently one selected RHV host as migration host. Both environments makes use of iSCSI data-stores.
 
Version-Release number of selected component (if applicable):
RHV 4.2.3.2-0.1.el7
VMware vSphere 6.5

How reproducible:
100%

Steps to Reproduce:
From compute->migration:
1. Create infrastructure mapping between VMware to RHV:
Select source/target clusters, source/target data-store, source/target networks.
2. Create a local CSV file with the following columns: <vm_name>, <migration_plan_name>, <id>
3. Run migration plan, choose the infrastructure mapping from step 1 and CSV from step 2.

Actual results:
1. 'VM Transformations failed'. 
2. Progress is not updated in the UI and stays at 0%.

Expected results:
Should succeed, and progress bar should be updated. 

Additional info:
[1] - https://github.com/RedHatDemos/RHS-Optimize_IT-Infrastructure_Migration/blob/master/docs/00-redhat_solutions-insfrastructure_migration_v2-deployment.adoc

[2] Automation.log
[----] E, [2018-05-03T12:06:21.918450 #36140:cb24628] ERROR -- : <AEMethod vmchecktransformed_vmwarews2rhevm_vddk> The following error occurred during method evaluation:
[----] E, [2018-05-03T12:06:21.920016 #36140:cb24628] ERROR -- : <AEMethod vmchecktransformed_vmwarews2rhevm_vddk>   NoMethodError: undefined method `each' for nil:NilClass
[----] E, [2018-05-03T12:06:21.925603 #36140:cb24628] ERROR -- : <AEMethod vmchecktransformed_vmwarews2rhevm_vddk>   /V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwarews2rhevm_vddk:40:i
n `main'
[----] E, [2018-05-03T12:06:21.937422 #36140:cb24628] ERROR -- : Method STDERR: /V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwarews2rhevm_vddk:40:in `main': undefined method `each' for
 nil:NilClass (NoMethodError)
[----] E, [2018-05-03T12:06:21.938896 #36140:cb24628] ERROR -- : Method STDERR:         from /V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwarews2rhevm_vddk:86:in `<main>'
[----] I, [2018-05-03T12:06:22.015798 #36140:b1114c]  INFO -- : Q-task_id([r16_service_template_transformation_plan_task_16]) <AEMethod [/V2V/Transformation/TransformationHosts/ovirt_host/VMCheckTransformed_vmwa
rews2rhevm_vddk]> Ending
[----] E, [2018-05-03T12:06:22.016976 #36140:b1114c] ERROR -- : Q-task_id([r16_service_template_transformation_plan_task_16]) Aborting instantiation (unknown method return code) because [Method exited with rc=Un
known RC: [1]]
[----] E, [2018-05-03T12:06:22.018069 #36140:b1114c] ERROR -- : Q-task_id([r16_service_template_transformation_plan_task_16]) State=<State2> running  raised exception: <Method exited with rc=Unknown RC: [1]>

[3] v2v-import-date-number-wrapper.log
--- tailed log lines --
2018-05-03 09:58:56,283:DEBUG: Updated progress: 100.00 (virt-v2v-wrapper:142)
2018-05-03 10:25:28,154:INFO: virt-v2v terminated with return code 0 (virt-v2v-wrapper:323)
2018-05-03 10:25:28,158:INFO: Removing password files (virt-v2v-wrapper:460)
2018-05-03 10:25:28,159:INFO: Finished (virt-v2v-wrapper:479)

Comment 2 Mor 2018-05-03 10:50:51 UTC
MIQ version: master.20180417121918_2dd64f8.

Comment 5 Richard W.M. Jones 2018-05-03 14:45:26 UTC
I see two v2v logs:

v2v-import-20180430T085329-43940.log
v2v-import-20180503T090617-41661.log

Both are successful conversions.

Comment 6 Sudhir Mallamprabhakara 2018-05-03 14:48:04 UTC
E-mail conversation from KK:

Hey Mor,

Thanks for filing the BZ. I would like to request you to provide me access to your environment . And I would also like to request you to try migration with latest nightly appliance as there have been a lot of changes in the latest nightly. 

Please let me know the info of your appliance for further investigation.


Thanks,
KK.

Comment 7 Mor 2018-05-03 15:16:27 UTC
(In reply to Richard W.M. Jones from comment #5)
> I see two v2v logs:
> 
> v2v-import-20180430T085329-43940.log
> v2v-import-20180503T090617-41661.log
> 
> Both are successful conversions.

The logs shows success, but no VM is created on RHV, and MIQ reports migration failure.

Comment 8 Mor 2018-05-03 16:21:38 UTC
Quick update on this ticket:

I had a great session with KK regarding the v2v workflow, and the new nightly version which contains significant improvements. It has decided not to waste time on my current version, so I will try upgrade my environment to the latest nightly MIQ next week and update you with results.

Meanwhile - lowering the severity to medium.

Comment 9 Mor 2018-05-07 08:19:09 UTC
Update on latest versions:
RHV 4.2.3.5-0.1.el7
MIQ master.20180504232106_a937a31 

I tried to run the migration plan again, to migrate a single VM, but I still get the error: "VM Transformations failed".

I'm changing the severity back to HIGH and attaching new logs.

Comment 10 Mor 2018-05-07 08:24:31 UTC
Created attachment 1432525 [details]
automation and import logs

Comment 11 Michal Skrivanek 2018-05-07 09:24:30 UTC
this is still using -o rhv, not the new rhv-upload. What are you testing?

Comment 17 Daniel Gur 2018-05-10 07:33:58 UTC
relevant bug to this one
Bug 1574734 - raw + sparse isn't working with iSCSI storage domain for upload (edit)  https://bugzilla.redhat.com/show_bug.cgi?id=1574734

Comment 18 Richard W.M. Jones 2018-05-15 07:56:53 UTC
https://people.redhat.com/~rjones/virt-v2v-RHEL-7.5-rhv-preview/

virt-v2v >= -6.10.rhvpreview from the repository above removes the
restriction on sparse.  You can now use ‘-oa preallocated’ on the
command line which should work for iSCSI & FC transfers.

It doesn't (because it cannot) solve the problem that different
storage domains have unknowable requirements so you have to somehow
just know whether or not to pass ‘-oa preallocated’.

Comment 19 Mor 2018-05-15 08:15:04 UTC
(In reply to Richard W.M. Jones from comment #18)
> https://people.redhat.com/~rjones/virt-v2v-RHEL-7.5-rhv-preview/
> 
> virt-v2v >= -6.10.rhvpreview from the repository above removes the
> restriction on sparse.  You can now use ‘-oa preallocated’ on the
> command line which should work for iSCSI & FC transfers.
> 
> It doesn't (because it cannot) solve the problem that different
> storage domains have unknowable requirements so you have to somehow
> just know whether or not to pass ‘-oa preallocated’.

Richard, the workaround works manually from the command line, not if triggered virt-v2v >= -6.10.rhvpreview from MIQ, right?

Comment 20 Richard W.M. Jones 2018-05-15 08:37:04 UTC
I believe that is correct, yes.

Comment 31 Brett Thurber 2018-05-29 16:10:32 UTC
Testing last week was successful in migrating to iSCSI storage domains using -oa preallocated with RHV compose.

Comment 37 Mor 2018-06-18 20:54:46 UTC
I'm failing this bug on CFME version 5.10.0.0.20180613200131_887cc81

When trying to migrate a VM from vSphere 6.5 I get "VM Transformations failed" with the following error:

[----] E, [2018-06-18T23:34:23.638528 #11725:aaf5820] ERROR -- : <AEMethod vmtransform_vmwarews2rhevm_vddk> The following error occurred during method evaluation:
[----] E, [2018-06-18T23:34:23.640039 #11725:aaf5820] ERROR -- : <AEMethod vmtransform_vmwarews2rhevm_vddk>   Errno::ENOTTY: Inappropriate ioctl for device
[----] E, [2018-06-18T23:34:23.644992 #11725:aaf5820] ERROR -- : <AEMethod vmtransform_vmwarews2rhevm_vddk>   /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/prompt.rb:44:in `noecho'
[----] E, [2018-06-18T23:34:23.686174 #11725:aaf5820] ERROR -- : Method STDERR: /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/prompt.rb:44:in `noecho': Inappropriate ioctl for device (Errno::ENOTTY)
[----] E, [2018-06-18T23:34:23.690540 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/prompt.rb:44:in `ask'
[----] E, [2018-06-18T23:34:23.692198 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/password.rb:68:in `ask_password'
[----] E, [2018-06-18T23:34:23.693836 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/methods/password.rb:21:in `authenticate'
[----] E, [2018-06-18T23:34:23.695531 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:80:in `block in authenticate'
[----] E, [2018-06-18T23:34:23.697077 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `each'
[----] E, [2018-06-18T23:34:23.698736 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh/authentication/session.rb:66:in `authenticate'
[----] E, [2018-06-18T23:34:23.700160 #11725:aaf5820] ERROR -- : Method STDERR:         from /opt/rh/cfme-gemset/gems/net-ssh-4.2.0/lib/net/ssh.rb:241:in `start'
[----] E, [2018-06-18T23:34:23.701780 #11725:aaf5820] ERROR -- : Method STDERR:         from ManageIQ/Transformation/TransformationHosts/ovirt_host/Utils:15:in `remote_command'
[----] E, [2018-06-18T23:34:23.703321 #11725:aaf5820] ERROR -- : Method STDERR:         from /ManageIQ/Transformation/TransformationHosts/ovirt_host/VMTransform_vmwarews2rhevm_vddk:74:in `main'
[----] E, [2018-06-18T23:34:23.704753 #11725:aaf5820] ERROR -- : Method STDERR:         from /ManageIQ/Transformation/TransformationHosts/ovirt_host/VMTransform_vmwarews2rhevm_vddk:103:in `<main>'
[----] E, [2018-06-18T23:34:23.854573 #11725:128ef78] ERROR -- : Q-task_id([r1000000000010_service_template_transformation_plan_task_1000000000022]) Aborting instantiation (unknown method return code) because [Method exited with rc=Unknown RC: [1]]
[----] E, [2018-06-18T23:34:23.855223 #11725:128ef78] ERROR -- : Q-task_id([r1000000000010_service_template_transformation_plan_task_1000000000022]) State=<State1> running  raised exception: <Method exited with rc=Unknown RC: [1]>

Logs will be attached.

Comment 39 Mor 2018-06-19 08:28:35 UTC
There are no import logs generated. It fails when executing the virt-v2v wrapper. 

PLEASE NOTE: 
evm.log contains another unrelated keystone error. It's on a totally different RHV/VMWare registered environment. We are still investigating it. If you aware of this issue, please let us know.

[----] E, [2018-06-19T11:00:31.733875 #23546:358f7c] ERROR -- : MIQ(ManageIQ::Providers::Redhat::NetworkManager#verify_api_credentials) Error Class=Excon::Error::Unauthorized, Message=Expected([200, 204]) <=> Actual(401 Unauthorized)
excon.error.response
  :body          => "{\n  \"error\": {\n    \"message\": \"Error during SSO authentication invalid_request : Missing parameter: 'client_secret'\",\n    \"code\": 401,\n    \"title\": \"Unauthorized\"\n  }\n}\n"
  :cookies       => [
  ]
  :headers       => {
    "Connection"   => "close"
    "Content-Type" => "application/json"
    "Date"         => "Tue, 19 Jun 2018 08:00:31 GMT"
    "Server"       => "BaseHTTP/0.3 Python/2.7.5"
  }
  :host          => "nsimsolo41.scl.lab.tlv.redhat.com"
  :local_address => "10.12.69.26"
  :local_port    => 35058
  :path          => "/v2.0/tokens"
  :port          => 35357
  :reason_phrase => "Unauthorized"
  :remote_ip     => "10.35.161.173"
  :status        => 401
  :status_line   => "HTTP/1.0 401 Unauthorized\r\n"

[----] W, [2018-06-19T11:00:31.734057 #23546:358f7c]  WARN -- : MIQ(ManageIQ::Providers::Redhat::NetworkManager::EventCatcher::Runner#start_event_monitor) EMS [nsimsolo41.scl.lab.tlv.redhat.com] as [admin@internal] Login failed due to a bad username or password.

Comment 40 Mor 2018-06-19 08:29:52 UTC
Created attachment 1452842 [details]
evm and automation logs

Comment 44 Mor 2018-06-21 02:35:56 UTC
MIQ nightly: master.20180619230249_84e9fa9 found to be working. I will downgrade the CFME from 5.10 to 5.9 and reopen the bug in case it fails.


Note You need to log in before you can comment on or make changes to this bug.