The error seems to be when adding the SSH private key to ssh-agent. What are the privileges on the /var/lib/vdsm/.ssh directory on the conversion host ?

[root@cfme-smicro-628-b11 ~]# ls -la /var/lib/vdsm
drwx------.  2 vdsm kvm    39 May  9 03:57 .ssh


[root@cfme-smicro-628-b11 ~]# ls -la /var/lib/vdsm/.ssh
total 8
drwx------.  2 vdsm kvm    39 May  9 03:57 .
drwxr-xr-x. 12 vdsm kvm   163 May  6 03:08 ..
-rw-------.  1 vdsm root 1823 May  6 03:08 id_rsa
-rw-r--r--.  1 vdsm kvm  1253 May  9 04:00 known_hosts

@Fabien, please confirm if it is not correct?

Actually the issue is that it tries to use keys from ~/.ssh which is wrong. This happens when you forget to pass the key (via 'ssh_key') in input JSON.

If I understood comment #4 correctly, we have passed key in 'VMware hypervisors SSH private key' section though. Tomáš/Fabien can you confirm if it is working on devel setup?

However, I see there is a real bug in the wrapper that it does not run the agent as 'vdsm' user anymore. It runs as root and I guess /root/.ssh does not exist in your setup. I see you have a separate key in /var/lib/vdsm/.ssh which is fine but I assumed that it is not possible to run conversion without a custom key in CFME anymore.

> Actually the issue is that it tries to use keys from ~/.ssh which is wrong. This happens when you forget to pass the key (via 'ssh_key') in input JSON.

I was told we had switched from using 'ssh_key' to using 'conversion_host_ssh_private_key' and 'vmware_ssh_private_key' properties, which is what the UI passes. Is that not what this code is handling? https://github.com/ManageIQ/manageiq/blob/32fa0345ed8822482d3e6a1e5aa03dfcf4dacb24/app/models/conversion_host/configurations.rb#L69

Yes, I confirmed that with Fabien. So you are indeed facing the bug in wrapper.

Fixed upstream by:

https://github.com/oVirt/v2v-conversion-host/commit/778d63b7bf2db9b9a5e9b4e5317aa85f8b795c5e

Created attachment 1566796 [details]
wrapper logs

Failing in 5.10.4.3.20190509221642_282f74e.
Logs are attached

How is the new log different from the previous log? Does the new run include patch in comment #9 and you're implying the fix does not work?

Hello Tomas, 
No Not tested with patch. No difference in logs , error is same.
Just that we got a new build and I didn't want to open a new BZ on same issue so commented the build number on which it is failing .
Will test it again once the BZ comes to ON_QA with fix .

Created attachment 1568084 [details]
Logs after applying patch

Hello Tomas , Brett , 

After applying patch too migration failed .
I replaced the virt-v2v-wrapper.py file with new code and restarted migration.
Logs are attached.
Appliance : https://10.8.196.131/
Let me know if any help needed in debugging.

This seems like completely different problem. What I see in the automation.log is:

    [----] E, [2019-05-13T15:49:07.742414 #53370:5f79e8c] ERROR -- : MiqAeServiceModelBase.ar_method raised: <RuntimeError>: <Starting conversion failed on 'env-rhv43-01.cfme2.lab.eng.rdu2.redhat.com' with [JSON::ParserError: 765: unexpected token at '']>

It seems the conversion was not even started. Is your environment OK?

FYI I did a new build with the patch from comment #9 applied: v2v-conversion-host-1.13.1-1.el7ev

I'm confused. Isn't the patch supposed to be applied on the conversion host by installing the new virt-v2v-wrapper.py ?
@shveta, what patch did you apply on your appliance ?

So on Rhev conversion host (env-rhv43-01.cfme2.lab.eng.rdu2.redhat.com)I replaced the virt-v2v-wrapper.py with new code as in the commit and started migration .

Conversion Host is there :

irb(main):001:0> ConversionHost.all
PostgreSQLAdapter#log_after_checkout, connection_pool: size: 5, connections: 1, in use: 1, waiting_in_queue: 0
=> #<ActiveRecord::Relation [#<ConversionHost id: 25, name: "env-rhv43-01.cfme2.lab.eng.rdu2.redhat.com", address: nil, type: nil, resource_type: "Host", resource_id: 3, version: nil, max_concurrent_tasks: nil, vddk_transport_supported: nil, ssh_transport_supported: true, created_at: "2019-05-13 18:35:44", updated_at: "2019-05-13 18:36:01", concurrent_transformation_limit: nil, cpu_limit: nil, memory_limit: nil, network_limit: nil, blockio_limit: nil>]>


If there is a different way to apply patch please let me know .

(In reply to Shveta from comment #20)

> If there is a different way to apply patch please let me know .

You forgot to set executable bit on the new file. I guess that's the source of the problem. I fixed that on `env-rhv43-01....` please retest.

Created attachment 1568512 [details]
second_automation.log

We have applied patches to cloudforms (https://bugzilla.redhat.com/show_bug.cgi?id=1708739#c3) and added conversion host with SSH transformation method.
But when we tried end to end migration it failed with similar ssh-agent error in wrapper log, this time it failed on line 1734 of /usr/bin/virt-v2v-wrapper.py.

Tested with
CFME 5.10.4.3.20190509221642_282f74e 
RHV 4.3.3.3 with conversion package 1.12.1-5
Used following HTTP request to add conversion host via CFME with SSH.

```
{
  "name": "reducted-hostname.com",
  "resource_type": "ManageIQ::Providers::Redhat::InfraManager::Host",
  "resource_id": "1",
  "vmware_ssh_private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\n-reducted-key\n-----END OPENSSH PRIVATE KEY-----\n",
  "auth_user": "root"
}
```

Marking failedQA since it did not worked successfully.

Created attachment 1568513 [details]
second_wrapper.log

The log says:

    2019-05-14 11:16:29,674:ERROR: ssh-add output: Enter passphrase for /var/lib/vdsm/.ssh/id_rsa:  (virt-v2v-wrapper:1932)

It looks like the SSH key you have created requires passphrase. This is not supported. Use a key without passphrase, please.

Changing this back to assigned as their is no build with the fix yet to test .
Also the patch provided did not work. I created a key without passphrase and restarted migration but it failed. 
Tomas is investigating.

Please test from  env-rhv43-02.cfme2.lab.eng.rdu2.redhat.com.The Key is copied to it.
and this is the vmware host smicro-5037-07.cfme.lab.eng.rdu2.redhat.com.

reverting back to urgent, sorry it was due to collision I guess.

The SSH key is not properly configured because of bug 1710448 in CFME. The key is striped after first space which makes it invalid.

Can we still validate the fix on the wrapper by setting the key manually on the conversion host ? This would avoid waiting on bug 1710448.

Log after making the above change in conversion_host.rb
http://pastebin.test.redhat.com/763433

Good news! SSH for RHV worked after applying patch from BZ1710448. Fabien also confirmed it on .106.
We are still confirming OSP + SSH usecase with 1.31.1-1 conversion host package.

Created attachment 1569530 [details]
rhv_SSH_5.10.4.3_patches.png

Created attachment 1570025 [details]
osp_ssh_works.png

@Shveta confirmed that migration works with OSP + SSH after applying BZ1710448's patch on cloudforms.
So overall, RHV and OSP with SSH fully functional with Tomáš's patch from Comment #9.

Many thanks to @Fabien, @Tomáš, @Daniel, @MikeT.

*** Bug 1708802 has been marked as a duplicate of this bug. ***

Please move the bug to VERIFIED if things are working with v2v-conversion-host-1.13.1-1.el7ev

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1559