Bug 1707983 - [RHV]SSH : Migration over SSH with RHV fails with error "Failed to add SSH keys to the agent"
Summary: [RHV]SSH : Migration over SSH with RHV fails with error "Failed to add SSH ke...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: v2v-conversion-host
Version: 4.3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ovirt-4.3.4
: ---
Assignee: Tomáš Golembiovský
QA Contact: Shveta
URL:
Whiteboard: v2v
Depends On: 1710448
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-05-08 20:55 UTC by Shveta
Modified: 2019-06-26 08:34 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when converting virtual machines to Red Hat Virtualization using SSH transformation, the virt-v2v wrapper used the root SSH key instead of the VDSM user's SSH key. This bug is now fixed.
Clone Of:
Environment:
Last Closed: 2019-06-20 14:47:53 UTC
oVirt Team: Virt
Target Upstream Version:


Attachments (Terms of Use)
log (6.74 KB, application/octet-stream)
2019-05-08 20:55 UTC, Shveta
no flags Details
wrapper logs (6.56 KB, application/octet-stream)
2019-05-10 20:11 UTC, Shveta
no flags Details
Logs after applying patch (255.81 KB, text/plain)
2019-05-13 19:55 UTC, Shveta
no flags Details
second_automation.log (307.75 KB, text/plain)
2019-05-14 15:19 UTC, Yadnyawalk Tale
no flags Details
second_wrapper.log (6.68 KB, application/octet-stream)
2019-05-14 15:20 UTC, Yadnyawalk Tale
no flags Details
rhv_SSH_5.10.4.3_patches.png (43.65 KB, image/png)
2019-05-16 14:21 UTC, Yadnyawalk Tale
no flags Details
osp_ssh_works.png (120.38 KB, image/png)
2019-05-17 10:44 UTC, Yadnyawalk Tale
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:1559 None None None 2019-06-20 14:47:56 UTC

Description Shveta 2019-05-08 20:55:05 UTC
Created attachment 1565831 [details]
log

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Add RHV and Vmware  and configure conversion host for SSH transformation.
2. Create Infrastructure mapping .
3. Create migration plan .
4. Plan fails with error in logs 

Actual results:


Expected results:


Additional info:
Appliance : https://10.8.196.203
Plan : plan_JG1IjtlVdE

Comment 2 Fabien Dupont 2019-05-09 13:48:01 UTC
The error seems to be when adding the SSH private key to ssh-agent. What are the privileges on the /var/lib/vdsm/.ssh directory on the conversion host ?

Comment 3 Yadnyawalk Tale 2019-05-09 14:44:59 UTC
[root@cfme-smicro-628-b11 ~]# ls -la /var/lib/vdsm
drwx------.  2 vdsm kvm    39 May  9 03:57 .ssh


[root@cfme-smicro-628-b11 ~]# ls -la /var/lib/vdsm/.ssh
total 8
drwx------.  2 vdsm kvm    39 May  9 03:57 .
drwxr-xr-x. 12 vdsm kvm   163 May  6 03:08 ..
-rw-------.  1 vdsm root 1823 May  6 03:08 id_rsa
-rw-r--r--.  1 vdsm kvm  1253 May  9 04:00 known_hosts

@Fabien, please confirm if it is not correct?

Comment 4 Tomáš Golembiovský 2019-05-09 14:47:52 UTC
Actually the issue is that it tries to use keys from ~/.ssh which is wrong. This happens when you forget to pass the key (via 'ssh_key') in input JSON.

Comment 5 Yadnyawalk Tale 2019-05-09 15:08:09 UTC
If I understood comment #4 correctly, we have passed key in 'VMware hypervisors SSH private key' section though. Tomáš/Fabien can you confirm if it is working on devel setup?

Comment 6 Tomáš Golembiovský 2019-05-09 15:25:53 UTC
However, I see there is a real bug in the wrapper that it does not run the agent as 'vdsm' user anymore. It runs as root and I guess /root/.ssh does not exist in your setup. I see you have a separate key in /var/lib/vdsm/.ssh which is fine but I assumed that it is not possible to run conversion without a custom key in CFME anymore.

Comment 7 Mike Turley 2019-05-09 17:49:04 UTC
> Actually the issue is that it tries to use keys from ~/.ssh which is wrong. This happens when you forget to pass the key (via 'ssh_key') in input JSON.

I was told we had switched from using 'ssh_key' to using 'conversion_host_ssh_private_key' and 'vmware_ssh_private_key' properties, which is what the UI passes. Is that not what this code is handling? https://github.com/ManageIQ/manageiq/blob/32fa0345ed8822482d3e6a1e5aa03dfcf4dacb24/app/models/conversion_host/configurations.rb#L69

Comment 8 Tomáš Golembiovský 2019-05-09 19:16:25 UTC
Yes, I confirmed that with Fabien. So you are indeed facing the bug in wrapper.

Comment 9 Tomáš Golembiovský 2019-05-10 13:25:40 UTC
Fixed upstream by:

https://github.com/oVirt/v2v-conversion-host/commit/778d63b7bf2db9b9a5e9b4e5317aa85f8b795c5e

Comment 10 Shveta 2019-05-10 20:11:46 UTC
Created attachment 1566796 [details]
wrapper logs

Failing in 5.10.4.3.20190509221642_282f74e.
Logs are attached

Comment 11 Tomáš Golembiovský 2019-05-13 08:20:24 UTC
How is the new log different from the previous log? Does the new run include patch in comment #9 and you're implying the fix does not work?

Comment 12 Shveta 2019-05-13 17:27:11 UTC
Hello Tomas, 
No Not tested with patch. No difference in logs , error is same.
Just that we got a new build and I didn't want to open a new BZ on same issue so commented the build number on which it is failing .
Will test it again once the BZ comes to ON_QA with fix .

Comment 16 Shveta 2019-05-13 19:55:48 UTC
Created attachment 1568084 [details]
Logs after applying patch

Hello Tomas , Brett , 

After applying patch too migration failed .
I replaced the virt-v2v-wrapper.py file with new code and restarted migration.
Logs are attached.
Appliance : https://10.8.196.131/
Let me know if any help needed in debugging.

Comment 17 Tomáš Golembiovský 2019-05-13 21:36:16 UTC
This seems like completely different problem. What I see in the automation.log is:

    [----] E, [2019-05-13T15:49:07.742414 #53370:5f79e8c] ERROR -- : MiqAeServiceModelBase.ar_method raised: <RuntimeError>: <Starting conversion failed on 'env-rhv43-01.cfme2.lab.eng.rdu2.redhat.com' with [JSON::ParserError: 765: unexpected token at '']>

It seems the conversion was not even started. Is your environment OK?

Comment 18 Tomáš Golembiovský 2019-05-13 21:39:18 UTC
FYI I did a new build with the patch from comment #9 applied: v2v-conversion-host-1.13.1-1.el7ev

Comment 19 Fabien Dupont 2019-05-13 21:40:59 UTC
I'm confused. Isn't the patch supposed to be applied on the conversion host by installing the new virt-v2v-wrapper.py ?
@shveta, what patch did you apply on your appliance ?

Comment 20 Shveta 2019-05-13 22:31:13 UTC
So on Rhev conversion host (env-rhv43-01.cfme2.lab.eng.rdu2.redhat.com)I replaced the virt-v2v-wrapper.py with new code as in the commit and started migration .

Conversion Host is there :

irb(main):001:0> ConversionHost.all
PostgreSQLAdapter#log_after_checkout, connection_pool: size: 5, connections: 1, in use: 1, waiting_in_queue: 0
=> #<ActiveRecord::Relation [#<ConversionHost id: 25, name: "env-rhv43-01.cfme2.lab.eng.rdu2.redhat.com", address: nil, type: nil, resource_type: "Host", resource_id: 3, version: nil, max_concurrent_tasks: nil, vddk_transport_supported: nil, ssh_transport_supported: true, created_at: "2019-05-13 18:35:44", updated_at: "2019-05-13 18:36:01", concurrent_transformation_limit: nil, cpu_limit: nil, memory_limit: nil, network_limit: nil, blockio_limit: nil>]>


If there is a different way to apply patch please let me know .

Comment 21 Tomáš Golembiovský 2019-05-14 08:41:52 UTC
(In reply to Shveta from comment #20)

> If there is a different way to apply patch please let me know .

You forgot to set executable bit on the new file. I guess that's the source of the problem. I fixed that on `env-rhv43-01....` please retest.

Comment 23 Yadnyawalk Tale 2019-05-14 15:19:15 UTC
Created attachment 1568512 [details]
second_automation.log

We have applied patches to cloudforms (https://bugzilla.redhat.com/show_bug.cgi?id=1708739#c3) and added conversion host with SSH transformation method.
But when we tried end to end migration it failed with similar ssh-agent error in wrapper log, this time it failed on line 1734 of /usr/bin/virt-v2v-wrapper.py.

Tested with
CFME 5.10.4.3.20190509221642_282f74e 
RHV 4.3.3.3 with conversion package 1.12.1-5
Used following HTTP request to add conversion host via CFME with SSH.

```
{
  "name": "reducted-hostname.com",
  "resource_type": "ManageIQ::Providers::Redhat::InfraManager::Host",
  "resource_id": "1",
  "vmware_ssh_private_key": "-----BEGIN OPENSSH PRIVATE KEY-----\n-reducted-key\n-----END OPENSSH PRIVATE KEY-----\n",
  "auth_user": "root"
}
```

Marking failedQA since it did not worked successfully.

Comment 24 Yadnyawalk Tale 2019-05-14 15:20:18 UTC
Created attachment 1568513 [details]
second_wrapper.log

Comment 25 Tomáš Golembiovský 2019-05-14 15:34:41 UTC
The log says:

    2019-05-14 11:16:29,674:ERROR: ssh-add output: Enter passphrase for /var/lib/vdsm/.ssh/id_rsa:  (virt-v2v-wrapper:1932)

It looks like the SSH key you have created requires passphrase. This is not supported. Use a key without passphrase, please.

Comment 26 Shveta 2019-05-14 20:01:28 UTC
Changing this back to assigned as their is no build with the fix yet to test .
Also the patch provided did not work. I created a key without passphrase and restarted migration but it failed. 
Tomas is investigating.

Comment 28 Shveta 2019-05-14 22:14:49 UTC
Please test from  env-rhv43-02.cfme2.lab.eng.rdu2.redhat.com.The Key is copied to it.
and this is the vmware host smicro-5037-07.cfme.lab.eng.rdu2.redhat.com.

Comment 42 Yadnyawalk Tale 2019-05-15 14:22:51 UTC
reverting back to urgent, sorry it was due to collision I guess.

Comment 43 Tomáš Golembiovský 2019-05-15 14:35:59 UTC
The SSH key is not properly configured because of bug 1710448 in CFME. The key is striped after first space which makes it invalid.

Comment 44 Fabien Dupont 2019-05-15 14:58:57 UTC
Can we still validate the fix on the wrapper by setting the key manually on the conversion host ? This would avoid waiting on bug 1710448.

Comment 48 Shveta 2019-05-15 18:24:49 UTC
Log after making the above change in conversion_host.rb
http://pastebin.test.redhat.com/763433

Comment 49 Yadnyawalk Tale 2019-05-16 14:18:16 UTC
Good news! SSH for RHV worked after applying patch from BZ1710448. Fabien also confirmed it on .106.
We are still confirming OSP + SSH usecase with 1.31.1-1 conversion host package.

Comment 50 Yadnyawalk Tale 2019-05-16 14:21:23 UTC
Created attachment 1569530 [details]
rhv_SSH_5.10.4.3_patches.png

Comment 51 Yadnyawalk Tale 2019-05-17 10:44:57 UTC
Created attachment 1570025 [details]
osp_ssh_works.png

@Shveta confirmed that migration works with OSP + SSH after applying BZ1710448's patch on cloudforms.
So overall, RHV and OSP with SSH fully functional with Tomáš's patch from Comment #9.

Many thanks to @Fabien, @Tomáš, @Daniel, @MikeT.

Comment 52 Yadnyawalk Tale 2019-05-17 11:40:22 UTC
*** Bug 1708802 has been marked as a duplicate of this bug. ***

Comment 53 Tomáš Golembiovský 2019-05-17 13:30:16 UTC
Please move the bug to VERIFIED if things are working with v2v-conversion-host-1.13.1-1.el7ev

Comment 55 errata-xmlrpc 2019-06-20 14:47:53 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:1559


Note You need to log in before you can comment on or make changes to this bug.