Bug 1401312

Created attachment 1227940 [details]
evm log

The other bug mentioned in the bug description with error: "Unable to mount..." is bug 1401315.

Apparently in evm.log, there was an error on SSH to host.
The root cause of this error is not clear, and this might be another bug.

Having this error, VM provision, failed as well on the same error: "version 3 of the api..."
 
However, after removing RHV from the CFME appliance, and adding it back,
VM provision, and VM SSA (for the CFME appliance - RHEL7.3, that has NFS disk), 
succeed.

Therefore, it can be confirmed that bug 1401315 (VM SSA on RHEL7.3) is not occurring on CFME-5.7.0.13.

Even if there is SSH problem to host, we shouldn't have operations like SSA/VM provision/login with bad credentials fail on: "version 3 of the api...", that do not imply on the problem itself.

Moti,
The host connection issue might be a bug.
Is it possible to find on evm.log, the root cause of the host connection issue?

(In reply to Ilanit Stein from comment #4)
> Apparently in evm.log, there was an error on SSH to host.
> The root cause of this error is not clear, and this might be another bug.
> 
> Having this error, VM provision, failed as well on the same error: "version
> 3 of the api..."
>  
> However, after removing RHV from the CFME appliance, and adding it back,
> VM provision, and VM SSA (for the CFME appliance - RHEL7.3, that has NFS
> disk), 
> succeed.
> 
> Therefore, it can be confirmed that bug 1401315 (VM SSA on RHEL7.3) is not
> occurring on CFME-5.7.0.13.
> 
> Even if there is SSH problem to host, we shouldn't have operations like
> SSA/VM provision/login with bad credentials fail on: "version 3 of the
> api...", that do not imply on the problem itself.
> 
> Moti,
> The host connection issue might be a bug.
> Is it possible to find on evm.log, the root cause of the host connection
> issue?

The failed attempts to connect the host from the manageiq server can be found on /etc/secure:

Dec  5 10:00:44 seal10 sshd[20696]: Failed password for root from 10.35.93.2 port 37286 ssh2
Dec  5 10:00:44 seal10 sshd[20696]: Failed password for root from 10.35.93.2 port 37286 ssh2
Dec  5 10:00:44 seal10 sshd[20696]: Failed password for root from 10.35.93.2 port 37286 ssh2

and on the manageiq side, evm.log:

root.lab.tlv.redhat.com's password: stty: standard input: Input/output error
stty: standard input: Input/output error
stty: standard input: Input/output error

root.lab.tlv.redhat.com's password: stty: standard input: Input/output error
stty: standard input: Input/output error
stty: standard input: Input/output error

E, [2016-12-05T03:00:44.741421 #18755] ERROR -- net.ssh.authentication.session[cfe4e60]: all authorization methods failed (tried none, publickey, password)
[----] E, [2016-12-05T03:00:44.741748 #18755:10a9690] ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Host#connect_ssh) SSH connection failed for [seal10.qa.l
ab.tlv.redhat.com] with [Net::SSH::AuthenticationFailed: Authentication failed for user root.lab.tlv.redhat.com]
[----] E, [2016-12-05T03:00:44.742226 #18755:10a9690] ERROR -- : MIQ(host_controller-update): Login failed due to a bad username or password.

Therefore a wrong password was provided for the host.

Created attachment 1228054 [details]
evm.log during SSA on VM

Hello Ilanit,

I just ran SSA on one of the VM, got some SSH failures in evm.log.
See attached log.

Thanks,
Satyajit Bulage.

New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/cd32fa61d924694d9d8c500131d3b225897f6cb7

commit cd32fa61d924694d9d8c500131d3b225897f6cb7
Author:     Adam Grare <agrare>
AuthorDate: Tue Dec 6 10:37:39 2016 -0500
Commit:     Oleg Barenboim <chessbyte>
CommitDate: Tue Dec 6 10:52:51 2016 -0500

    Merge pull request #13009 from borod108/bugs/1401312versions_cache
    
    Retry getting api supported versions when empty
    (cherry picked from commit 854046ae4719ff0f89abfc7aa41d148f8f33b6c0)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1401312

 .../providers/redhat/infra_manager/api_integration.rb      | 14 +++++++++++---
 .../models/manageiq/providers/redhat/infra_manager_spec.rb | 14 ++++++++++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/f141598cac90abfa5b7f87e328eb57ab4a5a25bb

commit f141598cac90abfa5b7f87e328eb57ab4a5a25bb
Author:     borod108 <bodnopoz>
AuthorDate: Tue Dec 6 11:13:06 2016 +0200
Commit:     borod108 <bodnopoz>
CommitDate: Tue Dec 6 12:21:01 2016 +0200

    Retry getting api supported versions when empty
    
    If the stored supported_api_versions is empty try
    to fetch it even if cache is not stale.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1401312

 .../providers/redhat/infra_manager/api_integration.rb      | 14 +++++++++++---
 .../models/manageiq/providers/redhat/infra_manager_spec.rb | 14 ++++++++++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

Never mind... PR: https://github.com/ManageIQ/manageiq/pull/13009

Verified on CFME-5.8.0.13/RHV-4.1.1.8
VM SSA for NFS disk worked OK. 
There was no such error "version 3 of the api.."
and there was no error in evm.log