Bug 1401312 - Smart State analysis for VM fail on "version 3 of the api is not supported by the provider"
Summary: Smart State analysis for VM fail on "version 3 of the api is not supported by...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: GA
: 5.8.0
Assignee: Boriso
QA Contact: Ilanit Stein
URL:
Whiteboard: rhev, smartstate analysis
Depends On:
Blocks: 1402082
TreeView+ depends on / blocked
 
Reported: 2016-12-04 16:13 UTC by Ilanit Stein
Modified: 2017-06-12 17:40 UTC (History)
9 users (show)

Fixed In Version: 5.8.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1402082 (view as bug list)
Environment:
Last Closed: 2017-06-12 17:40:06 UTC
Category: Bug
Cloudforms Team: RHEVM
Target Upstream Version:


Attachments (Terms of Use)
evm log (999.95 KB, application/x-gzip)
2016-12-04 16:22 UTC, Ilanit Stein
no flags Details
evm.log during SSA on VM (1.98 KB, text/plain)
2016-12-05 12:49 UTC, Satyajit Bulage
no flags Details

Description Ilanit Stein 2016-12-04 16:13:58 UTC
Description of problem:
Smart state analysis for a VM with NFS disk fail on:
"version 3 of the api is not supported by the provider"

Version-Release number of selected component (if applicable):
CFME-5.7.0.13-1
RHV-4.0.5

Steps to Reproduce:
Have a RHV-4.0.5, run a VM, based on CFME-5.7.0.13-1 template.
From CFME, perform SSA for the CFME-5.7.0.13-1 VM (RHEL-7.3).
This VM is also set in "Edit management engine relationship". 

Expected results:
SSA for VM should succeed.

Additional info:
1. SSA for a RHEL7.3 VM with NFS disk work fine on CFME-5.7.0.12.
 
2. There is another bug, yet opened (I'll add it later on),
that fail VM SSA (for any disk type),
on CFME-5.6.3.2 (and CFME-5.6.3.3),
with ERROR: "Unable to mount filesystem. Reason:[No root filesystem found.]"
This bug may also occur on CFME-5.7.0.13, 
but the "version 3.." error, block from seeing that.

Comment 2 Ilanit Stein 2016-12-04 16:22:16 UTC
Created attachment 1227940 [details]
evm log

Comment 3 Ilanit Stein 2016-12-04 16:37:48 UTC
The other bug mentioned in the bug description with error: "Unable to mount..." is bug 1401315.

Comment 4 Ilanit Stein 2016-12-05 11:18:39 UTC
Apparently in evm.log, there was an error on SSH to host.
The root cause of this error is not clear, and this might be another bug.

Having this error, VM provision, failed as well on the same error: "version 3 of the api..."
 
However, after removing RHV from the CFME appliance, and adding it back,
VM provision, and VM SSA (for the CFME appliance - RHEL7.3, that has NFS disk), 
succeed.

Therefore, it can be confirmed that bug 1401315 (VM SSA on RHEL7.3) is not occurring on CFME-5.7.0.13.

Even if there is SSH problem to host, we shouldn't have operations like SSA/VM provision/login with bad credentials fail on: "version 3 of the api...", that do not imply on the problem itself.

Moti,
The host connection issue might be a bug.
Is it possible to find on evm.log, the root cause of the host connection issue?

Comment 5 Moti Asayag 2016-12-05 12:23:55 UTC
(In reply to Ilanit Stein from comment #4)
> Apparently in evm.log, there was an error on SSH to host.
> The root cause of this error is not clear, and this might be another bug.
> 
> Having this error, VM provision, failed as well on the same error: "version
> 3 of the api..."
>  
> However, after removing RHV from the CFME appliance, and adding it back,
> VM provision, and VM SSA (for the CFME appliance - RHEL7.3, that has NFS
> disk), 
> succeed.
> 
> Therefore, it can be confirmed that bug 1401315 (VM SSA on RHEL7.3) is not
> occurring on CFME-5.7.0.13.
> 
> Even if there is SSH problem to host, we shouldn't have operations like
> SSA/VM provision/login with bad credentials fail on: "version 3 of the
> api...", that do not imply on the problem itself.
> 
> Moti,
> The host connection issue might be a bug.
> Is it possible to find on evm.log, the root cause of the host connection
> issue?

The failed attempts to connect the host from the manageiq server can be found on /etc/secure:

Dec  5 10:00:44 seal10 sshd[20696]: Failed password for root from 10.35.93.2 port 37286 ssh2
Dec  5 10:00:44 seal10 sshd[20696]: Failed password for root from 10.35.93.2 port 37286 ssh2
Dec  5 10:00:44 seal10 sshd[20696]: Failed password for root from 10.35.93.2 port 37286 ssh2

and on the manageiq side, evm.log:

root.lab.tlv.redhat.com's password: stty: standard input: Input/output error
stty: standard input: Input/output error
stty: standard input: Input/output error

root.lab.tlv.redhat.com's password: stty: standard input: Input/output error
stty: standard input: Input/output error
stty: standard input: Input/output error

E, [2016-12-05T03:00:44.741421 #18755] ERROR -- net.ssh.authentication.session[cfe4e60]: all authorization methods failed (tried none, publickey, password)
[----] E, [2016-12-05T03:00:44.741748 #18755:10a9690] ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Host#connect_ssh) SSH connection failed for [seal10.qa.l
ab.tlv.redhat.com] with [Net::SSH::AuthenticationFailed: Authentication failed for user root.lab.tlv.redhat.com]
[----] E, [2016-12-05T03:00:44.742226 #18755:10a9690] ERROR -- : MIQ(host_controller-update): Login failed due to a bad username or password.

Therefore a wrong password was provided for the host.

Comment 6 Satyajit Bulage 2016-12-05 12:49:58 UTC
Created attachment 1228054 [details]
evm.log during SSA on VM

Hello Ilanit,

I just ran SSA on one of the VM, got some SSH failures in evm.log.
See attached log.

Thanks,
Satyajit Bulage.

Comment 8 CFME Bot 2016-12-06 17:14:27 UTC
New commit detected on ManageIQ/manageiq/euwe:
https://github.com/ManageIQ/manageiq/commit/cd32fa61d924694d9d8c500131d3b225897f6cb7

commit cd32fa61d924694d9d8c500131d3b225897f6cb7
Author:     Adam Grare <agrare>
AuthorDate: Tue Dec 6 10:37:39 2016 -0500
Commit:     Oleg Barenboim <chessbyte>
CommitDate: Tue Dec 6 10:52:51 2016 -0500

    Merge pull request #13009 from borod108/bugs/1401312versions_cache
    
    Retry getting api supported versions when empty
    (cherry picked from commit 854046ae4719ff0f89abfc7aa41d148f8f33b6c0)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1401312

 .../providers/redhat/infra_manager/api_integration.rb      | 14 +++++++++++---
 .../models/manageiq/providers/redhat/infra_manager_spec.rb | 14 ++++++++++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

Comment 9 CFME Bot 2016-12-06 17:16:50 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/f141598cac90abfa5b7f87e328eb57ab4a5a25bb

commit f141598cac90abfa5b7f87e328eb57ab4a5a25bb
Author:     borod108 <bodnopoz>
AuthorDate: Tue Dec 6 11:13:06 2016 +0200
Commit:     borod108 <bodnopoz>
CommitDate: Tue Dec 6 12:21:01 2016 +0200

    Retry getting api supported versions when empty
    
    If the stored supported_api_versions is empty try
    to fetch it even if cache is not stale.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1401312

 .../providers/redhat/infra_manager/api_integration.rb      | 14 +++++++++++---
 .../models/manageiq/providers/redhat/infra_manager_spec.rb | 14 ++++++++++++++
 2 files changed, 25 insertions(+), 3 deletions(-)

Comment 10 Satoe Imaishi 2016-12-06 17:45:56 UTC
Never mind... PR: https://github.com/ManageIQ/manageiq/pull/13009

Comment 12 Ilanit Stein 2017-05-04 07:00:08 UTC
Verified on CFME-5.8.0.13/RHV-4.1.1.8
VM SSA for NFS disk worked OK. 
There was no such error "version 3 of the api.."
and there was no error in evm.log


Note You need to log in before you can comment on or make changes to this bug.