Bug 1402035 - Host SSH error in evm.log, further to VM Smart state Analysis
Summary: Host SSH error in evm.log, further to VM Smart state Analysis
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.7.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: GA
: 5.8.0
Assignee: Joe Rafaniello
QA Contact: Ilanit Stein
URL:
Whiteboard: core
: 1392460 (view as bug list)
Depends On:
Blocks: 1402887
TreeView+ depends on / blocked
 
Reported: 2016-12-06 15:41 UTC by Ilanit Stein
Modified: 2017-06-12 17:07 UTC (History)
9 users (show)

Fixed In Version: 5.8.0.0
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1402887 (view as bug list)
Environment:
Last Closed: 2017-06-12 17:07:03 UTC
Category: ---
Cloudforms Team: RHEVM
Target Upstream Version:


Attachments (Terms of Use)
evm.log (1.04 MB, application/x-gzip)
2016-12-06 15:56 UTC, Ilanit Stein
no flags Details

Description Ilanit Stein 2016-12-06 15:41:37 UTC
Description of problem:

Further to VM SSA, evm.log contain this error:

[----] E, [2016-12-06T08:54:39.865473 #18632:f3514c] ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Host#connect_ssh) SSH connection failed for [seal10.qa.la
b.tlv.redhat.com] with [ArgumentError: invalid option(s): attempt]

masayag:
It seems that the SSH errors in the code,
Are regression due to the following commit [1]:

[1] https://github.com/ManageIQ/manageiq/pull/11964

commit 2e8f5985e61d1566cca42649cd7c0b0f21fccdec
Author: Joe Rafaniello <jrafanie>
Date:   Fri Oct 14 18:50:43 2016 -0400

    Retry retryable authentication failures

Version-Release number of selected component (if applicable):
CFME-5.7.0.13
(Seen on RHV-4.0.5)

Additional info:
Not sure as for the impact this SSH problem may cause.

Comment 2 Joe Rafaniello 2016-12-06 15:53:12 UTC
Can you provide a full backtrace or an example of how to recreate this error?

What ovirt/rhev provider code are you seeing this with?  I searched for callers to authentication_check* methods to make sure this wasn't a breaking change but maybe I missed one.

Thanks!

Comment 3 Ilanit Stein 2016-12-06 15:56:15 UTC
Created attachment 1228616 [details]
evm.log

In this evm.log,
For VM SSA, done at 2016-12-06T10:43:52
can see this error in evm.log, few minutes after:

ERROR -- : MIQ(ManageIQ::Providers::Redhat::InfraManager::Host#connect_ssh) SSH connection failed for [seal10.qa.lab.tlv.redhat.com] with [ArgumentError: invalid option(s): attempt]

Comment 4 Ilanit Stein 2016-12-06 16:03:38 UTC
Joe,

In order to recreate this error, need to run VM SSA.
I did it on RHV, but code indicate it's a core bug, so it may occur for other providers as well, like VMWare.

evm.log has been attached.

As for your question in the code, masayag is the one who tracked it:

Moti,
Would you please address Joe's question in comment 2? Thanks.

Comment 5 Joe Rafaniello 2016-12-06 18:04:14 UTC
Thank you for finding this Ilanit.  Can you try this PR?  I was able to recreate the problem and put a pretty ugly but working unit test on it.

https://github.com/ManageIQ/manageiq/pull/13026

Note, you can use this to get a patch file:

https://github.com/ManageIQ/manageiq/pull/13026.patch

Comment 6 Ilanit Stein 2016-12-07 08:34:55 UTC
Tested CFME-5.7.0.13 + https://github.com/ManageIQ/manageiq/pull/13026.patch
(Thanks masayag, for helping with adding the patch):

Same CFME-5.7.0.13, where bug was reported,
Running VM SSA for the same VM, that was run before.

Now there is no ERROR in the evm.log, 15 min after the SSA operation.

Therefore, this patch is verified.

Thank you very much for your quick response.

Comment 7 Oved Ourfali 2016-12-07 14:39:00 UTC
Can we push that to 5.7.0?

Comment 8 CFME Bot 2016-12-07 15:41:09 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/ab40ebedc07c7ebd06ac4d9f759cafada5bbfdfb

commit ab40ebedc07c7ebd06ac4d9f759cafada5bbfdfb
Author:     Joe Rafaniello <jrafanie>
AuthorDate: Tue Dec 6 12:51:43 2016 -0500
Commit:     Joe Rafaniello <jrafanie>
CommitDate: Tue Dec 6 13:00:38 2016 -0500

    Filter attempt from the authentication_check options
    
    Various implementations of verify_credentials aren't expecting an
    :attempt => 1 hash key/value.
    
    It's only used at a higher layer so exclude it from the
    authentication_check call.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1402035
    
    This was a regression introduced in #11964

 app/models/mixins/authentication_mixin.rb       | 2 +-
 spec/models/mixins/authentication_mixin_spec.rb | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

Comment 11 Joe Rafaniello 2016-12-08 14:44:19 UTC
> Can we push that to 5.7.0?

Oved, this has been marked as a blocker for backport to euwe and 5.7.

Your team found this JUST in time... a few days later and it might not have made it.  Thanks for finding this.

Comment 12 Oved Ourfali 2016-12-08 19:46:28 UTC
Thanks for the fast response and for pushing this to 5.7.0. 

Ilanit, thanks indeed for finding this.

Comment 13 Tzu-Mainn Chen 2017-01-09 22:05:56 UTC
*** Bug 1392460 has been marked as a duplicate of this bug. ***

Comment 14 Ilanit Stein 2017-05-04 07:01:25 UTC
Verified on CFME-5.8.0.13/RHV-4.1.1.8
VM SSA for NFS disk worked OK. 
There was no error in evm.log


Note You need to log in before you can comment on or make changes to this bug.