Bug 1353632

Summary: Automate Method: Amazon check_pre_retirement stuck in retry loop
Product: Red Hat CloudForms Management Engine Reporter: mkanoor
Component: AutomateAssignee: William Fitzgerald <wfitzger>
Status: CLOSED CURRENTRELEASE QA Contact: Matouš Mojžíš <mmojzis>
Severity: high Docs Contact:
Priority: high    
Version: 5.5.0CC: dajohnso, jhardy, jkrocil, kzvyahin, mkanoor, mmojzis, obarenbo, otsuman, simaishi, tfitzger, wfitzger
Target Milestone: GAKeywords: ZStream
Target Release: 5.7.0Flags: kzvyahin: automate_bug-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 5.7.0.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1372772 (view as bug list) Environment:
Last Closed: 2017-02-28 13:51:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1372772    
Attachments:
Description Flags
bug none

Description mkanoor 2016-07-07 15:34:04 UTC 
Description of problem:
An amazon instance can be running either in an "ebs" store or an "instance" store. Only ebs instances are stopped, the instance store instances can be just terminated. The amazon_check_pre_retirement method has a typo it uses "instance_store" instead of "instance-store". This typo causes the script to be stuck in a retry loop preventing the retirement from progressing.


Version-Release number of selected component (if applicable):
5.5

How reproducible:
Very

Steps to Reproduce:
1. Provision an Amazon instance on an instance store
2. Retire the VM
3.

Actual results:
The script gets stuck in the retry loop for instance-store backed amazon instances

Expected results:
The script should continue with the retirement

Additional info:
Comment 2 CFME Bot 2016-07-07 21:08:20 UTC 
https://github.com/ManageIQ/manageiq/pull/9673
Comment 3 CFME Bot 2016-07-07 21:57:29 UTC 
https://github.com/ManageIQ/manageiq/pull/9675
Comment 4 CFME Bot 2016-07-08 14:16:46 UTC 
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/3975f927d53382dbda640ef96e2b0bb711f3f64e

commit 3975f927d53382dbda640ef96e2b0bb711f3f64e
Author:     Madhu Kanoor <mkanoor>
AuthorDate: Thu Jul 7 17:45:15 2016 -0400
Commit:     Madhu Kanoor <mkanoor>
CommitDate: Thu Jul 7 17:45:15 2016 -0400

    Fixed root_device_type typo
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1353632
    
    Should be checking for instance-store and not instance_store

 .../Methods.class/__methods__/amazon_check_pre_retirement.rb            | 2 +-
 spec/automation/unit/method_validation/amazon_pre_retirement_spec.rb    | 2 +-
 .../unit/method_validation/check_amazon_pre_retirement_spec.rb          | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
Comment 8 Kyrylo Zvyagintsev 2016-11-29 10:42:37 UTC 
Created attachment 1225778 [details]
bug
Comment 9 William Fitzgerald 2016-11-29 22:35:18 UTC 
Kyrylo,

Can I have access to your appliance ? 

I tried to re-create but it works for me.

Thanks

Billy
Comment 10 Dave Johnson 2016-12-06 16:52:23 UTC 
Please assess the impact of this issue and update the severity accordingly.  Please refer to https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity for a reminder on each severity's definition.
Comment 11 mkanoor 2016-12-06 20:20:56 UTC 
This is a high priority issue. There were 2 main issues uncovered

(1) There was a typo in the Automate Method which was causing the retry loop
(2) When an instance is provisioned outside of CFME we were not able to retire it.


Marcel Hild had updated the Amazon gem to record the instance_types of EC2 instances provided outside CFME.
Comment 15 mkanoor 2016-12-20 21:47:24 UTC 
can you please check if you have set the lifecycle tags for the instance provisioned outside of CFME.

Details in 
https://bugzilla.redhat.com/show_bug.cgi?id=1372772
Comment 16 Dave Johnson 2017-01-10 00:16:28 UTC 
Matous, with a little guidance from Milan, please retest this
Comment 17 Tina Fitzgerald 2017-01-16 21:58:18 UTC 
Hi Matous,

Are you able to reproduce this issue?

Thanks,
Tina
Comment 18 Matouš Mojžíš 2017-01-20 10:42:39 UTC 
Can't reproduce neither on 5.6.3.3 nor on 5.7.0.17.
Instance-store images were terminated after I tried to retire them.
Comment 19 Matouš Mojžíš 2017-01-26 09:02:36 UTC 
The problem is with retirement or stop?
I just reproduced steps which Kyrylo wrote here and I can retire it, but I can't stop it. Only terminate/retire works.
As I can read in description of problem the problem is with stopping instance...
When I stop instance I get this error:
[----] E, [2017-01-26T04:01:56.996987 #11914:88d13c] ERROR -- : MIQ(MiqQueue#m_callback) Message id: [16520]: The instance 'i-0be3b54350b54c91d' does not have an 'ebs' root device type and cannot be stopped.
Comment 20 mkanoor 2017-01-26 19:28:14 UTC 
Hi Matous,
I think one of the things you might want to try is provision instances directly in EC2 console instead of using our provisioning. Then try to retire it.

If you want to retire instances provisioned outside of CFME, it is necessary that they have the lifecycle tag
https://bugzilla.redhat.com/show_bug.cgi?id=1372772


Thanks,
Madhu
Comment 21 Matouš Mojžíš 2017-01-27 09:03:22 UTC 
I tried it and it also works. So I can set it to verified then?
Comment 22 mkanoor 2017-01-27 14:06:43 UTC 
Yes please
Comment 23 Matouš Mojžíš 2017-01-30 09:36:30 UTC 
Verified in 5.7.1.0. Instance-store instance is retired correctly.