Bug 1351262

Summary:	Takes too long for Amazon VM instances to service retire
Product:	Red Hat CloudForms Management Engine	Reporter:	Tsai Li Ming <ltsai>
Component:	Automate	Assignee:	William Fitzgerald <wfitzger>
Status:	CLOSED ERRATA	QA Contact:	Shveta <sshveta>
Severity:	medium	Docs Contact:
Priority:	high
Version:	5.4.0	CC:	gblomqui, jhardy, ltsai, mkanoor, obarenbo, simaishi, tfitzger, wfitzger
Target Milestone:	GA	Keywords:	ZStream
Target Release:	5.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	ec2:service:retire
Fixed In Version:	5.7.0.0	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1372771 (view as bug list)		Environment:
Last Closed:	2017-01-04 12:56:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1372771

Description Tsai Li Ming 2016-06-29 14:39:32 UTC

Description of problem:
1. An Amazon instance is provisioned using a service catalogue.
2. Service catalogue is then retired using Lifecycle -> Retire this service

The instances goes to the "powering down" state and gets stuck for a long time. It doesn't go to the "power off" state unless I force a provider refresh from the UI (I believe VM refresh state will be the same). 

I had to wait till the provider got refreshed before the state will change, which will then continue the retirement process. This took about 30mins. 

Compared to RHOS, RHOS is significantly faster in the service retirement process.

Version-Release number of selected component (if applicable):
Latest CF 3.2 (5.4.4)

How reproducible:
Always

Steps to Reproduce:
1. Provision an Amazon instance using a service catalogue
2. Retire the service under My Services
3. Observe the slowness and state of the VM

Actual results:
VM doesn't retire / gets stuck in the retirement process

Expected results:
VM should retire as soon as possible.

Additional info:

evm logs:
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
[----] I, [2016-06-29T22:10:12.997874 #14980:4bfc1e8]  INFO -- : <AutomationEngine> <AEMethod amazon_check_pre_retirement> Instanc
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
[----] I, [2016-06-29T22:12:24.423405 #27746:5a338b0]  INFO -- : <AutomationEngine> <AEMethod amazon_check_pre_retirement> Instanc
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
[----] I, [2016-06-29T22:13:30.178788 #27746:5df541c]  INFO -- : <AutomationEngine> <AEMethod amazon_check_pre_retirement> Instanc
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
[----] I, [2016-06-29T22:14:37.548781 #14980:4f59c50]  INFO -- : <AutomationEngine> <AEMethod amazon_check_pre_retirement> Instanc
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
[----] I, [2016-06-29T22:16:49.380637 #14980:52a2970]  INFO -- : <AutomationEngine> <AEMethod amazon_check_pre_retirement> Instanc
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
[----] I, [2016-06-29T22:19:01.864513 #14980:53ca6cc]  INFO -- : <AutomationEngine> <AEMethod amazon_check_pre_retirement> Instanc
e:<test_retirement_aws_009002> on EMS:<Amazon has Power State:<powering_down>
.... many more ...

Comment 2 Tina Fitzgerald 2016-06-29 14:53:57 UTC

I think this is an issue with VM retirement. 
Can you retire an Amazon VM using VM retirement to see if you get the same results?

Thanks,
Tina

Comment 3 Tsai Li Ming 2016-06-29 15:13:02 UTC

Same issue if I use VM lifecycle retirement. Has been > 10 mins.

Comment 4 William Fitzgerald 2016-06-29 19:50:34 UTC

Can I have access to your appliance?

Thanks

Bill

Comment 6 William Fitzgerald 2016-06-30 13:35:32 UTC

I didn't know this was happening at a customer site.  Can I get a copy of the automation.log and evm.log when this problem occurred.  If this is a customer site, then don't try to get me accees, just the logs.

Thanks

Bill

Comment 11 William Fitzgerald 2016-07-19 18:15:02 UTC

Changed retirement state machine to do a vm refresh for Amazon.

This issue has been resolved in the following PR:

https://github.com/ManageIQ/manageiq/pull/9706

Comment 14 Shveta 2016-09-20 19:07:20 UTC

Verified in 5.7.0.0.20160906172503_12f65cb

Comment 16 errata-xmlrpc 2017-01-04 12:56:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2017-0012.html