Bug 1697600 - Retirement request setting auto approve on request creation.
Summary: Retirement request setting auto approve on request creation.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Automate
Version: 5.10.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.11.0
Assignee: Tina Fitzgerald
QA Contact: Niyaz Akhtar Ansari
Red Hat CloudForms Documentation
URL:
Whiteboard:
Depends On:
Blocks: 1702075
TreeView+ depends on / blocked
 
Reported: 2019-04-08 19:24 UTC by Tina Fitzgerald
Modified: 2023-03-24 14:42 UTC (History)
8 users (show)

Fixed In Version: 5.11.0.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1702075 (view as bug list)
Environment:
Last Closed: 2019-12-13 14:58:16 UTC
Category: Bug
Cloudforms Team: CFME Core
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Tina Fitzgerald 2019-04-08 19:24:37 UTC
Description of problem:

Retirement request creation was setting the auto_approve flag at request creation  preventing users from being able to manually approve/deny request in Automate 

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Tina Fitzgerald 2019-04-09 13:22:11 UTC
https://github.com/ManageIQ/manageiq/pull/18638 resolves this issue has been merged.

Comment 5 Tina Fitzgerald 2019-04-10 20:17:28 UTC
Hi Dmitry,

It looks like the service retirement approval state machine is set to auto-approve

3287 [----] I, [2019-04-10T14:15:33.131958 #9406:3e0f58]  INFO -- : <AEMethod [/Metinvest/Service/Retirement/StateMachines/ServiceRetirementRequestApproval/approve_request]> Starting
13288 [----] I, [2019-04-10T14:15:33.420225 #9406:3d07318]  INFO -- : <AEMethod approve_request> Checking for auto_approval
13289 [----] I, [2019-04-10T14:15:33.421567 #9406:3d07318]  INFO -- : <AEMethod approve_request> AUTO-APPROVING
13290 [----] I, [2019-04-10T14:15:33.469925 #9406:3e0f58]  INFO -- : <AEMethod [/Metinvest/Service/Retirement/StateMachines/ServiceRetirementRequestApproval/approve_request]> Ending
13291 [----] I, [2019-04-10T14:15:33.470003 #9406:3e0f58]  INFO -- : Method exited with rc=MIQ_OK
13292 [----] I, [2019-04-10T14:15:33.470253 #9406:3e0f58]  INFO -- : Processed State=[ApproveRequest]
13293 [----] I, [2019-04-10T14:15:33.470390 #9406:3e0f58]  INFO -- : Next State=[]
13294 [----] I, [2019-04-10T14:15:33.470484 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/Service/Retirement/StateMachines/ServiceRetirementRequestApproval/Default#create]
13295 [----] I, [2019-04-10T14:15:33.470751 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/System/Policy/ServiceRetireRequest_created#create]
13296 [----] I, [2019-04-10T14:15:33.471067 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/System/Policy/request_created#create]
13297 [----] I, [2019-04-10T14:15:33.471348 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/System/Event/RequestEvent/Request/request_created#create]
13298 [----] I, [2019-04-10T14:15:37.888145 #13327:3e0f58]  INFO -- : Q-task_id([r1000000000399_vm_retire_task_1000000000807]) User [admin] 

The service will continue to try to be retired unless the retirement date is extended, or some other action is taken.

Can you set the auto-approve to manual?

Thanks,
Tina

Comment 7 Niladri Roy 2019-04-11 09:58:55 UTC
Cu response

---
Right now approval type is set to manual and request is only approved when certain conditions are met. Anyway when first retirement step comes there is two separate requests spawned. First one is denied. The second one retires the VM and the service.

I do believe there shall be only one service retirement request spawned, not two. Since service retirement workflow is one of the key points of the project, please solve this double retirement issue.
---

Logs and request numbers given in next comment

Comment 10 Niladri Roy 2019-04-17 08:45:55 UTC
Hi Tina,

Did you get a chance to check the customers recent query?
Please suggest

Comment 11 Tina Fitzgerald 2019-04-22 16:40:47 UTC
Hi Niladri,

I suspect the issue is that the scheduler is running frequently, and even though a previous retirement request was not approved, it doesn't prevent the scheduler from initiating retirement again, and again for the same object.  If you want to stop or delay the retirement in the retirement auto approval state machine, you'll need to change the "retireability" of the object. Extending the retirement date is one way to accomplish this. Another option is to allow the object to enter retirement, and customize the retirement state machine for the desired result. Once the retirement has been started, the retirement state will be 'retiring' and the scheduler will ignore the object. 

More detail:
The scheduled retirement check method initiates the retirement process for any object(service, vm, stack) where the retires on date has passed. An miq_request is created to process the object retirement.  Since the normal request flow includes an approval process, the retirement approval state machine is where a request can be denied, or left unapproved(pending). The assumption is that some action would be taken on the object to change the retireablility based on the customers requirements. Extending the retires_on date for example.  There is no mechanism in place that will prevent the scheduled retirement from creating a new request for the same object if the object is still retireable which will result in multiple retirement requests for the same object.

Let me know if you have any questions.

Thanks,
Tina

Comment 18 Tina Fitzgerald 2019-04-30 19:27:14 UTC
Hi Niladri

The customer identified 2 service retirement requests for the same service: 
https://bugzilla.redhat.com/show_bug.cgi?id=1697600#c15

The customer states that the 2nd request (477) should not have been created and approved for retirement.

A combination of circumstances are responsible for the second request getting created/approved.

1. The requests were created 4 seconds apart.
Request 476 was created on 4/25/19 at 11:26:41
Request 477 was created on 4/25/19 at 11:26:45

2. Request 476 - validate_request runs at 11:26:47  <----- The first request entered the retirement approval state machine validate_request method AFTER the second request was created.

3. Scheduler runs(queues) RetirementManager.check at the following times around that time period: 
ws1 - 11:11, 11:16, 11:21
ws2 - 11:06, 11:26, 11:31 <------ I suspect the 2nd request (477) was created from the 11:26 retirement check.

4. For Request 476, The custom validate_request Automate method sets the service custom attributes(in addition to extending the retirement date) (see code below)   So, the service will have a custom attribute 'Queued for removal' set to 'True'

5. For Request 477, the service custom attribute, 'Queued for removal' is still true from #4, so the rest of the code is ignored, and the the state machine progresses to approve_request

6. For Request 477, the custom approve_request Automate method code checks if the custom attribute 'Queued for removal' is true, (and it is as a result of #4), so the second request is approved and proceeds to retirement.

7. One way to prevent this from happening is to update the custom validate_request Automate method to check if the service retires_on date is in the future, and deny the request if that is the case. This would have caught the second request and would have prevented it from being approved.  

This is custom code, so it would be up to the customer to decide how to handle the situation. 

Let me know if you have any questions.

Thanks,
Tina



The customer approve_request Automate method code snippet:
-----------------------------------------------------------------------------------------------
if service.custom_get('Queued for removal') == 'True'
	$evm.log("info", "Service queued for retirement")
	$evm.root["miq_request"].approve("admin", "Service queued for retirement")
end 


The customer validate_request Automate method:
-----------------------------------------------------------------------------------------------
service = $evm.vmdb(:service).find_by_id($evm.root['miq_request'].options[:src_ids][0].to_i)
#Investigative_Debugging::Discovery::ObjectWalker.walk_objects

if service.custom_get('Queued for removal') == 'False'
	service.custom_set('Queued for removal', 'True')
  	if $evm.object['retirement_units'] == 'days'
    	service.retires_on = (Time.now + $evm.object['retirement_length'].days).to_s
    elsif $evm.object['retirement_units'] == 'minutes'
  		service.retires_on = (Time.now + $evm.object['retirement_length'].minutes).to_s
    end
  	$evm.root['ae_reason'] = 'auto_retire_schedule'
    $evm.root['ae_result'] = 'error'
  	#exit MIQ_ABORT
end


Note You need to log in before you can comment on or make changes to this bug.