Bug 1697600

https://github.com/ManageIQ/manageiq/pull/18638 resolves this issue has been merged.

Hi Dmitry,

It looks like the service retirement approval state machine is set to auto-approve

3287 [----] I, [2019-04-10T14:15:33.131958 #9406:3e0f58]  INFO -- : <AEMethod [/Metinvest/Service/Retirement/StateMachines/ServiceRetirementRequestApproval/approve_request]> Starting
13288 [----] I, [2019-04-10T14:15:33.420225 #9406:3d07318]  INFO -- : <AEMethod approve_request> Checking for auto_approval
13289 [----] I, [2019-04-10T14:15:33.421567 #9406:3d07318]  INFO -- : <AEMethod approve_request> AUTO-APPROVING
13290 [----] I, [2019-04-10T14:15:33.469925 #9406:3e0f58]  INFO -- : <AEMethod [/Metinvest/Service/Retirement/StateMachines/ServiceRetirementRequestApproval/approve_request]> Ending
13291 [----] I, [2019-04-10T14:15:33.470003 #9406:3e0f58]  INFO -- : Method exited with rc=MIQ_OK
13292 [----] I, [2019-04-10T14:15:33.470253 #9406:3e0f58]  INFO -- : Processed State=[ApproveRequest]
13293 [----] I, [2019-04-10T14:15:33.470390 #9406:3e0f58]  INFO -- : Next State=[]
13294 [----] I, [2019-04-10T14:15:33.470484 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/Service/Retirement/StateMachines/ServiceRetirementRequestApproval/Default#create]
13295 [----] I, [2019-04-10T14:15:33.470751 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/System/Policy/ServiceRetireRequest_created#create]
13296 [----] I, [2019-04-10T14:15:33.471067 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/System/Policy/request_created#create]
13297 [----] I, [2019-04-10T14:15:33.471348 #9406:3e0f58]  INFO -- : Followed  Relationship [miqaedb:/System/Event/RequestEvent/Request/request_created#create]
13298 [----] I, [2019-04-10T14:15:37.888145 #13327:3e0f58]  INFO -- : Q-task_id([r1000000000399_vm_retire_task_1000000000807]) User [admin] 

The service will continue to try to be retired unless the retirement date is extended, or some other action is taken.

Can you set the auto-approve to manual?

Thanks,
Tina

Cu response

---
Right now approval type is set to manual and request is only approved when certain conditions are met. Anyway when first retirement step comes there is two separate requests spawned. First one is denied. The second one retires the VM and the service.

I do believe there shall be only one service retirement request spawned, not two. Since service retirement workflow is one of the key points of the project, please solve this double retirement issue.
---

Logs and request numbers given in next comment

Hi Tina,

Did you get a chance to check the customers recent query?
Please suggest

Hi Niladri,

I suspect the issue is that the scheduler is running frequently, and even though a previous retirement request was not approved, it doesn't prevent the scheduler from initiating retirement again, and again for the same object.  If you want to stop or delay the retirement in the retirement auto approval state machine, you'll need to change the "retireability" of the object. Extending the retirement date is one way to accomplish this. Another option is to allow the object to enter retirement, and customize the retirement state machine for the desired result. Once the retirement has been started, the retirement state will be 'retiring' and the scheduler will ignore the object. 

More detail:
The scheduled retirement check method initiates the retirement process for any object(service, vm, stack) where the retires on date has passed. An miq_request is created to process the object retirement.  Since the normal request flow includes an approval process, the retirement approval state machine is where a request can be denied, or left unapproved(pending). The assumption is that some action would be taken on the object to change the retireablility based on the customers requirements. Extending the retires_on date for example.  There is no mechanism in place that will prevent the scheduled retirement from creating a new request for the same object if the object is still retireable which will result in multiple retirement requests for the same object.

Let me know if you have any questions.

Thanks,
Tina

Hi Niladri

The customer identified 2 service retirement requests for the same service: 
https://bugzilla.redhat.com/show_bug.cgi?id=1697600#c15

The customer states that the 2nd request (477) should not have been created and approved for retirement.

A combination of circumstances are responsible for the second request getting created/approved.

1. The requests were created 4 seconds apart.
Request 476 was created on 4/25/19 at 11:26:41
Request 477 was created on 4/25/19 at 11:26:45

2. Request 476 - validate_request runs at 11:26:47  <----- The first request entered the retirement approval state machine validate_request method AFTER the second request was created.

3. Scheduler runs(queues) RetirementManager.check at the following times around that time period: 
ws1 - 11:11, 11:16, 11:21
ws2 - 11:06, 11:26, 11:31 <------ I suspect the 2nd request (477) was created from the 11:26 retirement check.

4. For Request 476, The custom validate_request Automate method sets the service custom attributes(in addition to extending the retirement date) (see code below)   So, the service will have a custom attribute 'Queued for removal' set to 'True'

5. For Request 477, the service custom attribute, 'Queued for removal' is still true from #4, so the rest of the code is ignored, and the the state machine progresses to approve_request

6. For Request 477, the custom approve_request Automate method code checks if the custom attribute 'Queued for removal' is true, (and it is as a result of #4), so the second request is approved and proceeds to retirement.

7. One way to prevent this from happening is to update the custom validate_request Automate method to check if the service retires_on date is in the future, and deny the request if that is the case. This would have caught the second request and would have prevented it from being approved.  

This is custom code, so it would be up to the customer to decide how to handle the situation. 

Let me know if you have any questions.

Thanks,
Tina



The customer approve_request Automate method code snippet:
-----------------------------------------------------------------------------------------------
if service.custom_get('Queued for removal') == 'True'
	$evm.log("info", "Service queued for retirement")
	$evm.root["miq_request"].approve("admin", "Service queued for retirement")
end 


The customer validate_request Automate method:
-----------------------------------------------------------------------------------------------
service = $evm.vmdb(:service).find_by_id($evm.root['miq_request'].options[:src_ids][0].to_i)
#Investigative_Debugging::Discovery::ObjectWalker.walk_objects

if service.custom_get('Queued for removal') == 'False'
	service.custom_set('Queued for removal', 'True')
  	if $evm.object['retirement_units'] == 'days'
    	service.retires_on = (Time.now + $evm.object['retirement_length'].days).to_s
    elsif $evm.object['retirement_units'] == 'minutes'
  		service.retires_on = (Time.now + $evm.object['retirement_length'].minutes).to_s
    end
  	$evm.root['ae_reason'] = 'auto_retire_schedule'
    $evm.root['ae_result'] = 'error'
  	#exit MIQ_ABORT
end