Red Hat Bugzilla – Bug 451798
ec2 status for terminated and removed job not handled properly
Last modified: 2011-01-07 12:58:27 EST
EC2's DescribeInstances command keeps instances in the terminated state around
for a set period of time, after that period they are no longer reported and can
result in either no response or InvalidInstanceId.NotFound (former is observed).
If the latter is reported, the amazon-gahp returns '1'
'0' '' '' '', not InvalidInstanceId.NotFound to the gridmanager.
This behavior is intentional. The amazon_gahp will turn InvalidInstanceID.NotFound errors into successful operations for AMAZON_VM_STATUS and AMAZON_VM_STOP commands. This was done to simplify the error-handling in the gridmanager. If we don't want the gahp to eat these errors, we'll have to modify the gridmanager to recognize them and react appropriately.
I think my description may have been poorly worded. The issue was that instances that are no longer reported by EC2 as even existing were not being handled properly. This is the case of EC2 removing all knowledge of an instance before AMAZON_VM_STATUS can be sent to query the instance. The semantics should be that in such a case the instance is considered terminated, but that wasn't happening. Testing this means withholding AMAZON_VM_STATUS commands until the instance has moved into the terminated state and been flushed from the output of ec2-describe-instances.
FYI: amazon-gahp was renamed to amazon_gahp
in version 7.2.0-0.8 (7.2.0 pre-release)
If this appears again or more frequently it can be re-opened.