Bug 1207865 - some EC2 policy event not triggering actions
Summary: some EC2 policy event not triggering actions
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Control
Version: 5.4.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: GA
: 5.4.0
Assignee: Bill Wei
QA Contact: Nandini Chandra
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-03-31 22:09 UTC by Nandini Chandra
Modified: 2017-02-17 17:04 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-06-16 12:56:32 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:1100 0 normal SHIPPED_LIVE CFME 5.4.0 bug fixes, and enhancement update 2015-06-16 16:28:42 UTC

Description Nandini Chandra 2015-03-31 22:09:02 UTC
Description of problem:
-----------------------
The /var/www/miq/vmdb/config/event_handling.tmpl.yml has these entries corresponding to EC2 instances.The following EC2 EMS events are expected to raise policy events.


 AWS_EC2_Instance_CREATE:
  - policy:
    - src_vm
    - vm_create
  - refresh:
    - ems
   AWS_EC2_Instance_running:
  - policy:
    - src_vm
    - vm_start
  - refresh:
    - ems
  AWS_EC2_Instance_stopped:
  - policy:
    - src_vm
    - vm_power_off
  - refresh:
    - ems
  AWS_EC2_Instance_shutting-down:
  - policy:
    - src_vm
    - vm_power_off
  - refresh:
    - ems


When a VM control policy with vm_power_off event is defined and is assigned to an EC2 instance,it doesn't  trigger the action defined in the control policy.


Version-Release number of selected component (if applicable):
-----------------
5.4.0.0.14


How reproducible:
----------------
Always


Steps to Reproduce:
------------------
1.Define a VM control policy with vm_power_off event;Action could be anything appropriate,eg:Remove Tags.Assign the control policy to an EC2 instance
2.Stop the VM either through CFME or through the AWS console.When the vm_power_off event occurs,it doesn't trigger the defined policy action.



Actual results:
--------------
The vm_power_off policy event doesn't trigger an action when the control policy is assigned to EC2 instances.


Expected results:
--------------
The vm_power_off policy event should trigger an action when the control policy is assigned to EC2 instances.


Additional info:
---------------

Comment 2 Nandini Chandra 2015-03-31 22:24:24 UTC
Additionally,
1)For EC2,a control policy with request_vm_reboot_guest event triggers the defined action.I'm curious to know how this is happening when the 
/var/www/miq/vmdb/config/event_handling.tmpl.yml doesn't even list this policy event for EC2.

[----] I, [2015-03-31T14:29:20.487613 #7283:11a5ea4]  INFO -- : MIQ(policy-enforce_policy): Event: [request_vm_reboot_guest], To: [nachandr-ec2-f20]

[----] I, [2015-03-31T14:29:20.658343 #7283:11a5ea4]  INFO -- : MIQ(policy-enforce_policy): Resolving policy [Instance:request_vm_reboot_guest]...

[----] I, [2015-03-31T14:29:21.215998 #7283:11a5ea4]  INFO -- : MIQ(action-invoke) Invoking action [Tag] for successful policy [Instance:request_vm_reboot_guest], event: [VM Guest Reboot Request], entity name: [nachandr-ec2-f20], entity type: [VmAmazon], sequence: [1], synchronous? [true]

[----] I, [2015-03-31T14:29:21.216406 #7283:11a5ea4]  INFO -- : MIQ(action_tag): Applying tags [["/managed/location/chicago"]] to [(VmAmazon) nachandr-ec2-f20]

[----] I, [2015-03-31T14:29:21.259462 #7283:11a5ea4]  INFO -- : MIQ(policy-enforce_policy): Event: [request_assign_company_tag], To: [nachandr-ec2-f20]

2)Also,this entry in the yaml file indicates that the policy event associated with the EMS AWS_EC2_Instance_running event  is vm_start.However,a control policy with vm_start doesn't trigger an action,although the request_vm_start event does trigger an action.How is that happening?Should I open a BZ so that the yaml file can be updated?

 AWS_EC2_Instance_running:
  - policy:
    - src_vm
    - vm_start
  - refresh:
    - ems

Comment 3 Nandini Chandra 2015-03-31 22:52:30 UTC
This is a follow up to my comment#2.The messages logged to policy.log show that 'request_vm_start' triggers an action.

Snippet from policy.log:

[----] I, [2015-03-31T15:29:27.414589 #7286:68be9c]  INFO -- : MIQ(policy-enforce_policy): Event: [request_vm_start], To: [nachandr-ec2-f20]

[----] I, [2015-03-31T15:29:27.583237 #7286:68be9c]  INFO -- : MIQ(policy-enforce_policy): Resolving policy [Instance power on]...

[----] I, [2015-03-31T15:29:27.970266 #7286:68be9c]  INFO -- : MIQ(action-invoke) Invoking action [Tag] for successful policy [Instance power on], event: [VM Power On Request], entity name: [nachandr-ec2-f20], entity type: [VmAmazon], sequence: [1], synchronous? [true]

[----] I, [2015-03-31T15:29:27.973547 #7286:68be9c]  INFO -- : MIQ(action_tag): Applying tags [["/managed/location/chicago"]] to [(VmAmazon) nachandr-ec2-f20]

Comment 4 Nandini Chandra 2015-03-31 22:53:26 UTC
Here's a summary of the events that work and don't work.

Events defined in the yaml file and that trigger action properly:
1)vm_create

Events defined in the yaml file and that *don't* trigger action:
1)vm_power_off
2)vm_start

Events not defined in the yaml file,but that still trigger cation:
1)request_vm_reboot_guest
2)request_vm_start

Comment 5 Greg McCullough 2015-04-02 16:10:44 UTC
Nandini - Have you validated that you are getting any events from Amazon?  The amazon config service has to be enabled on the AWS side.

For the ones that are working:
vm_create is generated internally and should be removed from the event_handling.tmpl.yml file.

For most places in that file where you would have a vm_create call you will see a comment like this:
  # Ignore <event_name> for policy for now the vm_create event will be raised when the VM is created

This is because when this event is received from the provider the VM object does not exist yet in our database and there is no object to raise the event against.  Instead we raise the vm_create event as part of the post-creation processing for VM objects.

The request_* events are also internally raised events.  The purpose of these events is to allow policy to prevent the action from continuing before calling the provider.  Therefore these would not be linked to events coming from the provider.

So my current take away is:
1) It does not seem like you are getting any events from Amazon.
2) vm_create should be removed from the event_handling.tmpl.yml file for the AWS event and updated with the same basic comment to make clear why it is not provided as part of the configuration.
3) Request_* events are working as expected.

Please confirm.

Comment 6 Nandini Chandra 2015-04-03 04:02:37 UTC
1)My appliance is managing one EC2 provider.Going by this,I do believe the appliance is getting events from Amazon.

vmdb_production=# select event_type from ems_events\x\g\x
Expanded display is on.
-[ RECORD 1 ]----------------------------
event_type | AWS_EC2_Instance_running
-[ RECORD 2 ]----------------------------
event_type | AWS_EC2_Volume_CREATE
-[ RECORD 3 ]----------------------------
event_type | AWS_EC2_SecurityGroup_UPDATE
-[ RECORD 4 ]----------------------------
event_type | AWS_EC2_Instance_CREATE
-[ RECORD 5 ]----------------------------
event_type | AWS_EC2_SecurityGroup_UPDATE
-[ RECORD 6 ]----------------------------
event_type | AWS_EC2_Volume_CREATE
-[ RECORD 7 ]----------------------------
event_type | AWS_EC2_Instance_CREATE
-[ RECORD 8 ]----------------------------
event_type | AWS_EC2_Instance_running
-[ RECORD 9 ]----------------------------
event_type | AWS_EC2_SecurityGroup_UPDATE
-[ RECORD 10 ]---------------------------
event_type | AWS_EC2_Instance_DELETE
-[ RECORD 11 ]---------------------------
event_type | AWS_EC2_Instance_DELETE


2)Additionally,when I stop an ec2 instance from *CFME*,vm_power_off event doesn't get logged to policy.log.

Comment 7 Nandini Chandra 2015-04-06 17:37:50 UTC
(In reply to Greg McCullough from comment #5)
> Nandini - Have you validated that you are getting any events from Amazon? 
> The amazon config service has to be enabled on the AWS side.
> 

Yes,the Amazon Config service is running.

Comment 8 Greg McCullough 2015-04-06 20:57:09 UTC
Bill - Looks like we might need to add AWS_EC2_Instance_DELETE to the list of processed events from AWS.  The event_handling.tmpl.yml file currently defines AWS_EC2_Instance_stopped and AWS_EC2_Instance_shutting-down as provider events that would generate a vm_power_off policy event.

(I also think AWS_EC2_Instance_CREATE should be removed as per comment #5.)

The odd thing is that they are getting the AWS_EC2_Instance_running event but not seeing the associated vm_start policy call.  That could mean no events are being processed and is hopefully just a configuration issue.

Comment 11 Bill Wei 2015-05-04 21:47:43 UTC
vm_start is working.
There is a typo in the event_handling.tmp.yml. It should be vm_poweroff, not vm_power_off.

I removed vm_create and add AWS_EC2_Instance_DELETE.

Comment 12 Bill Wei 2015-05-04 21:54:57 UTC
There could be a long delay (in minutes) between the provider and manageiq event triggering. Be patient to see the result!

Comment 13 Nandini Chandra 2015-05-04 23:32:05 UTC
(In reply to Greg McCullough from comment #8)
> Bill - Looks like we might need to add AWS_EC2_Instance_DELETE to the list
> of processed events from AWS.  The event_handling.tmpl.yml file currently
> defines AWS_EC2_Instance_stopped and AWS_EC2_Instance_shutting-down as
> provider events that would generate a vm_power_off policy event.
> 
> (I also think AWS_EC2_Instance_CREATE should be removed as per comment #5.)
> 
> The odd thing is that they are getting the AWS_EC2_Instance_running event
> but not seeing the associated vm_start policy call.  That could mean no
> events are being processed and is hopefully just a configuration issue.

I've filed these bugs to address these issues:

https://bugzilla.redhat.com/show_bug.cgi?id=1217597 Remove vm_create for AWS_EC2_Instance_CREATE in event_handling.tmpl.yml

https://bugzilla.redhat.com/show_bug.cgi?id=1217601  Add AWS_EC2_Instance_DELETE to the list of processed events from AWS

Comment 14 CFME Bot 2015-05-05 18:01:11 UTC
New commit detected on manageiq/master:
https://github.com/ManageIQ/manageiq/commit/42f61c707540f477ff8ebf1e7daac8cd57896389

commit 42f61c707540f477ff8ebf1e7daac8cd57896389
Author:     Bill Wei <bilwei>
AuthorDate: Mon May 4 17:49:53 2015 -0400
Commit:     Bill Wei <bilwei>
CommitDate: Mon May 4 17:52:23 2015 -0400

    Fix AWS event mappings
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1207865

 vmdb/config/event_handling.tmpl.yml | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

Comment 15 Bill Wei 2015-05-05 18:37:56 UTC
How to interpret the policy log:
Each event has a name and a description. The log uses both name and description in different places.

Event name vm_start, description VM Power On

[----] I, [2015-05-04T17:04:53.967396 #32465:3fd4c9c65bf0]  INFO -- : MIQ(policy-enforce_policy): Event: [vm_start], To: [bill_test]
[----] I, [2015-05-04T17:04:53.983222 #32465:3fd4c9c65bf0]  INFO -- : MIQ(policy-enforce_policy): Resolving policy [aws powerop]...
[----] I, [2015-05-04T17:04:54.072356 #32465:3fd4c9c65bf0]  INFO -- : MIQ(action-invoke) Invoking action [Generate log message] for successful policy [aws powerop], event: [VM Power On], entity name: [bill_test], entity type: [VmAmazon], sequence: [1], synchronous? [true]

Event name vm_poweroff, description VM Power Off

[----] I, [2015-05-04T17:38:52.659303 #56775:3fef84851bec]  INFO -- : MIQ(policy-enforce_policy): Event: [vm_poweroff], To: [bill_test]
[----] I, [2015-05-04T17:38:52.848334 #56775:3fef84851bec]  INFO -- : MIQ(policy-enforce_policy): Resolving policy [aws powerop]...
[----] I, [2015-05-04T17:38:52.888006 #56775:3fef84851bec]  INFO -- : MIQ(action-invoke) Invoking action [Generate log message] for successful policy [aws powerop], event: [VM Power Off], entity name: [bill_test], entity type: [VmAmazon], sequence: [1], synchronous? [true]

Comment 17 Nandini Chandra 2015-05-14 17:47:40 UTC
Verified that when a VM control policy with vm_poweroff event is defined and is assigned to an EC2 instance,it triggers the action defined in the control policy.

Verified in 5.4.0.0.26

Comment 19 errata-xmlrpc 2015-06-16 12:56:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-1100.html


Note You need to log in before you can comment on or make changes to this bug.