Bug 1673848 - AWS_EC2 events are not received by CloudForms
Summary: AWS_EC2 events are not received by CloudForms
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers
Version: 5.9.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: GA
: 5.10.2
Assignee: Alexander Zagaynov
QA Contact: Matouš Mojžíš
Red Hat CloudForms Documentation
URL:
Whiteboard:
Depends On:
Blocks: 1679295
TreeView+ depends on / blocked
 
Reported: 2019-02-08 09:54 UTC by Niladri Roy
Modified: 2019-11-06 20:00 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-03-20 12:21:42 UTC
Category: Bug
Cloudforms Team: AWS
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
reproducer instance stop (10.75 KB, text/plain)
2019-02-08 10:10 UTC, Niladri Roy
no flags Details
reproducer instance start (12.22 KB, text/plain)
2019-02-08 10:11 UTC, Niladri Roy
no flags Details
reproducer instance terminate (10.33 KB, text/plain)
2019-02-08 10:11 UTC, Niladri Roy
no flags Details

Description Niladri Roy 2019-02-08 09:54:31 UTC
Description of problem:
After adding AWS provider, we do receive events with Type like event_type=>AWS_API_*
eg. "AWS_API_CALL_RunInstances",
 
But we do not get any events like event_type=>AWS_EC2_*
eg. "AWS_EC2_Instance_Running".
 
 
Version-Release number of selected component (if applicable):
5.9.5
 
How reproducible:
Always
 
Steps to Reproduce:
1. Add AWS Provider
 
2. Enable AWS provider to send events to CloudForms
("4.3.6. Enabling Amazon EC2 Events
" : https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.6/html-single/managing_providers/#amazon-provider-aws-events)
 
3. Provision a new instance in AWS (not from CloudForms)
4. Power off the instance, observe the incoming events in evm.log
5. Power on the instance, observe the events
6. Terminate the instance 
 
Actual results:
AWS_EC2_* are not seen in evm.logs
 
Expected results:
CloudForms should get AWS_EC2_* events
 
Additional info:
1. Attaching the log excerpt files of my reproducer environment
2. I don't see the EC2_Instance_State_change_Notification_* events as well
3. My customer has been able to get the EC2_Instance_State_change_Notification_* events by the below workaround

"  A new CloudWatch rule will be defined to stream all the EC2 configuration update events to the AWSConfig_topic. "

Comment 3 Niladri Roy 2019-02-08 10:10:19 UTC
Created attachment 1528018 [details]
reproducer instance stop

Comment 4 Niladri Roy 2019-02-08 10:11:09 UTC
Created attachment 1528019 [details]
reproducer instance start

Comment 5 Niladri Roy 2019-02-08 10:11:44 UTC
Created attachment 1528020 [details]
reproducer instance terminate

Comment 6 Alexander Zagaynov 2019-02-11 20:08:09 UTC
> - AWS_EC2_* and EC2_Instance_State_change_Notification_* events are created
> by AWS Config service ?
Yes, here: https://github.com/ManageIQ/manageiq-providers-amazon/blob/068c4eb30e38580cdbe8a706d1310ef9f7613bda/app/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb#L243-L266


> * Then why do we only see the AWS_API_CALL_* events in the evm.log file?
Try to set log level to "debug" and look for "Parsed event from SNS Message not recognized" string.
https://github.com/ManageIQ/manageiq-providers-amazon/blob/068c4eb30e38580cdbe8a706d1310ef9f7613bda/app/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb#L198


> has created a new cloudwatch rule for the EC2 state change and in their logs we can see incoming EC2_Instance_State_change_Notification_* events
> * Is this extra step a necessary step? it is not mentioned in our documentation
Not sure, but should be not. I'll check.


> - We see the below error in the logs for which this seems to be the solution
> [https://access.redhat.com/solutions/2532401] 
> * Is this error in any way related for not receiving AWS_EC2_* events?
No, it's not about this.


You can also see a brief description of how it works here: 
https://github.com/ManageIQ/manageiq-providers-amazon/blob/068c4eb30e38580cdbe8a706d1310ef9f7613bda/app/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream.rb#L2-L7

Comment 7 Niladri Roy 2019-02-12 07:29:46 UTC
Hello Alex,

Thanks for you reply. I have forwarded your suggestion to our customer. I'll wait for their reply with new logs.

Meanwhile, I enabled debug logging in my reproducer environment and I could see the AWS_EC2_* events this time. These events came in after 2-3 minutes of the AWS_API_CALL events.

[----] I, [2019-02-12T12:49:10.030730 #1286:a8c8c0]  INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::EventCatcher::Stream#) Found SNS Message with message type AWS_EC2_Instance_stopped coming from 
[----] I, [2019-02-12T12:49:10.030851 #1286:a8c8c0]  INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::EventCatcher::Stream#) Parsed event from SNS Message AWS_EC2_Instance_stopped coming from 


So I'm not sure what the issue is, as if you see customer logs, the AWS_EC2 events are not present. 
Additionally, when I reported the BZ, that time too the AWS_EC2 events were not seen in my reproducer environment.

I also turned off debug logging and tested, I still see the AWS_EC2 events.

Comment 8 Alexander Zagaynov 2019-02-13 14:50:02 UTC
Hi Niladri, do you have any updates?

Comment 9 Niladri Roy 2019-02-14 03:29:19 UTC
No update from Cu, I'm following up

Comment 10 Alexander Zagaynov 2019-02-14 03:36:09 UTC
Got it, thank you!
I'm lowering priority now, as it seems that it happened accidentally, and not reproducible.
Feel free to change it back if I'm wrong.

Comment 11 Alexander Zagaynov 2019-02-21 12:21:12 UTC
Since there was no updates for more than a week, I'm going to close current BZ tomorrow.

Comment 21 dmetzger 2019-03-15 19:43:05 UTC
Removing from 5.10.2 per input from CEE. Not reproducible and lack of response from customer.

Comment 22 dmetzger 2019-03-15 19:49:26 UTC
Previous comment / change was meant for a different ticket, putting back in 5.10.2 appropriate disposition.


Note You need to log in before you can comment on or make changes to this bug.