Bug 1254905 - [AWS::CloudFormation::Errors::Throttling]: Rate exceeded Method:[rescue in block in refresh] in evm.log
[AWS::CloudFormation::Errors::Throttling]: Rate exceeded Method:[rescue in b...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
5.4.0
Unspecified Unspecified
high Severity medium
: GA
: 5.5.0
Assigned To: Joe Vlcek
Dave Johnson
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-19 04:41 EDT by Aziza Karol
Modified: 2015-11-11 15:16 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-09-28 15:00:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Aziza Karol 2015-08-19 04:41:18 EDT
Description of problem:


Version-Release number of selected component (if applicable):
5.4.1

How reproducible:


Steps to Reproduce:
1.Provision an ec2 instance.
2.
3.

Actual results:
ERROR -- : [AWS::CloudFormation::Errors::Throttling]: Rate exceeded  Method:[rescue in block in refresh] in evm.log

Expected results:


Additional info:
evm.log
erver: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: []
[----] E, [2015-08-19T02:24:15.412307 #5938:445ea8] ERROR -- : MIQ(EmsRefresh::Refreshers::Ec2Refresher.refresh) EMS: [ec2], id: [3] Refresh failed
[----] E, [2015-08-19T02:24:15.412561 #5938:445ea8] ERROR -- : [AWS::CloudFormation::Errors::Throttling]: Rate exceeded  Method:[rescue in block in refresh]
[----] E, [2015-08-19T02:24:15.412756 #5938:445ea8] ERROR -- : /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:375:in `return_or_raise'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `list_stack_resources'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/cloud_formation/stack_resource_summary_collection.rb:55:in `_each_item'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection/with_next_token.rb:50:in `_each_batch'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:80:in `each_batch'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:47:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:469:in `to_a'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:469:in `find_stack_resources'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:428:in `parse_stack'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:119:in `block in get_stacks'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:157:in `block in process_collection'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:48:in `each'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:48:in `block in each'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection/with_next_token.rb:77:in `_each_batch'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:80:in `each_batch'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:47:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:156:in `process_collection'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:119:in `get_stacks'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:32:in `block in ems_inv_to_hashes'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core.rb:617:in `memoize'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:28:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:6:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ec2_refresher.rb:6:in `parse_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:20:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:8:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/base_refresher.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:80:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:356:in `block in deliver'
/opt/rh/ruby200/root/usr/share/ruby/timeout.rb:66:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:352:in `deliver'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/var/www/miq/vmdb/lib/workers/worker_base.rb:323:in `block in do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:320:in `loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:320:in `do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:141:in `run'
/var/www/miq/vmdb/lib/workers/worker_base.rb:122:in `start'
/var/www/miq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:3:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `eval'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `require'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'
[----] E, [2015-08-19T02:24:15.412844 #5938:445ea8] ERROR -- : MIQ(EmsRefresh::Refreshers::Ec2Refresher.refresh) EMS: [ec2], id: [3] Unable to perform refresh for the following targets:
[----] E, [2015-08-19T02:24:15.413026 #5938:445ea8] ERROR -- :  --- EmsAmazon [ec2] id [3]
Comment 2 Greg Blomquist 2015-08-21 19:59:18 EDT
I need to know some specifics about the environment where this happened and under what conditions.

This happened during refresh, which definitely should *not* be a heavy number of requests to AWS.

A few questions:

1. How many stacks / cloudformation templates are in this AWS environment?

2. How many times was refresh kicked off against this environment?

3. Are there logs or other information that can be gathered?  For instance, the error claims that the "rate" was "exceeded".  What rate?  And, by how much was it exceeded?

4. How reproducible is this?  If this happened once, and doesn't happen again, then I feel pretty strongly about dropping the priority.

The proposed solution here is to just capture the error during refresh and then log a warning indicating that we were unable to collect orchestration stacks from AWS.
Comment 3 Aziza Karol 2015-08-27 04:05:14 EDT
Greg,

Answers to your queries:

1.The AWS environment setup is on appliance https://10.16.5.162/ with 2 stacks and 586 images 

2.Not sure about how many time it was refreshed.

3.The above error is observed in the log dated "evm.log-20150819" on appliance https://10.16.5.162/

4.This issue was observed only once.


Thanks,
Aziza
Comment 4 Joe Vlcek 2015-09-28 10:02:14 EDT
(In reply to Aziza Karol from comment #3)
> Greg,
> 
> Answers to your queries:
> 
> 1.The AWS environment setup is on appliance https://10.16.5.162/ with 2
> stacks and 586 images 
> 
> 2.Not sure about how many time it was refreshed.
> 
> 3.The above error is observed in the log dated "evm.log-20150819" on
> appliance https://10.16.5.162/
> 
> 4.This issue was observed only once.
> 
> 
> Thanks,
> Aziza

Thank you for the reply Aziza.

Dave, Since Aziza has reported only seeing this issue once I think the
priority should be dropped to medium or even low. Do you agree?

JoeV
Comment 5 Dave Johnson 2015-09-28 15:00:26 EDT
Lets close it and reopen if we see it again.

Note You need to log in before you can comment on or make changes to this bug.