Red Hat Bugzilla – Bug 1254905
[AWS::CloudFormation::Errors::Throttling]: Rate exceeded Method:[rescue in block in refresh] in evm.log
Last modified: 2015-11-11 15:16:49 EST
Description of problem:
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.Provision an ec2 instance.
ERROR -- : [AWS::CloudFormation::Errors::Throttling]: Rate exceeded Method:[rescue in block in refresh] in evm.log
erver: , Ident: [generic], Target id: , Instance id: , Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: , Priority: , State: [ready], Deliver On: , Data: , Args: 
[----] E, [2015-08-19T02:24:15.412307 #5938:445ea8] ERROR -- : MIQ(EmsRefresh::Refreshers::Ec2Refresher.refresh) EMS: [ec2], id:  Refresh failed
[----] E, [2015-08-19T02:24:15.412561 #5938:445ea8] ERROR -- : [AWS::CloudFormation::Errors::Throttling]: Rate exceeded Method:[rescue in block in refresh]
[----] E, [2015-08-19T02:24:15.412756 #5938:445ea8] ERROR -- : /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:375:in `return_or_raise'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:119:in `block in get_stacks'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:157:in `block in process_collection'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/collection.rb:48:in `block in each'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/ec2.rb:32:in `block in ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:20:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:80:in `block in refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:356:in `block in deliver'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/var/www/miq/vmdb/lib/workers/worker_base.rb:323:in `block in do_work_loop'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:3:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `<top (required)>'
[----] E, [2015-08-19T02:24:15.412844 #5938:445ea8] ERROR -- : MIQ(EmsRefresh::Refreshers::Ec2Refresher.refresh) EMS: [ec2], id:  Unable to perform refresh for the following targets:
[----] E, [2015-08-19T02:24:15.413026 #5938:445ea8] ERROR -- : --- EmsAmazon [ec2] id 
I need to know some specifics about the environment where this happened and under what conditions.
This happened during refresh, which definitely should *not* be a heavy number of requests to AWS.
A few questions:
1. How many stacks / cloudformation templates are in this AWS environment?
2. How many times was refresh kicked off against this environment?
3. Are there logs or other information that can be gathered? For instance, the error claims that the "rate" was "exceeded". What rate? And, by how much was it exceeded?
4. How reproducible is this? If this happened once, and doesn't happen again, then I feel pretty strongly about dropping the priority.
The proposed solution here is to just capture the error during refresh and then log a warning indicating that we were unable to collect orchestration stacks from AWS.
Answers to your queries:
1.The AWS environment setup is on appliance https://10.16.5.162/ with 2 stacks and 586 images
2.Not sure about how many time it was refreshed.
3.The above error is observed in the log dated "evm.log-20150819" on appliance https://10.16.5.162/
4.This issue was observed only once.
(In reply to Aziza Karol from comment #3)
> Answers to your queries:
> 1.The AWS environment setup is on appliance https://10.16.5.162/ with 2
> stacks and 586 images
> 2.Not sure about how many time it was refreshed.
> 3.The above error is observed in the log dated "evm.log-20150819" on
> appliance https://10.16.5.162/
> 4.This issue was observed only once.
Thank you for the reply Aziza.
Dave, Since Aziza has reported only seeing this issue once I think the
priority should be dropped to medium or even low. Do you agree?
Lets close it and reopen if we see it again.