Description of problem: When AWS has API failures in their regions, the appliance generates a exception from /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79#perf_capture_data_amazon' See log segment below. This was originally reported in BZ https://bugzilla.redhat.com/show_bug.cgi?id=1250444 which is really reporting 2 separate issues. This BZ will be used to track the [undefined method `keys' for nil:NilClass] exception. Version-Release number of selected component (if applicable): master.20150803085357_e08ddc8 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: AWS Status Issue: ================= Amazon Elastic Compute Cloud (N. Virginia) Increased API Error Rates less 12:22 PM PST We are investigating increased API error rates in the US-EAST-1 Region. 12:53 PM PST We can confirm increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted. 1:58 PM PST We continue to work toward resolving increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted. 2:51 PM PST We are beginning to see recovery for EC2 API error rates in the US-EAST-1 Region. evm.log: ======== [----] E, [2015-02-24T20:47:30.038345 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture' /opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver' /home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout' /opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work' /opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run' /opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start' /opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker' /opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>' script/rails:30:in `require' script/rails:30:in `<main>' [----] E, [2015-02-24T20:47:30.038694 #20069:80beac] ERROR -- : MIQ(MiqQueue.deliver) Message id: [4135889], Error: [undefined method `keys' for nil:NilClass] [----] E, [2015-02-24T20:47:30.038828 #20069:80beac] ERROR -- : [NoMethodError]: undefined method `keys' for nil:NilClass Method:[rescue in deliver] [----] E, [2015-02-24T20:47:30.038913 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture' /opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver' /home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout' /opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work' /opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run' /opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start' /opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker' /opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>' script/rails:30:in `require' script/rails:30:in `<main>'
https://github.com/ManageIQ/manageiq/pull/4634
https://github.com/ManageIQ/manageiq/pull/4628
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/d76eb1c550f7cbf9a32be19c3e37d676d2e80dcc commit d76eb1c550f7cbf9a32be19c3e37d676d2e80dcc Author: Joe VLcek <jvlcek> AuthorDate: Thu Oct 1 16:49:29 2015 -0400 Commit: Joe VLcek <jvlcek> CommitDate: Fri Oct 2 15:58:22 2015 -0400 Handle no AWS metrics collected When AWS has an API failures in their regions, the appliance generates a NoMethodError exception The solution it to ignore empty metrics collections. https://bugzilla.redhat.com/show_bug.cgi?id=1256404 .../amazon/cloud_manager/metrics_capture.rb | 4 +++- spec/factories/metrics_capture_amazon.rb | 12 ++++++++++ spec/factories/vm_amazon.rb | 5 ++++ .../amazon/cloud_manager/metrics_capture_spec.rb | 28 ++++++++++++++++++++++ 4 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 spec/factories/metrics_capture_amazon.rb create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/metrics_capture_spec.rb
Joe, any guesses as to how we can fake this?
You could try disabling the network on the appliance after metrics collection has started to emulate an AWS API failures but this may be problematic. I have added new spec tests to exercise the code change for the fix for this BZ. Perhaps the best thing to do is just ensure the appliance correctly collect metrics and rely on the specs to exercise the AWS API failure.
Verified that ec2 metrics collection is still happening with the fix. Verified in 5.5.0.13
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2551