Bug 1256404 - Amazon provider fails with: [NoMethodError]: undefined method `keys' for nil:NilClass
Amazon provider fails with: [NoMethodError]: undefined method `keys' for nil:...
Status: CLOSED ERRATA
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
5.5.0
Unspecified Unspecified
high Severity medium
: GA
: 5.5.0
Assigned To: Joe Vlcek
Nandini Chandra
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-24 09:40 EDT by Joe Vlcek
Modified: 2015-12-08 08:28 EST (History)
6 users (show)

See Also:
Fixed In Version: 5.5.0.5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-08 08:28:15 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Joe Vlcek 2015-08-24 09:40:45 EDT
Description of problem:
When AWS has API failures in their regions, the appliance generates a
exception from
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79#perf_capture_data_amazon'

See log segment below.

This was originally reported in BZ
https://bugzilla.redhat.com/show_bug.cgi?id=1250444
which is really reporting 2 separate issues.

This BZ will be used to track the [undefined method `keys' for nil:NilClass]
exception.




Version-Release number of selected component (if applicable):
master.20150803085357_e08ddc8

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
AWS Status Issue:
=================
Amazon Elastic Compute Cloud (N. Virginia)  Increased API Error Rates     less 
12:22 PM PST We are investigating increased API error rates in the US-EAST-1 Region.
12:53 PM PST We can confirm increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.
1:58 PM PST We continue to work toward resolving increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.
2:51 PM PST We are beginning to see recovery for EC2 API error rates in the US-EAST-1 Region.



evm.log:
========
[----] E, [2015-02-24T20:47:30.038345 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture'
/opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:30:in `require'
script/rails:30:in `<main>'
[----] E, [2015-02-24T20:47:30.038694 #20069:80beac] ERROR -- : MIQ(MiqQueue.deliver)    Message id: [4135889], Error: [undefined method `keys' for nil:NilClass]
[----] E, [2015-02-24T20:47:30.038828 #20069:80beac] ERROR -- : [NoMethodError]: undefined method `keys' for nil:NilClass  Method:[rescue in deliver]
[----] E, [2015-02-24T20:47:30.038913 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture'
/opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:30:in `require'
script/rails:30:in `<main>'
Comment 4 CFME Bot 2015-10-02 20:40:37 EDT
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/d76eb1c550f7cbf9a32be19c3e37d676d2e80dcc

commit d76eb1c550f7cbf9a32be19c3e37d676d2e80dcc
Author:     Joe VLcek <jvlcek@redhat.com>
AuthorDate: Thu Oct 1 16:49:29 2015 -0400
Commit:     Joe VLcek <jvlcek@redhat.com>
CommitDate: Fri Oct 2 15:58:22 2015 -0400

    Handle no AWS metrics collected
    
    When AWS has an API failures in their regions, the appliance generates a
    NoMethodError exception
    
    The solution it to ignore empty metrics collections.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1256404

 .../amazon/cloud_manager/metrics_capture.rb        |  4 +++-
 spec/factories/metrics_capture_amazon.rb           | 12 ++++++++++
 spec/factories/vm_amazon.rb                        |  5 ++++
 .../amazon/cloud_manager/metrics_capture_spec.rb   | 28 ++++++++++++++++++++++
 4 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 spec/factories/metrics_capture_amazon.rb
 create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/metrics_capture_spec.rb
Comment 5 Dave Johnson 2015-12-02 23:34:01 EST
Joe, any guesses as to how we can fake this?
Comment 6 Joe Vlcek 2015-12-03 08:42:35 EST
You could try disabling the network on the appliance after metrics collection has
started to emulate an AWS API failures but this may be problematic.

I have added new spec tests to exercise the code change for the fix for this
BZ. Perhaps the best thing to do is just ensure the appliance correctly collect
metrics and rely on the specs to exercise the AWS API failure.
Comment 8 Nandini Chandra 2015-12-03 17:13:39 EST
Verified that ec2 metrics collection is still happening with the fix.

Verified in 5.5.0.13
Comment 10 errata-xmlrpc 2015-12-08 08:28:15 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551

Note You need to log in before you can comment on or make changes to this bug.