Bug 1256404

Summary:	Amazon provider fails with: [NoMethodError]: undefined method `keys' for nil:NilClass
Product:	Red Hat CloudForms Management Engine	Reporter:	Joe Vlcek <jvlcek>
Component:	Providers	Assignee:	Joe Vlcek <jvlcek>
Status:	CLOSED ERRATA	QA Contact:	Nandini Chandra <nachandr>
Severity:	medium	Docs Contact:
Priority:	high
Version:	5.5.0	CC:	dajohnso, jfrey, jhardy, jvlcek, mfeifer, obarenbo
Target Milestone:	GA
Target Release:	5.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	5.5.0.5	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-08 13:28:15 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Joe Vlcek 2015-08-24 13:40:45 UTC

Description of problem:
When AWS has API failures in their regions, the appliance generates a
exception from
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79#perf_capture_data_amazon'

See log segment below.

This was originally reported in BZ
https://bugzilla.redhat.com/show_bug.cgi?id=1250444
which is really reporting 2 separate issues.

This BZ will be used to track the [undefined method `keys' for nil:NilClass]
exception.




Version-Release number of selected component (if applicable):
master.20150803085357_e08ddc8

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
AWS Status Issue:
=================
Amazon Elastic Compute Cloud (N. Virginia)  Increased API Error Rates     less 
12:22 PM PST We are investigating increased API error rates in the US-EAST-1 Region.
12:53 PM PST We can confirm increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.
1:58 PM PST We continue to work toward resolving increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.
2:51 PM PST We are beginning to see recovery for EC2 API error rates in the US-EAST-1 Region.



evm.log:
========
[----] E, [2015-02-24T20:47:30.038345 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture'
/opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:30:in `require'
script/rails:30:in `<main>'
[----] E, [2015-02-24T20:47:30.038694 #20069:80beac] ERROR -- : MIQ(MiqQueue.deliver)    Message id: [4135889], Error: [undefined method `keys' for nil:NilClass]
[----] E, [2015-02-24T20:47:30.038828 #20069:80beac] ERROR -- : [NoMethodError]: undefined method `keys' for nil:NilClass  Method:[rescue in deliver]
[----] E, [2015-02-24T20:47:30.038913 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture'
/opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:30:in `require'
script/rails:30:in `<main>'

Comment 2 CFME Bot 2015-10-01 21:05:57 UTC

https://github.com/ManageIQ/manageiq/pull/4634

Comment 3 CFME Bot 2015-10-01 21:06:19 UTC

https://github.com/ManageIQ/manageiq/pull/4628

Comment 4 CFME Bot 2015-10-03 00:40:37 UTC

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/d76eb1c550f7cbf9a32be19c3e37d676d2e80dcc

commit d76eb1c550f7cbf9a32be19c3e37d676d2e80dcc
Author:     Joe VLcek <jvlcek>
AuthorDate: Thu Oct 1 16:49:29 2015 -0400
Commit:     Joe VLcek <jvlcek>
CommitDate: Fri Oct 2 15:58:22 2015 -0400

    Handle no AWS metrics collected
    
    When AWS has an API failures in their regions, the appliance generates a
    NoMethodError exception
    
    The solution it to ignore empty metrics collections.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1256404

 .../amazon/cloud_manager/metrics_capture.rb        |  4 +++-
 spec/factories/metrics_capture_amazon.rb           | 12 ++++++++++
 spec/factories/vm_amazon.rb                        |  5 ++++
 .../amazon/cloud_manager/metrics_capture_spec.rb   | 28 ++++++++++++++++++++++
 4 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 spec/factories/metrics_capture_amazon.rb
 create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/metrics_capture_spec.rb

Comment 5 Dave Johnson 2015-12-03 04:34:01 UTC

Joe, any guesses as to how we can fake this?

Comment 6 Joe Vlcek 2015-12-03 13:42:35 UTC

You could try disabling the network on the appliance after metrics collection has
started to emulate an AWS API failures but this may be problematic.

I have added new spec tests to exercise the code change for the fix for this
BZ. Perhaps the best thing to do is just ensure the appliance correctly collect
metrics and rely on the specs to exercise the AWS API failure.

Comment 8 Nandini Chandra 2015-12-03 22:13:39 UTC

Verified that ec2 metrics collection is still happening with the fix.

Verified in 5.5.0.13

Comment 10 errata-xmlrpc 2015-12-08 13:28:15 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551