Bug 1250444 - Log directory filling up when AWS was having API issues
Log directory filling up when AWS was having API issues
Status: CLOSED ERRATA
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
5.5.0
Unspecified Unspecified
high Severity medium
: GA
: 5.5.0
Assigned To: Joe Vlcek
Ramesh A
:
: 1196853 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-05 07:31 EDT by Ramesh A
Modified: 2015-12-08 08:26 EST (History)
8 users (show)

See Also:
Fixed In Version: 5.5.0.10
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2015-12-08 08:26:00 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ramesh A 2015-08-05 07:31:41 EDT
Description of problem:
Whenever AWS has API issues in their regions, Appliance log files are flooded with error messages and then crash itself as appliance keep getting AWS API errors.

Version-Release number of selected component (if applicable):
master.20150803085357_e08ddc8

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
AWS Status Issue:
=================
Amazon Elastic Compute Cloud (N. Virginia)  Increased API Error Rates     less 
12:22 PM PST We are investigating increased API error rates in the US-EAST-1 Region.
12:53 PM PST We can confirm increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.
1:58 PM PST We continue to work toward resolving increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted.
2:51 PM PST We are beginning to see recovery for EC2 API error rates in the US-EAST-1 Region.



evm.log:
========
[----] E, [2015-02-24T20:47:30.038345 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture'
/opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:30:in `require'
script/rails:30:in `<main>'
[----] E, [2015-02-24T20:47:30.038694 #20069:80beac] ERROR -- : MIQ(MiqQueue.deliver)    Message id: [4135889], Error: [undefined method `keys' for nil:NilClass]
[----] E, [2015-02-24T20:47:30.038828 #20069:80beac] ERROR -- : [NoMethodError]: undefined method `keys' for nil:NilClass  Method:[rescue in deliver]
[----] E, [2015-02-24T20:47:30.038913 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store'
/opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block'
/opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture'
/opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver'
/home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start'
/opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require'
/home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:30:in `require'
script/rails:30:in `<main>'



aws.log:
=======
[----] W, [2015-02-24T20:48:50.925473 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:50.925526 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:50.987433 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:50.987488 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.108150 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:51.108220 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.232265 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:51.232338 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.307270 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:51.307366 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.373489 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:51.373563 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.436662 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:51.436734 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.509434 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA
[----] W, [2015-02-24T20:48:51.509486 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events.
[----] W, [2015-02-24T20:48:51.583734 #20086:31b8994]  WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Acce
Comment 2 Ramesh A 2015-08-05 07:33:17 EDT
Github link - https://github.com/ManageIQ/manageiq/issues/1871
Comment 3 Joe Vlcek 2015-08-19 11:01:08 EDT
*** Bug 1196853 has been marked as a duplicate of this bug. ***
Comment 4 Joe Vlcek 2015-08-24 09:44:19 EDT
This BZ was reporting 2 separate issues and is being separated into
2 BZ

This BZ will be use to track the issue:

- Log directory filling up when AWS was having API issues

A new BZ [1] has been created to track the NoMethodError.


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1256404 - Amazon provider fails with: [NoMethodError]: undefined method `keys' for nil:NilClass
Comment 6 CFME Bot 2015-10-27 10:45:46 EDT
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/c19fa700a14017df7520ab685eea6ffcda9496dc

commit c19fa700a14017df7520ab685eea6ffcda9496dc
Author:     Joe VLcek <jvlcek@redhat.com>
AuthorDate: Fri Oct 16 17:08:13 2015 -0400
Commit:     Joe VLcek <jvlcek@redhat.com>
CommitDate: Mon Oct 26 17:48:49 2015 -0400

    Raise an exception if provider is unreachable
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1250444

 .../amazon/cloud_manager/event_catcher/runner.rb   |  1 +
 .../amazon/cloud_manager/event_catcher/stream.rb   |  5 ++++-
 .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++-
 app/models/miq_worker/runner.rb                    | 23 ++++++++++++++++++++
 .../cloud_manager/event_catcher/stream_spec.rb     | 25 ++++++++++++++++++++++
 spec/models/miq_worker/runner_spec.rb              |  7 ++++++
 6 files changed, 77 insertions(+), 2 deletions(-)
 create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream_spec.rb
Comment 8 CFME Bot 2015-11-06 09:19:22 EST
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=04acb6f37d66ea9049cdf8e303352ac6b9392041

commit 04acb6f37d66ea9049cdf8e303352ac6b9392041
Author:     Joe VLcek <jvlcek@redhat.com>
AuthorDate: Fri Oct 16 17:08:13 2015 -0400
Commit:     Joe VLcek <jvlcek@redhat.com>
CommitDate: Thu Nov 5 10:52:07 2015 -0500

    Raise an exception if provider is unreachable
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1250444

 .../amazon/cloud_manager/event_catcher/runner.rb   |  1 +
 .../amazon/cloud_manager/event_catcher/stream.rb   |  5 ++++-
 .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++-
 app/models/miq_worker/runner.rb                    | 23 ++++++++++++++++++++
 .../cloud_manager/event_catcher/stream_spec.rb     | 25 ++++++++++++++++++++++
 spec/models/miq_worker/runner_spec.rb              |  7 ++++++
 6 files changed, 77 insertions(+), 2 deletions(-)
 create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream_spec.rb
Comment 9 CFME Bot 2015-11-06 09:19:29 EST
New commit detected on cfme/5.5.z:
https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a1ec8baa9b4f187c556d05828cb29cf64806a713

commit a1ec8baa9b4f187c556d05828cb29cf64806a713
Merge: e4188f3 04acb6f
Author:     Greg Blomquist <gblomqui@redhat.com>
AuthorDate: Fri Nov 6 09:16:43 2015 -0500
Commit:     Greg Blomquist <gblomqui@redhat.com>
CommitDate: Fri Nov 6 09:16:43 2015 -0500

    Merge branch '1250444' into '5.5.z'
    
    Raise an exception if provider is unreachable
    
    Merged cleanly
    
    BZ:
    https://bugzilla.redhat.com/show_bug.cgi?id=1250444
    
    PR:
    https://github.com/ManageIQ/manageiq/pull/4939
    
    See merge request !366

 .../amazon/cloud_manager/event_catcher/runner.rb   |  1 +
 .../amazon/cloud_manager/event_catcher/stream.rb   |  5 ++++-
 .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++-
 app/models/miq_worker/runner.rb                    | 23 ++++++++++++++++++++
 .../cloud_manager/event_catcher/stream_spec.rb     | 25 ++++++++++++++++++++++
 spec/models/miq_worker/runner_spec.rb              |  7 ++++++
 6 files changed, 77 insertions(+), 2 deletions(-)
Comment 10 Ramesh A 2015-11-20 04:08:15 EST
Good to go. Verified and working fine in 5.5.0.10-beta2.1.20151110134042_d6f5459.  Log files are not getting flooded with error messages anymore.

To Test this issue, following method was used:

1. Blocked the ip's connecting amazon ec2:
a) used "netstat -ct |grep https" to see the ip which was getting established
b) Blocked the ip from the appliance using the command "firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -d <ip_address> -j DROP"

2. Performed a manual refresh for the provider

3. Following behavior was noticed
a) netstat -ct|grep https
Proto Recv-Q Send-Q Local Address        Foreign Address     State 
tcp   38      0     <ip_address:port>    <ip_address:port>   CLOSE_WAIT 
tcp   38      0     <ip_address:port>    <ip_address:port>   CLOSE_WAIT
tcp   38      0     <ip_address:port>    <ip_address:port>   CLOSE_WAIT
tcp   38      0     <ip_address:port>    <ip_address:port>   CLOSE_WAIT

b) No ERROR / WARNING message logged in aws.log file
d)evm.log file: Event Montior thread was getting restarted few times and after few tries, single error message was logged.  Please find the below mentioned "event monitor" and "error message" for more details
c) WebUI: Provider "Last Refresh" status showed "Error - 2 Minutes Ago, execution expired" 
d) Waited for 30 min by repeating the provider refresh and Instance "Refresh Relationships and Power States" but did not see log files flooding.



event monitor:
==============
[----] I, [2015-11-20T02:48:32.376011 #16050:b27988]  INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::EventCatcher::Runner#do_work) EMS [] as [<id>] Event Monitor Thread gone. Restarting...
[----] I, [2015-11-20T02:48:34.440585 #2469:d81990]  INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:server_dequeue=>0.004467010498046875, :worker_monitor=>1.0034005641937256, :worker_dequeue=>0.011390447616577148, :total_time=>1.019702672958374}

error message:
==============
[----] E, [2015-11-20T02:41:54.723755 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [ec2], id: [1] Refresh failed
[----] E, [2015-11-20T02:41:54.724163 #16047:11f998c] ERROR -- : [Net::OpenTimeout]: execution expired  Method:[rescue in block in refresh]
[----] E, [2015-11-20T02:41:54.724333 #16047:11f998c] ERROR -- : /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `initialize'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `open'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `block in connect'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `call'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `timeout'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:878:in `connect'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:863:in `do_start'
/opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:858:in `start'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/connection_pool.rb:327:in `start_session'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/connection_pool.rb:127:in `session_for'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/net_http_handler.rb:55:in `handle'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:253:in `block in make_sync_request'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:289:in `retry_server_errors'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:249:in `make_sync_request'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:511:in `block (2 levels) in client_request'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:391:in `log_client_request'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:477:in `block in client_request'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:373:in `return_or_raise'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:476:in `client_request'
(eval):3:in `describe_availability_zones'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/ec2/filtered_collection.rb:44:in `filtered_request'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/ec2/availability_zone_collection.rb:24:in `each'
/var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:153:in `process_collection'
/var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:59:in `get_availability_zones'
/var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:29:in `block in ems_inv_to_hashes'
/opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core.rb:617:in `memoize'
/var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:27:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:5:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresher.rb:5:in `parse_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:19:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:10:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:77:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:345:in `block in deliver'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch'
/opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:341:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:151:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:29:in `start_worker'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:2:in `<top (required)>'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
/opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>'
/var/www/miq/vmdb/bin/rails:4:in `require'
/var/www/miq/vmdb/bin/rails:4:in `<main>'
[----] E, [2015-11-20T02:41:54.724456 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [ec2], id: [1] Unable to perform refresh for the following targets:
[----] E, [2015-11-20T02:41:54.724643 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh)  --- ManageIQ::Providers::Amazon::CloudManager [ec2] id [1]
[----] I, [2015-11-20T02:41:54.810987 #16047:11f998c]  INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) Refreshing all targets...Complete
Comment 12 errata-xmlrpc 2015-12-08 08:26:00 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2551

Note You need to log in before you can comment on or make changes to this bug.