Red Hat Bugzilla – Bug 1250444
Log directory filling up when AWS was having API issues
Last modified: 2015-12-08 08:26:00 EST
Description of problem: Whenever AWS has API issues in their regions, Appliance log files are flooded with error messages and then crash itself as appliance keep getting AWS API errors. Version-Release number of selected component (if applicable): master.20150803085357_e08ddc8 How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: AWS Status Issue: ================= Amazon Elastic Compute Cloud (N. Virginia) Increased API Error Rates less 12:22 PM PST We are investigating increased API error rates in the US-EAST-1 Region. 12:53 PM PST We can confirm increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted. 1:58 PM PST We continue to work toward resolving increased API error rates for EC2 APIs in the US-EAST-1 Region. Running instances are not impacted. 2:51 PM PST We are beginning to see recovery for EC2 API error rates in the US-EAST-1 Region. evm.log: ======== [----] E, [2015-02-24T20:47:30.038345 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture' /opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver' /home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout' /opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work' /opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run' /opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start' /opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker' /opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>' script/rails:30:in `require' script/rails:30:in `<main>' [----] E, [2015-02-24T20:47:30.038694 #20069:80beac] ERROR -- : MIQ(MiqQueue.deliver) Message id: [4135889], Error: [undefined method `keys' for nil:NilClass] [----] E, [2015-02-24T20:47:30.038828 #20069:80beac] ERROR -- : [NoMethodError]: undefined method `keys' for nil:NilClass Method:[rescue in deliver] [----] E, [2015-02-24T20:47:30.038913 #20069:80beac] ERROR -- : /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block (2 levels) in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `collect' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:79:in `block in perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `each' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:78:in `perf_capture_data_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture/amazon.rb:13:in `perf_collect_metrics_amazon' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:11:in `perf_collect_metrics' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:161:in `block in perf_capture' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:12:in `realtime_store' /opt/manageiq/lib/util/extensions/miq-benchmark.rb:31:in `realtime_block' /opt/manageiq/vmdb/app/models/metric/ci_mixin/capture.rb:158:in `perf_capture' /opt/manageiq/vmdb/app/models/miq_queue.rb:360:in `block in deliver' /home/miqbuilder/.rvm/rubies/ruby-2.0.0-p598/lib/ruby/2.0.0/timeout.rb:66:in `timeout' /opt/manageiq/vmdb/app/models/miq_queue.rb:356:in `deliver' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop' /opt/manageiq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work' /opt/manageiq/vmdb/lib/workers/worker_base.rb:317:in `block in do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:314:in `do_work_loop' /opt/manageiq/vmdb/lib/workers/worker_base.rb:135:in `run' /opt/manageiq/vmdb/lib/workers/worker_base.rb:123:in `start' /opt/manageiq/vmdb/lib/workers/worker_base.rb:23:in `start_worker' /opt/manageiq/vmdb/lib/workers/bin/worker.rb:5:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `eval' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands/runner.rb:52:in `<top (required)>' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `require' /home/miqbuilder/.rvm/gems/ruby-2.0.0-p598/bundler/gems/rails-8f014fba21f9/railties/lib/rails/commands.rb:64:in `<top (required)>' script/rails:30:in `require' script/rails:30:in `<main>' aws.log: ======= [----] W, [2015-02-24T20:48:50.925473 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:50.925526 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:50.987433 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:50.987488 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.108150 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:51.108220 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.232265 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:51.232338 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.307270 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:51.307366 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.373489 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:51.373563 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.436662 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:51.436734 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.509434 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Access Key ID MYAWSACCESSKEYWASHEREHAHA [----] W, [2015-02-24T20:48:51.509486 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Contact Amazon to create the AWS Config service and topic for Amazon events. [----] W, [2015-02-24T20:48:51.583734 #20086:31b8994] WARN -- : MIQ(AmazonEventMonitor#find_or_create_queue) Unable to find the AWS Config Topic. Cannot collect Amazon events for AWS Acce
Github link - https://github.com/ManageIQ/manageiq/issues/1871
*** Bug 1196853 has been marked as a duplicate of this bug. ***
This BZ was reporting 2 separate issues and is being separated into 2 BZ This BZ will be use to track the issue: - Log directory filling up when AWS was having API issues A new BZ [1] has been created to track the NoMethodError. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1256404 - Amazon provider fails with: [NoMethodError]: undefined method `keys' for nil:NilClass
https://github.com/ManageIQ/manageiq/pull/4939
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/c19fa700a14017df7520ab685eea6ffcda9496dc commit c19fa700a14017df7520ab685eea6ffcda9496dc Author: Joe VLcek <jvlcek@redhat.com> AuthorDate: Fri Oct 16 17:08:13 2015 -0400 Commit: Joe VLcek <jvlcek@redhat.com> CommitDate: Mon Oct 26 17:48:49 2015 -0400 Raise an exception if provider is unreachable https://bugzilla.redhat.com/show_bug.cgi?id=1250444 .../amazon/cloud_manager/event_catcher/runner.rb | 1 + .../amazon/cloud_manager/event_catcher/stream.rb | 5 ++++- .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++- app/models/miq_worker/runner.rb | 23 ++++++++++++++++++++ .../cloud_manager/event_catcher/stream_spec.rb | 25 ++++++++++++++++++++++ spec/models/miq_worker/runner_spec.rb | 7 ++++++ 6 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream_spec.rb
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=04acb6f37d66ea9049cdf8e303352ac6b9392041 commit 04acb6f37d66ea9049cdf8e303352ac6b9392041 Author: Joe VLcek <jvlcek@redhat.com> AuthorDate: Fri Oct 16 17:08:13 2015 -0400 Commit: Joe VLcek <jvlcek@redhat.com> CommitDate: Thu Nov 5 10:52:07 2015 -0500 Raise an exception if provider is unreachable https://bugzilla.redhat.com/show_bug.cgi?id=1250444 .../amazon/cloud_manager/event_catcher/runner.rb | 1 + .../amazon/cloud_manager/event_catcher/stream.rb | 5 ++++- .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++- app/models/miq_worker/runner.rb | 23 ++++++++++++++++++++ .../cloud_manager/event_catcher/stream_spec.rb | 25 ++++++++++++++++++++++ spec/models/miq_worker/runner_spec.rb | 7 ++++++ 6 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream_spec.rb
New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a1ec8baa9b4f187c556d05828cb29cf64806a713 commit a1ec8baa9b4f187c556d05828cb29cf64806a713 Merge: e4188f3 04acb6f Author: Greg Blomquist <gblomqui@redhat.com> AuthorDate: Fri Nov 6 09:16:43 2015 -0500 Commit: Greg Blomquist <gblomqui@redhat.com> CommitDate: Fri Nov 6 09:16:43 2015 -0500 Merge branch '1250444' into '5.5.z' Raise an exception if provider is unreachable Merged cleanly BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1250444 PR: https://github.com/ManageIQ/manageiq/pull/4939 See merge request !366 .../amazon/cloud_manager/event_catcher/runner.rb | 1 + .../amazon/cloud_manager/event_catcher/stream.rb | 5 ++++- .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++- app/models/miq_worker/runner.rb | 23 ++++++++++++++++++++ .../cloud_manager/event_catcher/stream_spec.rb | 25 ++++++++++++++++++++++ spec/models/miq_worker/runner_spec.rb | 7 ++++++ 6 files changed, 77 insertions(+), 2 deletions(-)
Good to go. Verified and working fine in 5.5.0.10-beta2.1.20151110134042_d6f5459. Log files are not getting flooded with error messages anymore. To Test this issue, following method was used: 1. Blocked the ip's connecting amazon ec2: a) used "netstat -ct |grep https" to see the ip which was getting established b) Blocked the ip from the appliance using the command "firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -d <ip_address> -j DROP" 2. Performed a manual refresh for the provider 3. Following behavior was noticed a) netstat -ct|grep https Proto Recv-Q Send-Q Local Address Foreign Address State tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT b) No ERROR / WARNING message logged in aws.log file d)evm.log file: Event Montior thread was getting restarted few times and after few tries, single error message was logged. Please find the below mentioned "event monitor" and "error message" for more details c) WebUI: Provider "Last Refresh" status showed "Error - 2 Minutes Ago, execution expired" d) Waited for 30 min by repeating the provider refresh and Instance "Refresh Relationships and Power States" but did not see log files flooding. event monitor: ============== [----] I, [2015-11-20T02:48:32.376011 #16050:b27988] INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::EventCatcher::Runner#do_work) EMS [] as [<id>] Event Monitor Thread gone. Restarting... [----] I, [2015-11-20T02:48:34.440585 #2469:d81990] INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:server_dequeue=>0.004467010498046875, :worker_monitor=>1.0034005641937256, :worker_dequeue=>0.011390447616577148, :total_time=>1.019702672958374} error message: ============== [----] E, [2015-11-20T02:41:54.723755 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [ec2], id: [1] Refresh failed [----] E, [2015-11-20T02:41:54.724163 #16047:11f998c] ERROR -- : [Net::OpenTimeout]: execution expired Method:[rescue in block in refresh] [----] E, [2015-11-20T02:41:54.724333 #16047:11f998c] ERROR -- : /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `initialize' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `open' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `block in connect' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `call' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:878:in `connect' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:863:in `do_start' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:858:in `start' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/connection_pool.rb:327:in `start_session' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/connection_pool.rb:127:in `session_for' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/net_http_handler.rb:55:in `handle' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:253:in `block in make_sync_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:289:in `retry_server_errors' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:249:in `make_sync_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:511:in `block (2 levels) in client_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:391:in `log_client_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:477:in `block in client_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:373:in `return_or_raise' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:476:in `client_request' (eval):3:in `describe_availability_zones' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/ec2/filtered_collection.rb:44:in `filtered_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/ec2/availability_zone_collection.rb:24:in `each' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:153:in `process_collection' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:59:in `get_availability_zones' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:29:in `block in ems_inv_to_hashes' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core.rb:617:in `memoize' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:27:in `ems_inv_to_hashes' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:5:in `ems_inv_to_hashes' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresher.rb:5:in `parse_inventory' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:19:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `each' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `refresh' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:10:in `refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:77:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `each' /var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `refresh' /var/www/miq/vmdb/app/models/miq_queue.rb:345:in `block in deliver' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:341:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:151:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:29:in `start_worker' /var/www/miq/vmdb/lib/workers/bin/worker.rb:2:in `<top (required)>' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>' /var/www/miq/vmdb/bin/rails:4:in `require' /var/www/miq/vmdb/bin/rails:4:in `<main>' [----] E, [2015-11-20T02:41:54.724456 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [ec2], id: [1] Unable to perform refresh for the following targets: [----] E, [2015-11-20T02:41:54.724643 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) --- ManageIQ::Providers::Amazon::CloudManager [ec2] id [1] [----] I, [2015-11-20T02:41:54.810987 #16047:11f998c] INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) Refreshing all targets...Complete
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2551