Bug 1250444
Summary: | Log directory filling up when AWS was having API issues | ||
---|---|---|---|
Product: | Red Hat CloudForms Management Engine | Reporter: | Ramesh A <rananda> |
Component: | Providers | Assignee: | Joe Vlcek <jvlcek> |
Status: | CLOSED ERRATA | QA Contact: | Ramesh A <rananda> |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | 5.5.0 | CC: | abellott, gblomqui, jfrey, jhardy, jvlcek, mfeifer, obarenbo, simaishi |
Target Milestone: | GA | ||
Target Release: | 5.5.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | 5.5.0.10 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2015-12-08 13:26:00 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Ramesh A
2015-08-05 11:31:41 UTC
Github link - https://github.com/ManageIQ/manageiq/issues/1871 *** Bug 1196853 has been marked as a duplicate of this bug. *** This BZ was reporting 2 separate issues and is being separated into 2 BZ This BZ will be use to track the issue: - Log directory filling up when AWS was having API issues A new BZ [1] has been created to track the NoMethodError. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1256404 - Amazon provider fails with: [NoMethodError]: undefined method `keys' for nil:NilClass New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/c19fa700a14017df7520ab685eea6ffcda9496dc commit c19fa700a14017df7520ab685eea6ffcda9496dc Author: Joe VLcek <jvlcek> AuthorDate: Fri Oct 16 17:08:13 2015 -0400 Commit: Joe VLcek <jvlcek> CommitDate: Mon Oct 26 17:48:49 2015 -0400 Raise an exception if provider is unreachable https://bugzilla.redhat.com/show_bug.cgi?id=1250444 .../amazon/cloud_manager/event_catcher/runner.rb | 1 + .../amazon/cloud_manager/event_catcher/stream.rb | 5 ++++- .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++- app/models/miq_worker/runner.rb | 23 ++++++++++++++++++++ .../cloud_manager/event_catcher/stream_spec.rb | 25 ++++++++++++++++++++++ spec/models/miq_worker/runner_spec.rb | 7 ++++++ 6 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream_spec.rb New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=04acb6f37d66ea9049cdf8e303352ac6b9392041 commit 04acb6f37d66ea9049cdf8e303352ac6b9392041 Author: Joe VLcek <jvlcek> AuthorDate: Fri Oct 16 17:08:13 2015 -0400 Commit: Joe VLcek <jvlcek> CommitDate: Thu Nov 5 10:52:07 2015 -0500 Raise an exception if provider is unreachable https://bugzilla.redhat.com/show_bug.cgi?id=1250444 .../amazon/cloud_manager/event_catcher/runner.rb | 1 + .../amazon/cloud_manager/event_catcher/stream.rb | 5 ++++- .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++- app/models/miq_worker/runner.rb | 23 ++++++++++++++++++++ .../cloud_manager/event_catcher/stream_spec.rb | 25 ++++++++++++++++++++++ spec/models/miq_worker/runner_spec.rb | 7 ++++++ 6 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 spec/models/manageiq/providers/amazon/cloud_manager/event_catcher/stream_spec.rb New commit detected on cfme/5.5.z: https://code.engineering.redhat.com/gerrit/gitweb?p=cfme.git;a=commitdiff;h=a1ec8baa9b4f187c556d05828cb29cf64806a713 commit a1ec8baa9b4f187c556d05828cb29cf64806a713 Merge: e4188f3 04acb6f Author: Greg Blomquist <gblomqui> AuthorDate: Fri Nov 6 09:16:43 2015 -0500 Commit: Greg Blomquist <gblomqui> CommitDate: Fri Nov 6 09:16:43 2015 -0500 Merge branch '1250444' into '5.5.z' Raise an exception if provider is unreachable Merged cleanly BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1250444 PR: https://github.com/ManageIQ/manageiq/pull/4939 See merge request !366 .../amazon/cloud_manager/event_catcher/runner.rb | 1 + .../amazon/cloud_manager/event_catcher/stream.rb | 5 ++++- .../providers/base_manager/event_catcher/runner.rb | 18 +++++++++++++++- app/models/miq_worker/runner.rb | 23 ++++++++++++++++++++ .../cloud_manager/event_catcher/stream_spec.rb | 25 ++++++++++++++++++++++ spec/models/miq_worker/runner_spec.rb | 7 ++++++ 6 files changed, 77 insertions(+), 2 deletions(-) Good to go. Verified and working fine in 5.5.0.10-beta2.1.20151110134042_d6f5459. Log files are not getting flooded with error messages anymore. To Test this issue, following method was used: 1. Blocked the ip's connecting amazon ec2: a) used "netstat -ct |grep https" to see the ip which was getting established b) Blocked the ip from the appliance using the command "firewall-cmd --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -d <ip_address> -j DROP" 2. Performed a manual refresh for the provider 3. Following behavior was noticed a) netstat -ct|grep https Proto Recv-Q Send-Q Local Address Foreign Address State tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT tcp 38 0 <ip_address:port> <ip_address:port> CLOSE_WAIT b) No ERROR / WARNING message logged in aws.log file d)evm.log file: Event Montior thread was getting restarted few times and after few tries, single error message was logged. Please find the below mentioned "event monitor" and "error message" for more details c) WebUI: Provider "Last Refresh" status showed "Error - 2 Minutes Ago, execution expired" d) Waited for 30 min by repeating the provider refresh and Instance "Refresh Relationships and Power States" but did not see log files flooding. event monitor: ============== [----] I, [2015-11-20T02:48:32.376011 #16050:b27988] INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::EventCatcher::Runner#do_work) EMS [] as [<id>] Event Monitor Thread gone. Restarting... [----] I, [2015-11-20T02:48:34.440585 #2469:d81990] INFO -- : MIQ(MiqServer#monitor_loop) Server Monitoring Complete - Timings: {:server_dequeue=>0.004467010498046875, :worker_monitor=>1.0034005641937256, :worker_dequeue=>0.011390447616577148, :total_time=>1.019702672958374} error message: ============== [----] E, [2015-11-20T02:41:54.723755 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [ec2], id: [1] Refresh failed [----] E, [2015-11-20T02:41:54.724163 #16047:11f998c] ERROR -- : [Net::OpenTimeout]: execution expired Method:[rescue in block in refresh] [----] E, [2015-11-20T02:41:54.724333 #16047:11f998c] ERROR -- : /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `initialize' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `open' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:879:in `block in connect' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `call' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:878:in `connect' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:863:in `do_start' /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:858:in `start' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/connection_pool.rb:327:in `start_session' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/connection_pool.rb:127:in `session_for' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/http/net_http_handler.rb:55:in `handle' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:253:in `block in make_sync_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:289:in `retry_server_errors' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:249:in `make_sync_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:511:in `block (2 levels) in client_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:391:in `log_client_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:477:in `block in client_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:373:in `return_or_raise' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core/client.rb:476:in `client_request' (eval):3:in `describe_availability_zones' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/ec2/filtered_collection.rb:44:in `filtered_request' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/ec2/availability_zone_collection.rb:24:in `each' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:153:in `process_collection' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:59:in `get_availability_zones' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:29:in `block in ems_inv_to_hashes' /opt/rh/cfme-gemset/gems/aws-sdk-v1-1.56.0/lib/aws/core.rb:617:in `memoize' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:27:in `ems_inv_to_hashes' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresh_parser.rb:5:in `ems_inv_to_hashes' /var/www/miq/vmdb/app/models/manageiq/providers/amazon/cloud_manager/refresher.rb:5:in `parse_inventory' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:19:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `each' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:7:in `refresh' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:10:in `refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:77:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `each' /var/www/miq/vmdb/app/models/ems_refresh.rb:76:in `refresh' /var/www/miq/vmdb/app/models/miq_queue.rb:345:in `block in deliver' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `block in catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:34:in `catch' /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:104:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:341:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:106:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:151:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:145:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:29:in `start_worker' /var/www/miq/vmdb/lib/workers/bin/worker.rb:2:in `<top (required)>' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `load' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/runner.rb:60:in `<top (required)>' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:123:in `require_command!' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:90:in `runner' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands/commands_tasks.rb:39:in `run_command!' /opt/rh/cfme-gemset/gems/railties-4.2.5/lib/rails/commands.rb:17:in `<top (required)>' /var/www/miq/vmdb/bin/rails:4:in `require' /var/www/miq/vmdb/bin/rails:4:in `<main>' [----] E, [2015-11-20T02:41:54.724456 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) EMS: [ec2], id: [1] Unable to perform refresh for the following targets: [----] E, [2015-11-20T02:41:54.724643 #16047:11f998c] ERROR -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) --- ManageIQ::Providers::Amazon::CloudManager [ec2] id [1] [----] I, [2015-11-20T02:41:54.810987 #16047:11f998c] INFO -- : MIQ(ManageIQ::Providers::Amazon::CloudManager::Refresher#refresh) Refreshing all targets...Complete Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:2551 |