Description of problem: Amazon S3 Storage Manager refresh error. Version-Release number of selected component (if applicable): 5.8.0.9 (Alpha 2) How reproducible: Steps to Reproduce: 1. Add Amazon cloud provider 2. Review refresh status of associated Amazon S3 Storage Manager 3. Actual results: ERROR -- : [Seahorse::Client::NetworkingError]: Failed to open TCP connection to redhat-new-1000000000266.s3.EU.amazonaws.com:443 (getaddrinfo: Name or service not known) Expected results: Additional info: ERROR -- : MIQ(ManageIQ::Providers::Amazon::StorageManager::S3::Refresher#refresh) EMS: [AWS S3 Storage Manager], id: [1000000000005] Refresh failed [----] E, [2017-04-05T09:50:54.019122 #17424:12e9130] ERROR -- : [Seahorse::Client::NetworkingError]: Failed to open TCP connection to redhat-new-1000000000266.s3.EU.amazonaws.com:443 (getaddrinfo: Name or service not known) Method:[rescue in block in refresh] ERROR -- : /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:882:in `rescue in block in connect' /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:879:in `block in connect' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:101:in `timeout' /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:878:in `connect' /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:863:in `do_start' /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:858:in `start' /opt/rh/rh-ruby23/root/usr/share/ruby/delegate.rb:83:in `method_missing' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/connection_pool.rb:285:in `start_session' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/connection_pool.rb:92:in `session_for' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/handler.rb:119:in `session' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/handler.rb:71:in `transmit' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/handler.rb:45:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/content_length.rb:12:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/xml/error_handler.rb:8:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:88:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:119:in `retry_request' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:102:in `retry_if_possible' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:90:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:119:in `retry_request' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:102:in `retry_if_possible' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:90:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:119:in `retry_request' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:102:in `retry_if_possible' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:90:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/rest/handler.rb:7:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/endpoint.rb:41:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/logging.rb:39:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/response_target.rb:21:in `call' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/request.rb:70:in `send_request' /opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/collector/storage_manager/s3.rb:12:in `cloud_object_store_objects' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:49:in `process_objects' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:23:in `block in process_containers' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:21:in `each' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:21:in `process_containers' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:10:in `parse' /var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:23:in `block in inventory_collections' /var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:20:in `each' /var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:20:in `inventory_collections' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/storage_manager/s3/refresher.rb:27:in `block in parse_targeted_inventory' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/storage_manager/s3/refresher.rb:27:in `block in parse_targeted_inventory' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/storage_manager/s3/refresher.rb:25:in `parse_targeted_inventory' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `block in refresh_targets_for_ems' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each' /var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh' /var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh' /var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each' /var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh' /var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch' /opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout' /var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:107:in `deliver_queue_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:135:in `deliver_message' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:153:in `block in do_work' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `loop' /var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `do_work' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker' /var/www/miq/vmdb/app/models/miq_worker.rb:339:in `block in start_runner' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork' /opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork' /var/www/miq/vmdb/app/models/miq_worker.rb:337:in `start_runner' /var/www/miq/vmdb/app/models/miq_worker.rb:348:in `start' /var/www/miq/vmdb/app/models/miq_worker.rb:266:in `start_worker' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `each' /var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers' /var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers' /var/www/miq/vmdb/app/models/miq_server.rb:348:in `block in monitor' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:348:in `monitor' /var/www/miq/vmdb/app/models/miq_server.rb:370:in `block (2 levels) in monitor_loop' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store' /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block' /var/www/miq/vmdb/app/models/miq_server.rb:370:in `block in monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:369:in `loop' /var/www/miq/vmdb/app/models/miq_server.rb:369:in `monitor_loop' /var/www/miq/vmdb/app/models/miq_server.rb:252:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start' /var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start' /var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>' ERROR -- : MIQ(ManageIQ::Providers::Amazon::StorageManager::S3::Refresher#refresh) EMS: [AWS S3 Storage Manager], id: [1000000000005] Unable to perform refresh for the following targets: ERROR -- : MIQ(ManageIQ::Providers::Amazon::StorageManager::S3::Refresher#refresh) --- ManageIQ::Providers::Amazon::StorageManager::S3 [AWS S3 Storage Manager] id [1000000000005]
https://github.com/ManageIQ/manageiq-providers-amazon/pull/207
New commit detected on ManageIQ/manageiq-providers-amazon/master: https://github.com/ManageIQ/manageiq-providers-amazon/commit/fcd61edb7c7b6c6877dfb5263bdeac64d3ab359f commit fcd61edb7c7b6c6877dfb5263bdeac64d3ab359f Author: Ladislav Smola <lsmola> AuthorDate: Fri Apr 7 09:32:22 2017 +0200 Commit: Ladislav Smola <lsmola> CommitDate: Fri Apr 7 09:34:21 2017 +0200 Gracefully ignore S3 object listing errors for Graph Refresh Gracefully ignore S3 object listing errors for Graph Refresh, and log them as a warning. Fixes BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1439257 .../providers/amazon/inventory/collector/storage_manager/s3.rb | 4 ++++ 1 file changed, 4 insertions(+)
https://github.com/ManageIQ/manageiq-providers-amazon/pull/208
New commit detected on ManageIQ/manageiq-providers-amazon/master: https://github.com/ManageIQ/manageiq-providers-amazon/commit/67f30f2cbf02e170bcf93b65e0759607e198b1c5 commit 67f30f2cbf02e170bcf93b65e0759607e198b1c5 Author: Ladislav Smola <lsmola> AuthorDate: Fri Apr 7 15:32:38 2017 +0200 Commit: Ladislav Smola <lsmola> CommitDate: Fri Apr 7 15:37:21 2017 +0200 Use $aws_log instead of _log in Graph refresh Use $aws_log instead of _log in Graph refresh Fixes BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1439257 .../providers/amazon/inventory/collector/storage_manager/s3.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Verified in 5.9.0.20. I successfully refreshed S3 provider without errors.