Bug 1439257

Summary:	Amazon S3 Storage Manager \| Seahorse::Client::NetworkingError]: Failed to open TCP connection
Product:	Red Hat CloudForms Management Engine	Reporter:	ncatling
Component:	Providers	Assignee:	Ladislav Smola <lsmola>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Matouš Mojžíš <mmojzis>
Severity:	high	Docs Contact:
Priority:	high
Version:	5.8.0	CC:	jfrey, jhardy, mmojzis, ncatling, obarenbo, simaishi
Target Milestone:	GA	Keywords:	TestOnly
Target Release:	5.9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	ec2:s3:storage
Fixed In Version:	5.9.0.1	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1443087 (view as bug list)		Environment:
Last Closed:	2018-03-06 14:55:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1443087

Description ncatling 2017-04-05 14:04:26 UTC

Description of problem:
Amazon S3 Storage Manager refresh error.

Version-Release number of selected component (if applicable):
5.8.0.9 (Alpha 2)

How reproducible:

Steps to Reproduce:
1. Add Amazon cloud provider
2. Review refresh status of associated Amazon S3 Storage Manager
3.

Actual results:
ERROR -- : [Seahorse::Client::NetworkingError]: Failed to open TCP connection to redhat-new-1000000000266.s3.EU.amazonaws.com:443 (getaddrinfo: Name or service not known)

Expected results:


Additional info:

ERROR -- : MIQ(ManageIQ::Providers::Amazon::StorageManager::S3::Refresher#refresh) EMS: [AWS S3 Storage Manager], id: [1000000000005] Refresh failed
[----] E, [2017-04-05T09:50:54.019122 #17424:12e9130] ERROR -- : [Seahorse::Client::NetworkingError]: Failed to open TCP connection to redhat-new-1000000000266.s3.EU.amazonaws.com:443 (getaddrinfo: Name or service not known)  Method:[rescue in block in refresh]
ERROR -- : /opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:882:in `rescue in block in connect'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:879:in `block in connect'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:101:in `timeout'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:878:in `connect'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:863:in `do_start'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:858:in `start'
/opt/rh/rh-ruby23/root/usr/share/ruby/delegate.rb:83:in `method_missing'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/connection_pool.rb:285:in `start_session'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/connection_pool.rb:92:in `session_for'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/handler.rb:119:in `session'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/handler.rb:71:in `transmit'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/net_http/handler.rb:45:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/content_length.rb:12:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:88:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:119:in `retry_request'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:102:in `retry_if_possible'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:90:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:119:in `retry_request'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:102:in `retry_if_possible'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:90:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:119:in `retry_request'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:102:in `retry_if_possible'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/retry_errors.rb:90:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/rest/handler.rb:7:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/endpoint.rb:41:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/logging.rb:39:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/request.rb:70:in `send_request'
/opt/rh/cfme-gemset/gems/aws-sdk-core-2.8.14/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/collector/storage_manager/s3.rb:12:in `cloud_object_store_objects'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:49:in `process_objects'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:23:in `block in process_containers'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:21:in `each'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:21:in `process_containers'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/inventory/parser/storage_manager/s3.rb:10:in `parse'
/var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:23:in `block in inventory_collections'
/var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:20:in `each'
/var/www/miq/vmdb/app/models/manager_refresh/inventory.rb:20:in `inventory_collections'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/storage_manager/s3/refresher.rb:27:in `block in parse_targeted_inventory'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/storage_manager/s3/refresher.rb:27:in `block in parse_targeted_inventory'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/opt/rh/cfme-gemset/bundler/gems/manageiq-providers-amazon-fbdfcff83d68/app/models/manageiq/providers/amazon/storage_manager/s3/refresher.rb:25:in `parse_targeted_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `block in refresh_targets_for_ems'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:86:in `refresh_targets_for_ems'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh'
/var/www/miq/vmdb/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:98:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:97:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:347:in `block in deliver'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:91:in `block in timeout'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `block in catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:33:in `catch'
/opt/rh/rh-ruby23/root/usr/share/ruby/timeout.rb:106:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:343:in `deliver'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:135:in `deliver_message'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:153:in `block in do_work'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `loop'
/var/www/miq/vmdb/app/models/miq_queue_worker_base/runner.rb:147:in `do_work'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:334:in `block in do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:331:in `do_work_loop'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:153:in `run'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:128:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/miq_worker.rb:339:in `block in start_runner'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/opt/rh/cfme-gemset/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/var/www/miq/vmdb/app/models/miq_worker.rb:337:in `start_runner'
/var/www/miq/vmdb/app/models/miq_worker.rb:348:in `start'
/var/www/miq/vmdb/app/models/miq_worker.rb:266:in `start_worker'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/var/www/miq/vmdb/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/var/www/miq/vmdb/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/var/www/miq/vmdb/app/models/miq_server.rb:348:in `block in monitor'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:348:in `monitor'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `block (2 levels) in monitor_loop'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-aaee67427bce/lib/gems/pending/util/extensions/miq-benchmark.rb:30:in `realtime_block'
/var/www/miq/vmdb/app/models/miq_server.rb:370:in `block in monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:369:in `loop'
/var/www/miq/vmdb/app/models/miq_server.rb:369:in `monitor_loop'
/var/www/miq/vmdb/app/models/miq_server.rb:252:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:65:in `start'
/var/www/miq/vmdb/lib/workers/evm_server.rb:91:in `start'
/var/www/miq/vmdb/lib/workers/bin/evm_server.rb:4:in `<main>'
ERROR -- : MIQ(ManageIQ::Providers::Amazon::StorageManager::S3::Refresher#refresh) EMS: [AWS S3 Storage Manager], id: [1000000000005] Unable to perform refresh for the following targets:
ERROR -- : MIQ(ManageIQ::Providers::Amazon::StorageManager::S3::Refresher#refresh)  --- ManageIQ::Providers::Amazon::StorageManager::S3 [AWS S3 Storage Manager] id [1000000000005]

Comment 9 CFME Bot 2017-04-07 07:47:54 UTC

https://github.com/ManageIQ/manageiq-providers-amazon/pull/207

Comment 11 CFME Bot 2017-04-07 10:07:49 UTC

New commit detected on ManageIQ/manageiq-providers-amazon/master:
https://github.com/ManageIQ/manageiq-providers-amazon/commit/fcd61edb7c7b6c6877dfb5263bdeac64d3ab359f

commit fcd61edb7c7b6c6877dfb5263bdeac64d3ab359f
Author:     Ladislav Smola <lsmola>
AuthorDate: Fri Apr 7 09:32:22 2017 +0200
Commit:     Ladislav Smola <lsmola>
CommitDate: Fri Apr 7 09:34:21 2017 +0200

    Gracefully ignore S3 object listing errors for Graph Refresh
    
    Gracefully ignore S3 object listing errors for Graph Refresh,
    and log them as a warning.
    
    Fixes BZ:
    https://bugzilla.redhat.com/show_bug.cgi?id=1439257

 .../providers/amazon/inventory/collector/storage_manager/s3.rb        | 4 ++++
 1 file changed, 4 insertions(+)

Comment 12 CFME Bot 2017-04-07 13:47:52 UTC

https://github.com/ManageIQ/manageiq-providers-amazon/pull/208

Comment 13 CFME Bot 2017-04-12 07:27:59 UTC

New commit detected on ManageIQ/manageiq-providers-amazon/master:
https://github.com/ManageIQ/manageiq-providers-amazon/commit/67f30f2cbf02e170bcf93b65e0759607e198b1c5

commit 67f30f2cbf02e170bcf93b65e0759607e198b1c5
Author:     Ladislav Smola <lsmola>
AuthorDate: Fri Apr 7 15:32:38 2017 +0200
Commit:     Ladislav Smola <lsmola>
CommitDate: Fri Apr 7 15:37:21 2017 +0200

    Use $aws_log instead of _log in Graph refresh
    
    Use $aws_log instead of _log in Graph refresh
    
    Fixes BZ:
    https://bugzilla.redhat.com/show_bug.cgi?id=1439257

 .../providers/amazon/inventory/collector/storage_manager/s3.rb        | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Comment 16 Matouš Mojžíš 2018-02-08 17:02:09 UTC

Verified in 5.9.0.20. I successfully refreshed S3 provider without errors.