Bug 1222958

Summary:	EMS Refresh against RHOS provider with self-signed certificate results in error
Product:	Red Hat CloudForms Management Engine	Reporter:	Thom Carlin <tcarlin>
Component:	Documentation	Assignee:	Andrew Dahms <adahms>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Thom Carlin <tcarlin>
Severity:	medium	Docs Contact:
Priority:	high
Version:	5.5.0	CC:	gblomqui, jfrey, jhardy, mfeifer, obarenbo, tcarlin
Target Milestone:	GA	Keywords:	Reopened
Target Release:	5.5.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Known Issue
Doc Text:	When an OpenStack provider is set up with services behind SSL termination, and a Red Hat CloudForms appliance is configured to communicate to the Keystone SSL endpoint, users may see the following error in the evm.log (and fog.log): #<Excon::Errors::SocketError: end of file reached (EOFError)> This error occurs because Red Hat CloudForms tries both non-SSL and SSL connections when getting the Keystone security token. This is a benign known issue and is scheduled to be addressed in a future version.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-24 01:44:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Thom Carlin 2015-05-19 13:40:35 UTC

Description of problem:

After adding a RHOS provider with a self-signed certificate, noticed error in evm.log during refreshes.

Version-Release number of selected component (if applicable):

5.4

How reproducible:

100%

Steps to Reproduce:
1. Cloud > Provider
2. Add a new Cloud Provider [RHOS with SSL]
3. Validate

Actual results:

Tracebacks

Expected results:

No errors

Additional info:

Since this is a common occurrence, either this should be a blocker or we add a technical note.

For security concerns, it would be better if the user explicitly authorizes unverified certificates on a case-by-case basis (via checkbox/code?)

evm.log excerpt:
[----] I, [2015-05-18T04:49:52.963035 #28933:5d9ea4]  INFO -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [RHOS5 SSL], id: [1]   EmsOpenstack [RHOS5 SSL] id [1]
[----] E, [2015-05-18T04:49:56.345875 #28933:5d9ea4] ERROR -- : <Fog> excon.error     #<Excon::Errors::SocketError: Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure).>

[----] E, [2015-05-18T04:49:56.346267 #28933:5d9ea4] ERROR -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [RHOS5 SSL], id: [1] Refresh failed
[----] E, [2015-05-18T04:49:56.346548 #28933:5d9ea4] ERROR -- : [Excon::Errors::SocketError]: Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure).  Method:[rescue in block in refresh]
[----] E, [2015-05-18T04:49:56.346736 #28933:5d9ea4] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/ssl_socket.rb:120:in `connect_nonblock'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/ssl_socket.rb:120:in `initialize'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:387:in `new'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:387:in `socket'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:106:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/mock.rb:47:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:19:in `block in request_call'
/var/www/miq/vmdb/lib/vmdb/logging/fog_logger.rb:22:in `instrument'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:18:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:233:in `request'
/opt/rh/cfme-gemset/gems/fog-core-1.30.0/lib/fog/core/connection.rb:81:in `request'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/compute.rb:351:in `request'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/requests/compute/list_flavors_detail.rb:6:in `list_flavors_detail'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/models/compute/flavors.rb:11:in `all'
/opt/rh/cfme-gemset/gems/fog-core-1.30.0/lib/fog/core/collection.rb:113:in `lazy_load'
/opt/rh/cfme-gemset/gems/fog-core-1.30.0/lib/fog/core/collection.rb:17:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:178:in `process_collection'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:85:in `get_flavors'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:39:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:9:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/openstack_refresh[----] I, [2015-05-18T04:49:52.963035 #28933:5d9ea4]  INFO -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [RHOS5 SSL], id: [1]   EmsOpenstack [RHOS5 SSL] id [1]
[----] E, [2015-05-18T04:49:56.345875 #28933:5d9ea4] ERROR -- : <Fog> excon.error     #<Excon::Errors::SocketError: Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure).>

[----] E, [2015-05-18T04:49:56.346267 #28933:5d9ea4] ERROR -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [RHOS5 SSL], id: [1] Refresh failed
[----] E, [2015-05-18T04:49:56.346548 #28933:5d9ea4] ERROR -- : [Excon::Errors::SocketError]: Unable to verify certificate, please set `Excon.defaults[:ssl_ca_path] = path_to_certs`, `ENV['SSL_CERT_DIR'] = path_to_certs`, `Excon.defaults[:ssl_ca_file] = path_to_file`, `ENV['SSL_CERT_FILE'] = path_to_file`, `Excon.defaults[:ssl_verify_callback] = callback` (see OpenSSL::SSL::SSLContext#verify_callback), or `Excon.defaults[:ssl_verify_peer] = false` (less secure).  Method:[rescue in block in refresh]
[----] E, [2015-05-18T04:49:56.346736 #28933:5d9ea4] ERROR -- : /opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/ssl_socket.rb:120:in `connect_nonblock'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/ssl_socket.rb:120:in `initialize'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:387:in `new'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:387:in `socket'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:106:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/mock.rb:47:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:19:in `block in request_call'
/var/www/miq/vmdb/lib/vmdb/logging/fog_logger.rb:22:in `instrument'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/instrumentor.rb:18:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/middlewares/base.rb:15:in `request_call'
/opt/rh/cfme-gemset/gems/excon-0.45.3/lib/excon/connection.rb:233:in `request'
/opt/rh/cfme-gemset/gems/fog-core-1.30.0/lib/fog/core/connection.rb:81:in `request'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/compute.rb:351:in `request'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/requests/compute/list_flavors_detail.rb:6:in `list_flavors_detail'
/opt/rh/cfme-gemset/gems/fog-1.29.0/lib/fog/openstack/models/compute/flavors.rb:11:in `all'
/opt/rh/cfme-gemset/gems/fog-core-1.30.0/lib/fog/core/collection.rb:113:in `lazy_load'
/opt/rh/cfme-gemset/gems/fog-core-1.30.0/lib/fog/core/collection.rb:17:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:178:in `process_collection'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:85:in `get_flavors'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:39:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/parsers/openstack.rb:9:in `ems_inv_to_hashes'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/openstack_refresher.rb:6:in `parse_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:20:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:8:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/base_refresher.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:80:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:356:in `block in deliver'
/opt/rh/ruby200/root/usr/share/ruby/timeout.rb:66:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:352:in `deliver'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/var/www/miq/vmdb/lib/workers/worker_base.rb:323:in `block in do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:320:in `loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:320:in `do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:141:in `run'
/var/www/miq/vmdb/lib/workers/worker_base.rb:122:in `start'
/var/www/miq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:3:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `eval'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `require'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'
[----] E, [2015-05-18T04:49:56.346851 #28933:5d9ea4] ERROR -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [RHOS5 SSL], id: [1] Unable to perform refresh for the following targets:
[----] E, [2015-05-18T04:49:56.347163 #28933:5d9ea4] ERROR -- :  --- EmsOpenstack [RHOS5 SSL] id [1]er.rb:6:in `parse_inventory'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:20:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:8:in `each'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh/refreshers/base_refresher.rb:8:in `refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:80:in `block in refresh'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `each'
/var/www/miq/vmdb/app/models/ems_refresh.rb:78:in `refresh'
/var/www/miq/vmdb/app/models/miq_queue.rb:356:in `block in deliver'
/opt/rh/ruby200/root/usr/share/ruby/timeout.rb:66:in `timeout'
/var/www/miq/vmdb/app/models/miq_queue.rb:352:in `deliver'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:107:in `deliver_queue_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:135:in `deliver_message'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:152:in `block in do_work'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `loop'
/var/www/miq/vmdb/lib/workers/queue_worker_base.rb:146:in `do_work'
/var/www/miq/vmdb/lib/workers/worker_base.rb:323:in `block in do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:320:in `loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:320:in `do_work_loop'
/var/www/miq/vmdb/lib/workers/worker_base.rb:141:in `run'
/var/www/miq/vmdb/lib/workers/worker_base.rb:122:in `start'
/var/www/miq/vmdb/lib/workers/worker_base.rb:23:in `start_worker'
/var/www/miq/vmdb/lib/workers/bin/worker.rb:3:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `eval'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands/runner.rb:52:in `<top (required)>'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `require'
/opt/rh/cfme-gemset/bundler/gems/rails-4842a8377644/railties/lib/rails/commands.rb:64:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'
[----] E, [2015-05-18T04:49:56.346851 #28933:5d9ea4] ERROR -- : MIQ(EmsRefresh::Refreshers::OpenstackRefresher.refresh) EMS: [RHOS5 SSL], id: [1] Unable to perform refresh for the following targets:
[----] E, [2015-05-18T04:49:56.347163 #28933:5d9ea4] ERROR -- :  --- EmsOpenstack [RHOS5 SSL] id [1]

Comment 3 Dave Johnson 2015-05-19 23:53:37 UTC

Thom, I believe you used the default port 5000 for that environment.  Try using port 5443 and I believe you will have better results.

Comment 4 Thom Carlin 2015-05-20 15:55:41 UTC

We see "ERROR -- : excon.error     #<Excon::Errors::SocketError: end of file reached (EOFError)>" as part of the normal SSL/non-SSL process appearing in the logs.  Worth a mention in the documentation somewhere...

Comment 9 Andrew Dahms 2015-11-24 01:44:32 UTC

This known issue has been documented in the Release Notes document for Red Hat CloudForms 3.2

Closing.