Bug 1459248 - Unable to Add provider running on SSL (trusting custom CA)
Unable to Add provider running on SSL (trusting custom CA)
Status: ASSIGNED
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Providers (Show other bugs)
5.8.0
Unspecified Unspecified
high Severity urgent
: GA
: 5.9.0
Assigned To: John Mazzitelli
Prachi
middleware:provider
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-06 11:47 EDT by Prachi
Modified: 2017-07-12 08:15 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: Bug
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Middleware


Attachments (Terms of Use)
Screen-shot of validating provider (157.04 KB, image/png)
2017-06-06 11:47 EDT, Prachi
no flags Details
CFME logs (949.80 KB, application/zip)
2017-06-07 12:40 EDT, Prachi
no flags Details

  None (edit)
Description Prachi 2017-06-06 11:47:36 EDT
Created attachment 1285462 [details]
Screen-shot of validating provider

Description of problem: Unable to Add provider running on SSL (trusting custom CA). Uploading pem file is not working.

Version-Release number of selected component (if applicable):
5.8.0.17.20170525183055_6317a22 


How reproducible:


Steps to Reproduce:
1. Install HS using SSL with trusted Custom CA(no clear document how to do this)
*************************************
Create key file and pem file

1) mkdir /client-secrets and run below command inside this directory
2) openssl genrsa -out hawkular-services-private.key 2048
3) openssl req -new -sha256 -key hawkular-services-private.key -out csr.csr
4)openssl req -x509 -sha256 -days 365 -key hawkular-services-private.key -in csr.csr -out hawkular-services-public.pem

Start Casandra:

docker run --name hawkular-cassandra -d -e CASSANDRA_START_RPC=true  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-mm-7-tech-preview/middleware-manager-datastore

Start Hawkular server:

 docker run -d --link=hawkular-cassandra -e HAWKULAR_USER=jdoe -e HAWKULAR_PASSWORD=password -e CASSANDRA_NODES=hawkular-cassandra -e HAWKULAR_BACKEND=remote -e DB_TIMEOUT=300 -e CASSANDRA_CONNECT_TIMEOUT=40000 -e CASSANDRA_READ_TIMEOUT=40000 -e CASSANDRA_REQUEST_TIMEOUT=40000 -e HAWKULAR_USE_SSL=true -p 8443:8443 -p 9990:9990 -v /client-secrets:/client-secrets  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-mm-7-tech-preview/middleware-manager:7.0.0-16

*************************************

2. Access https://<IP:8443>, should accessible
3. Login to CFME
4. Add Middleware provider
5. Select "Security protocol" "SSL trusting custom CA"
6. Copy paste pem file (hawkular-services-public.pem)
6. Select IP and port=8443
7. Validate

Actual results: Validation failed


Expected results: Validation Successful

Additional info
Comment 2 Prachi 2017-06-07 12:17:00 EDT
CFME logs while validating provider.

*********************************
[----] I, [2017-06-07T12:15:37.086002 #5739:b1713c]  INFO -- : MIQ(MiqScheduleWorker::Runner#do_work) Number of scheduled items to be processed: 1.
[----] I, [2017-06-07T12:15:37.091506 #5739:b1713c]  INFO -- : MIQ(MiqQueue.put) Message id: [81019],  id: [], Zone: [default], Role: [], Server: [254da604-41a4-11e7-a2d4-001a4a4501eb], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [Session.check_session_timeout], Timeout: [600], Priority: [90], State: [ready], Deliver On: [], Data: [], Args: []
[----] I, [2017-06-07T12:15:37.095801 #5739:b1713c]  INFO -- : MIQ(MiqQueue.put) Message id: [81020],  id: [], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: []
[----] E, [2017-06-07T12:15:38.232754 #5801:14d1f4c] ERROR -- : hostname "10.16.23.128" does not match the server certificate (OpenSSL::SSL::SSLError)
/opt/rh/rh-ruby23/root/usr/share/ruby/openssl/ssl.rb:315:in `post_connection_check'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:944:in `connect'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:863:in `do_start'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:852:in `start'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/base_client.rb:41:in `http_get'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/inventory/inventory_api.rb:298:in `fetch_version_and_status'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/inventory/inventory_api.rb:28:in `initialize'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/hawkular_client.rb:35:in `new'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/hawkular_client.rb:35:in `inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/hawkular/middleware_manager.rb:37:in `verify_credentials'
/var/www/miq/vmdb/app/models/mixins/authentication_mixin.rb:326:in `authentication_check_no_validation'
/var/www/miq/vmdb/app/models/mixins/authentication_mixin.rb:304:in `authentication_check'
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-1048ffdbd63f/app/controllers/mixins/ems_common_angular.rb:58:in `update_ems_button_validate'
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-1048ffdbd63f/app/controllers/mixins/ems_common_angular.rb:105:in `create_ems_button_validate'
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-1048ffdbd63f/app/controllers/mixins/ems_common_angular.rb:76:in `create'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/base.rb:188:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:126:in `call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:455:in `call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:101:in `__run_callbacks__'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:90:in `run_callbacks'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/callbacks.rb:19:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/rescue.rb:20:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/notifications.rb:164:in `block in instrument'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/notifications.rb:164:in `instrument'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/params_wrapper.rb:248:in `process_action'
/opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/base.rb:126:in `process'
/opt/rh/cfme-gemset/gems/actionview-5.0.3/lib/action_view/rendering.rb:30:in `process'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal.rb:190:in `dispatch'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal.rb:262:in `dispatch'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/routing/route_set.rb:32:in `serve'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/journey/router.rb:39:in `block in serve'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/journey/router.rb:26:in `each'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/journey/router.rb:26:in `serve'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/routing/route_set.rb:725:in `call'
/opt/rh/cfme-gemset/gems/secure_headers-3.0.3/lib/secure_headers/middleware.rb:10:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:90:in `run_callbacks'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/opt/rh/cfme-gemset/gems/railties-5.0.3/lib/rails/rack/logger.rb:36:in `call_app'
/opt/rh/cfme-gemset/gems/railties-5.0.3/lib/rails/rack/logger.rb:26:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/request_id.rb:24:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/executor.rb:12:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
/opt/rh/cfme-gemset/gems/railties-5.0.3/lib/rails/engine.rb:522:in `call'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/configuration.rb:224:in `call'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/server.rb:561:in `handle_request'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/server.rb:406:in `process_client'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/server.rb:271:in `block in run'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/thread_pool.rb:111:in `block in spawn_thread'
[----] W, [2017-06-07T12:15:38.233111 #5801:14d1f4c]  WARN -- : MIQ(ManageIQ::Providers::Hawkular::MiddlewareManager#authentication_check_no_validation) type: ["default"] for [] [hawkular-SSL] Validation failed: error, Unable to verify credentials
[----] E, [2017-06-07T12:15:38.233426 #5801:14d1f4c] ERROR -- : MIQ(ems_middleware_controller-create): Credential validation was not successful: Unable to verify credentials
[----] I, [2017-06-07T12:15:38.484575 #1466:b1713c]  INFO -- : MIQ(MiqServer#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic, wcount=4, priority=200


[----] I, [2017-06-07T12:15:38.843388 #5721:b1713c]  INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [81020], MiqWorker id: [36], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [1.749476646] seconds
[----] I, [2017-06-07T12:15:38.843513 #5721:b1713c]  INFO -- : Q-task_id([job_dispatcher]) MIQ(MiqQueue#deliver) Message id: [81020], Delivering...
[----] I, [2017-06-07T12:15:38.846863 #5721:b1713c]  INFO -- : Q-task_id([job_dispatcher]) MIQ(JobProxyDispatcher#dispatch) Complete - Timings: {:pending_container_jobs=>0.001725912094116211, :container_jobs_to_dispatch_count=>0, :container_dispatching=>0.001737356185913086, :pending_vm_jobs=>0.000392913818359375, :vm_jobs_to_dispatch_count=>0, :total_time=>0.0031023025512695312}
[----] I, [2017-06-07T12:15:38.847067 #5721:b1713c]  INFO -- : Q-task_id([job_dispatcher]) MIQ(MiqQueue#delivered) Message id: [81020], State: [ok], Delivered in [0.003561065] seconds
[----] I, [2017-06-07T12:15:39.778624 #5703:b1713c]  INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [81019], MiqWorker id: [34], Zone: [default], Role: [], Server: [254da604-41a4-11e7-a2d4-001a4a4501eb], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [Session.check_session_timeout], Timeout: [600], Priority: [90], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [2.689453743] seconds
[----] I, [2017-06-07T12:15:39.778743 #5703:b1713c]  INFO -- : MIQ(MiqQueue#deliver) Message id: [81019], Delivering...
[----] I, [2017-06-07T12:15:39.780034 #5703:b1713c]  INFO -- : MIQ(MiqQueue#delivered) Message id: [81019], State: [ok], Delivered in [0.00126907] seconds
[----] I, [2017-06-07T12:15:43.487590 #1466:b1713c]  INFO -- : MIQ(MiqServer#heartbeat) Heartbeat [2017-06-07 16:15:43 UTC]...
[----] I, [2017-06-07T12:15:43.494119 #1466:b1713c]  INFO -- : MIQ(MiqServer#heartbeat) Heartbeat [2017-06-07 16:15:43 UTC]...Complete

*********************************
Comment 3 Prachi 2017-06-07 12:37:23 EDT
1) Is hawkular configured ok on ssl?

Able to access Hawkular on IP:8443 but do not have documented command so that we can confirm if the steps followed are correct and it is working as expected.
Steps already described in description. However, Hawkular agent is also not connecting to server.

2) the issue is in miq side not able to validate/connect to it? Or both?

AS per above logs we can see that MIQ is not validating HS and giving error.
Comment 4 Prachi 2017-06-07 12:40 EDT
Created attachment 1285838 [details]
CFME logs

Note You need to log in before you can comment on or make changes to this bug.