Bug 1459248

Summary: Unable to Add provider running on SSL (trusting custom CA)
Product: Red Hat CloudForms Management Engine Reporter: Prachi <pyadav>
Component: ProvidersAssignee: John Mazzitelli <mazz>
Status: CLOSED EOL QA Contact: Prachi <pyadav>
Severity: urgent Docs Contact:
Priority: high    
Version: 5.8.0CC: abonas, dajohnso, gblomqui, hhovsepy, jdoyle, jfrey, jhardy, jstickle, mazz, mfoley, mmahoney, obarenbo, theute
Target Milestone: GAKeywords: Triaged
Target Release: 5.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: middleware:provider
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-06 18:13:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: Bug
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: Middleware Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Screen-shot of validating provider
none
CFME logs none

Description Prachi 2017-06-06 15:47:36 UTC 
Created attachment 1285462 [details]
Screen-shot of validating provider

Description of problem: Unable to Add provider running on SSL (trusting custom CA). Uploading pem file is not working.

Version-Release number of selected component (if applicable):
5.8.0.17.20170525183055_6317a22 


How reproducible:


Steps to Reproduce:
1. Install HS using SSL with trusted Custom CA(no clear document how to do this)
*************************************
Create key file and pem file

1) mkdir /client-secrets and run below command inside this directory
2) openssl genrsa -out hawkular-services-private.key 2048
3) openssl req -new -sha256 -key hawkular-services-private.key -out csr.csr
4)openssl req -x509 -sha256 -days 365 -key hawkular-services-private.key -in csr.csr -out hawkular-services-public.pem

Start Casandra:

docker run --name hawkular-cassandra -d -e CASSANDRA_START_RPC=true  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-mm-7-tech-preview/middleware-manager-datastore

Start Hawkular server:

 docker run -d --link=hawkular-cassandra -e HAWKULAR_USER=jdoe -e HAWKULAR_PASSWORD=password -e CASSANDRA_NODES=hawkular-cassandra -e HAWKULAR_BACKEND=remote -e DB_TIMEOUT=300 -e CASSANDRA_CONNECT_TIMEOUT=40000 -e CASSANDRA_READ_TIMEOUT=40000 -e CASSANDRA_REQUEST_TIMEOUT=40000 -e HAWKULAR_USE_SSL=true -p 8443:8443 -p 9990:9990 -v /client-secrets:/client-secrets  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/jboss-mm-7-tech-preview/middleware-manager:7.0.0-16

*************************************

2. Access https://<IP:8443>, should accessible
3. Login to CFME
4. Add Middleware provider
5. Select "Security protocol" "SSL trusting custom CA"
6. Copy paste pem file (hawkular-services-public.pem)
6. Select IP and port=8443
7. Validate

Actual results: Validation failed


Expected results: Validation Successful

Additional info
Comment 2 Prachi 2017-06-07 16:17:00 UTC 
CFME logs while validating provider.

*********************************
[----] I, [2017-06-07T12:15:37.086002 #5739:b1713c]  INFO -- : MIQ(MiqScheduleWorker::Runner#do_work) Number of scheduled items to be processed: 1.
[----] I, [2017-06-07T12:15:37.091506 #5739:b1713c]  INFO -- : MIQ(MiqQueue.put) Message id: [81019],  id: [], Zone: [default], Role: [], Server: [254da604-41a4-11e7-a2d4-001a4a4501eb], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [Session.check_session_timeout], Timeout: [600], Priority: [90], State: [ready], Deliver On: [], Data: [], Args: []
[----] I, [2017-06-07T12:15:37.095801 #5739:b1713c]  INFO -- : MIQ(MiqQueue.put) Message id: [81020],  id: [], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [ready], Deliver On: [], Data: [], Args: []
[----] E, [2017-06-07T12:15:38.232754 #5801:14d1f4c] ERROR -- : hostname "10.16.23.128" does not match the server certificate (OpenSSL::SSL::SSLError)
/opt/rh/rh-ruby23/root/usr/share/ruby/openssl/ssl.rb:315:in `post_connection_check'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:944:in `connect'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:863:in `do_start'
/opt/rh/rh-ruby23/root/usr/share/ruby/net/http.rb:852:in `start'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
/opt/rh/cfme-gemset/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/base_client.rb:41:in `http_get'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/inventory/inventory_api.rb:298:in `fetch_version_and_status'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/inventory/inventory_api.rb:28:in `initialize'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/hawkular_client.rb:35:in `new'
/opt/rh/cfme-gemset/gems/hawkular-client-3.0.1/lib/hawkular/hawkular_client.rb:35:in `inventory'
/var/www/miq/vmdb/app/models/manageiq/providers/hawkular/middleware_manager.rb:37:in `verify_credentials'
/var/www/miq/vmdb/app/models/mixins/authentication_mixin.rb:326:in `authentication_check_no_validation'
/var/www/miq/vmdb/app/models/mixins/authentication_mixin.rb:304:in `authentication_check'
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-1048ffdbd63f/app/controllers/mixins/ems_common_angular.rb:58:in `update_ems_button_validate'
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-1048ffdbd63f/app/controllers/mixins/ems_common_angular.rb:105:in `create_ems_button_validate'
/opt/rh/cfme-gemset/bundler/gems/manageiq-ui-classic-1048ffdbd63f/app/controllers/mixins/ems_common_angular.rb:76:in `create'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/base.rb:188:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:126:in `call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:455:in `call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:101:in `__run_callbacks__'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:90:in `run_callbacks'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/callbacks.rb:19:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/rescue.rb:20:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/notifications.rb:164:in `block in instrument'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/notifications.rb:164:in `instrument'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal/params_wrapper.rb:248:in `process_action'
/opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/abstract_controller/base.rb:126:in `process'
/opt/rh/cfme-gemset/gems/actionview-5.0.3/lib/action_view/rendering.rb:30:in `process'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal.rb:190:in `dispatch'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_controller/metal.rb:262:in `dispatch'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/routing/route_set.rb:32:in `serve'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/journey/router.rb:39:in `block in serve'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/journey/router.rb:26:in `each'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/journey/router.rb:26:in `serve'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/routing/route_set.rb:725:in `call'
/opt/rh/cfme-gemset/gems/secure_headers-3.0.3/lib/secure_headers/middleware.rb:10:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/cookies.rb:613:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/callbacks.rb:90:in `run_callbacks'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/opt/rh/cfme-gemset/gems/railties-5.0.3/lib/rails/rack/logger.rb:36:in `call_app'
/opt/rh/cfme-gemset/gems/railties-5.0.3/lib/rails/rack/logger.rb:26:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/request_id.rb:24:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
/opt/rh/cfme-gemset/gems/activesupport-5.0.3/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
/opt/rh/cfme-gemset/gems/actionpack-5.0.3/lib/action_dispatch/middleware/executor.rb:12:in `call'
/opt/rh/cfme-gemset/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
/opt/rh/cfme-gemset/gems/railties-5.0.3/lib/rails/engine.rb:522:in `call'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/configuration.rb:224:in `call'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/server.rb:561:in `handle_request'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/server.rb:406:in `process_client'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/server.rb:271:in `block in run'
/opt/rh/rh-ruby23/root/usr/share/gems/gems/puma-3.3.0/lib/puma/thread_pool.rb:111:in `block in spawn_thread'
[----] W, [2017-06-07T12:15:38.233111 #5801:14d1f4c]  WARN -- : MIQ(ManageIQ::Providers::Hawkular::MiddlewareManager#authentication_check_no_validation) type: ["default"] for [] [hawkular-SSL] Validation failed: error, Unable to verify credentials
[----] E, [2017-06-07T12:15:38.233426 #5801:14d1f4c] ERROR -- : MIQ(ems_middleware_controller-create): Credential validation was not successful: Unable to verify credentials
[----] I, [2017-06-07T12:15:38.484575 #1466:b1713c]  INFO -- : MIQ(MiqServer#populate_queue_messages) Fetched 2 miq_queue rows for queue_name=generic, wcount=4, priority=200


[----] I, [2017-06-07T12:15:38.843388 #5721:b1713c]  INFO -- : MIQ(MiqPriorityWorker::Runner#get_message_via_drb) Message id: [81020], MiqWorker id: [36], Zone: [default], Role: [smartstate], Server: [], Ident: [generic], Target id: [], Instance id: [], Task id: [job_dispatcher], Command: [JobProxyDispatcher.dispatch], Timeout: [600], Priority: [20], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [1.749476646] seconds
[----] I, [2017-06-07T12:15:38.843513 #5721:b1713c]  INFO -- : Q-task_id([job_dispatcher]) MIQ(MiqQueue#deliver) Message id: [81020], Delivering...
[----] I, [2017-06-07T12:15:38.846863 #5721:b1713c]  INFO -- : Q-task_id([job_dispatcher]) MIQ(JobProxyDispatcher#dispatch) Complete - Timings: {:pending_container_jobs=>0.001725912094116211, :container_jobs_to_dispatch_count=>0, :container_dispatching=>0.001737356185913086, :pending_vm_jobs=>0.000392913818359375, :vm_jobs_to_dispatch_count=>0, :total_time=>0.0031023025512695312}
[----] I, [2017-06-07T12:15:38.847067 #5721:b1713c]  INFO -- : Q-task_id([job_dispatcher]) MIQ(MiqQueue#delivered) Message id: [81020], State: [ok], Delivered in [0.003561065] seconds
[----] I, [2017-06-07T12:15:39.778624 #5703:b1713c]  INFO -- : MIQ(MiqGenericWorker::Runner#get_message_via_drb) Message id: [81019], MiqWorker id: [34], Zone: [default], Role: [], Server: [254da604-41a4-11e7-a2d4-001a4a4501eb], Ident: [generic], Target id: [], Instance id: [], Task id: [], Command: [Session.check_session_timeout], Timeout: [600], Priority: [90], State: [dequeue], Deliver On: [], Data: [], Args: [], Dequeued in: [2.689453743] seconds
[----] I, [2017-06-07T12:15:39.778743 #5703:b1713c]  INFO -- : MIQ(MiqQueue#deliver) Message id: [81019], Delivering...
[----] I, [2017-06-07T12:15:39.780034 #5703:b1713c]  INFO -- : MIQ(MiqQueue#delivered) Message id: [81019], State: [ok], Delivered in [0.00126907] seconds
[----] I, [2017-06-07T12:15:43.487590 #1466:b1713c]  INFO -- : MIQ(MiqServer#heartbeat) Heartbeat [2017-06-07 16:15:43 UTC]...
[----] I, [2017-06-07T12:15:43.494119 #1466:b1713c]  INFO -- : MIQ(MiqServer#heartbeat) Heartbeat [2017-06-07 16:15:43 UTC]...Complete

*********************************
Comment 3 Prachi 2017-06-07 16:37:23 UTC 
1) Is hawkular configured ok on ssl?

Able to access Hawkular on IP:8443 but do not have documented command so that we can confirm if the steps followed are correct and it is working as expected.
Steps already described in description. However, Hawkular agent is also not connecting to server.

2) the issue is in miq side not able to validate/connect to it? Or both?

AS per above logs we can see that MIQ is not validating HS and giving error.
Comment 4 Prachi 2017-06-07 16:40:12 UTC 
Created attachment 1285838 [details]
CFME logs
Comment 6 Thomas Heute 2017-12-06 12:19:20 UTC 
This might have to be closed, Hayk is testing SSL with the new templates.

Prachi, could you check with Hayk ?
Comment 7 Hayk Hovsepyan 2017-12-12 07:19:29 UTC 
Verified on 5.9.0.12.20171205180333_3e32b3d with CFME SSL Template configured using TP3 DR2 release.