Bug 1254838

Summary:	[hsoted-engine] adding second host fails with: "SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"
Product:	Red Hat Enterprise Virtualization Manager	Reporter:	Ulhas Surse <usurse>
Component:	ovirt-hosted-engine-setup	Assignee:	Simone Tiraboschi <stirabos>
Status:	CLOSED ERRATA	QA Contact:	Artyom <alukiano>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.5.3	CC:	amureini, didi, istein, jhunsaker, lsurette, mavital, mkalinin, sbonazzo, usurse, ykaul, ylavi
Target Milestone:	ovirt-3.6.1	Keywords:	Triaged
Target Release:	3.6.1
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Previously, hosted-engine setup always used the Manager CA certificate to trust the signature of the REST API (apache) certificate, but some users replaced that with an externally signed one. In that case the validation, and so the deployment of additional hosts, failed. With this release, hosted-engine setup lets the user specify the local path of an external CA file, or proceed in insecure mode, if validation with the internal CA certificate fails.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-03-09 19:14:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Integration	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:	1059952
Bug Blocks:	1284979

Description Ulhas Surse 2015-08-19 01:56:07 UTC

Description of problem:
First host installed with RHEVM  appliance with this resolution: 
https://access.redhat.com/solutions/1530223

While adding second host, following error occurred:

[ INFO  ] Starting vdsmd
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Waiting for VDSM hardware info
[ INFO  ] Configuring VM
[ INFO  ] Updating hosted-engine configuration
[ INFO  ] Stage: Transaction commit
[ INFO  ] Stage: Closing up
[ ERROR ] Failed to execute stage 'Closing up': [ERROR]::RHEV API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20150811155922.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination

Performed following troubleshooting from bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1059952 but didn't worked. 

1] delete /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem

2] vdsm-tool configure --module certificates --force

3] openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem

4] service vdsmd restart

Error still exist!!!
[ ERROR ] Failed to execute stage 'Closing up': [ERROR]::RHEV API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Version-Release number of selected component (if applicable):
rhevm-3.5.3.1-1.4.el6ev.noarch

How reproducible:
always

Steps to Reproduce:
1. First host installed with rhevm appliance. 
2. Try to add second host with hosted-engine --deploy.
3. 

Actual results:
ConnectionError: [ERROR]::RHEV API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed


Expected results:
Host should add without error.

Additional info:

log:
---
2015-08-11 15:59:21 DEBUG otopi.plugins.ovirt_hosted_engine_setup.engine.add_host add_host._closeup:532 Connecting to the Engine
2015-08-11 15:59:21 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/engine/add_host.py", line 544, in _closeup
    ohostedcons.EngineEnv.TEMPORARY_CERT_FILE
  File "/usr/lib/python2.6/site-packages/ovirtsdk/api.py", line 154, in __init__
    url=''
  File "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", line 118, in request
    persistent_auth=self._persistent_auth)
  File "/usr/lib/python2.6/site-packages/ovirtsdk/infrastructure/proxy.py", line 146, in __doRequest
    persistent_auth=persistent_auth
  File "/usr/lib/python2.6/site-packages/ovirtsdk/web/connection.py", line 149, in doRequest
    raise ConnectionError, str(e)
ConnectionError: [ERROR]::RHEV API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-08-11 15:59:21 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Closing up': [ERROR]::RHEV API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
2015-08-11 15:59:21 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2015-08-11 15:59:21 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True'
2015-08-11 15:59:21 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<class 'ovirtsdk.infrastructure.errors.ConnectionError'>, ConnectionError('[ERROR]::RHEV API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed',), <traceback object at 0x395def0>)]'

Comment 2 Yaniv Lavi 2015-08-20 11:47:46 UTC

*** Bug 1199512 has been marked as a duplicate of this bug. ***

Comment 7 Artyom 2015-11-26 14:48:33 UTC

Verified on ovirt-hosted-engine-setup-1.3.1-1.el7ev.noarch
1) Start deploy on first host, after engine installation ⁠Replacing rhevm SSL certificate according to http://www.ovirt.org/OVirt_Administration_Guide#.E2.81.A0Replacing_oVirt_SSL_Certificate (you can install engine vm on other vm with the same hostname to get all necessary items, apache-ca.pem, apache.key.nopass, apache.cer)
2) Place new CA certificate to /etc/pki/CA/ovirtcustomcacert.pem on first host and continue deployment 
3) Answer NO on question:
"The REST API cert couldn't be trusted with the internal CA cert
          Would you like to continue in insecure mode (not recommended)?
          If not, please provide your CA cert at /etc/pki/CA/ovirtcustomcacert.pem before continuing
          (Yes, No)[No]?"
4) Finish deployment
5) Deploy second host and on the same question answer Yes
6) Finish deployment of second host

Deployment on both host succeed without any problem

Comment 9 errata-xmlrpc 2016-03-09 19:14:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0375.html