Bug 1059952

Summary:	hosted-engine --deploy (additional host) will fail if the engine is not using the default self-signed CA
Product:	[oVirt] ovirt-hosted-engine-setup	Reporter:	Andrew Lau <andrew>
Component:	Plugins.PKI	Assignee:	Simone Tiraboschi <stirabos>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Artyom <alukiano>
Severity:	high	Docs Contact:
Priority:	high
Version:	1.2.0	CC:	alonbl, alukiano, andrew, bugs, didi, fdeutsch, lveyde, mdavis, rbalakri, rmartins, sbonazzo, stirabos, vvasilev, yeylon, ylavi
Target Milestone:	ovirt-3.6.1	Keywords:	Triaged, ZStream
Target Release:	1.3.1	Flags:	rule-engine: ovirt-3.6.z+ ylavi: planning_ack+ sbonazzo: devel_ack+ istein: testing_ack+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	integration
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	hosted-engine setup was always using the engine CA cert to trust the signature of the REST API (apache) cert while some user replaced that with an externally signed one. In that case the validation, and so the deployment of additional hosts, will fail. Let the user specify the local path of an external CA file or let him proceed in insecure mode if the validation with the internal CA cert was failing.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-12-16 12:24:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1199512, 1254838, 1284979

Description Andrew Lau 2014-01-31 03:58:15 UTC

Description of problem:
Per list discussion: Hosted Engine adding host SSL Failure (w/ engine custom cert)

When adding a second host to the hosted-engine after replacing the CA to use a signed cert (freeipa) the setup will fail with this error:

[ ERROR ] Failed to execute stage 'Closing up': [ERROR]::oVirt API connection failure, [Errno 1] _ssl.c:492: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Setup logs: http://www.fpaste.org/72624/13909770/

Even when the signed cert has been accepted by the system:
curl https://engine.example.net with the custom SSL cert will succeed but with the original self-signed gives the expected "insecure" message.

Reverting back to the original self-signed CA will allow the setup to continue. Imho, a self-signed CA for the engine's web ui is a deal breaker. I also don't remember seeing this issue when the engine was not a VM. 

In the interim, I setup a reverse proxy infront of the engine but it'd be nice to have something similar packaged as apart of oVirt. Possibly if the setup is communicating to the API directly, it'd be an interesting idea to have a different cert for API and HTTP engine?

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Replace hosted-engine's cert
2. Attempt to install another host to the hosted-engine

Actual results:
Setup will fail with invalid SSL

Expected results:
setup will succeed.

Additional info:

Comment 1 Alon Bar-Lev 2014-08-26 17:26:00 UTC

I thought the certificate authority fingerprint is prompted of the ssl handshake chain is prompted to user and acked before proceeding, so it should not matter what chain is received as long as user confirm it.

Comment 2 Vladimir Vasilev 2015-04-26 14:53:53 UTC

I had the same problem and as a workaround I removed "ca.crt" from z-ovirt-engine-proxy.conf (LocationMatch) and placed my custom CA to the default DocumentRoot (/var/www/html/ca.crt)

Comment 3 aiminick 2015-05-18 05:25:46 UTC

It's vervy easy to solve this problem, just run command:

vdsm-tool configure --module certificates --force

Used this you can rebuild VDSM self-sign cert, and make check:

openssl verify -CAfile /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem
/etc/pki/vdsm/certs/vdsmcert.pem: OK

Then restart the vdsmd service.

service vdsmd restart.

Comment 4 Vladimir Vasilev 2015-05-18 09:00:15 UTC

aiminick, can you explain what this command is doing?
Will it replace the default self-signed cert with correct one?
And if yes from where it knows to take the cert and CA?

Comment 5 Yedidyah Bar David 2015-05-20 06:22:31 UTC

(In reply to Andrew Lau from comment #0)
> Description of problem:
> Per list discussion: Hosted Engine adding host SSL Failure (w/ engine custom
> cert)

You probably refer to:

http://lists.ovirt.org/pipermail/users/2014-January/thread.html#20538

Comment 6 Yedidyah Bar David 2015-05-20 06:37:45 UTC

(In reply to Yedidyah Bar David from comment #5)
> (In reply to Andrew Lau from comment #0)
> > Description of problem:
> > Per list discussion: Hosted Engine adding host SSL Failure (w/ engine custom
> > cert)
> 
> You probably refer to:
> 
> http://lists.ovirt.org/pipermail/users/2014-January/thread.html#20538

Since the thread does not provide details, assuming a procedure similar to [1] was used. That is, internal CA is untouched, engine<->vdsm traffic still uses it, only httpd on engine machine uses the replaced cert (signed by a third party CA).

[1] http://www.ovirt.org/OVirt_Administration_Guide#.E2.81.A0Replacing_oVirt_SSL_Certificate

Comment 7 aiminick 2015-05-20 07:16:01 UTC

(In reply to Vladimir Vasilev from comment #4)
> aiminick, can you explain what this command is doing?
> Will it replace the default self-signed cert with correct one?
> And if yes from where it knows to take the cert and CA?

That's a self-signed for vdsm, do not need for engine, in vdsm program script, it knew you host name, so it can create a CA cert used your host name and self-signed for vdsm a cert then reconfigure it (libvirtd).

I've the same problem before, after this operate, problem solved

Comment 8 aiminick 2015-05-28 19:30:00 UTC

(In reply to Vladimir Vasilev from comment #4)
> aiminick, can you explain what this command is doing?
> Will it replace the default self-signed cert with correct one?
> And if yes from where it knows to take the cert and CA?

befor you do 

vdsm-tool configure --module certificates --force

you should delete 
/etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/certs/vdsmcert.pem

2 files above first.

Comment 9 Sandro Bonazzola 2015-09-04 09:03:15 UTC

This is an automated message.
This Bugzilla report has been opened on a version which is not maintained anymore.
Please check if this bug is still relevant in oVirt 3.5.4.
If it's not relevant anymore, please close it (you may use EOL or CURRENT RELEASE resolution)
If it's an RFE please update the version to 4.0 if still relevant.

Comment 10 Simone Tiraboschi 2015-09-11 12:58:50 UTC

hosted-engine --deploy needs to talk with the engine via the REST API witch are exposed using apache as a front-end.

To improve the security level in the communication with the REST API hosted-engine tries to validate the signature of the Apache cert with the only CA it knows, the internal oVirt CA, but it obviously fails cause the new Apache cert it's not signed by that CA.

Three different options there:
1. allow (or document how) to use two distinct https certificates for the webadmin interface (signed by your organization CA) and for the REST API (still signed by oVirt internal CA)
2. let you load your CA cert into the engine and made it available on request from the PKI servlet
3. let the user replace (and carefully document how) the oVirt internal CA (and all the related certs) with an intermediate CA issued by your organization CA

Comment 11 Yedidyah Bar David 2015-09-16 07:28:24 UTC

4. Change 'hosted-engine --deploy', if validation fails, to not abort, but prompt the user with relevant info and allow "I know what I am doing", like common web browsers do.

5. Change 'hosted-engine --deploy' to also verify against other CA certs, either in a custom place or system-wide.

Simone - makes sense?

Comment 12 Yedidyah Bar David 2015-09-16 07:30:44 UTC

Still relevant, dropping needinfo on reporter.

Comment 13 Simone Tiraboschi 2015-09-16 07:57:40 UTC

4. Absolutely makes sense.
5. Could be a bit different: on hosted-engine setup we are just using the CA cert to trust the https connection to the engine REST API via python SDK so checking more than one CA doesn't really make sense, we have just to check the correct one. Maybe it would be enough to let the user manually save his organization CA cert on a specific place before starting the deploy process and having ovirt-hosted-engine-setup not downloading the internal CA cert if another cert is already in place.

So, on my opinion, the best solution is 5 but we have to properly couple it also with oVirt node where the file-system is normally read only and so on.

Then, to make it more user friendly, on errors we have to:
- absolutely avoid interrupting the setup, 
- explain what happened,
- let the user recover:
  * manually saving his organization CA file on a specified path,
  * just proceeding in insecure mode as browsers allow you to do,

We can continue the work there https://gerrit.ovirt.org/#/c/45270/ and then backport it to 3.6 too

Comment 14 Red Hat Bugzilla Rules Engine 2015-10-19 11:02:38 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 15 Red Hat Bugzilla Rules Engine 2015-10-22 14:17:13 UTC

Target release should be placed once a package build is known to fix a issue. Since this bug is not modified, the target version has been reset. Please use target milestone to plan a fix for a oVirt release.

Comment 16 Artyom 2015-11-26 14:48:57 UTC

Verified on ovirt-hosted-engine-setup-1.3.1-1.el7ev.noarch
1) Start deploy on first host, after engine installation ⁠Replacing rhevm SSL certificate according to http://www.ovirt.org/OVirt_Administration_Guide#.E2.81.A0Replacing_oVirt_SSL_Certificate (you can install engine vm on other vm with the same hostname to get all necessary items, apache-ca.pem, apache.key.nopass, apache.cer)
2) Place new CA certificate to /etc/pki/CA/ovirtcustomcacert.pem on first host and continue deployment 
3) Answer NO on question:
"The REST API cert couldn't be trusted with the internal CA cert
          Would you like to continue in insecure mode (not recommended)?
          If not, please provide your CA cert at /etc/pki/CA/ovirtcustomcacert.pem before continuing
          (Yes, No)[No]?"
4) Finish deployment
5) Deploy second host and on the same question answer Yes
6) Finish deployment of second host

Deployment on both host succeed without any problem

Comment 17 Sandro Bonazzola 2015-12-16 12:24:06 UTC

According to verification status and target milestone this issue should be fixed in oVirt 3.6.1. Closing current release.