Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1420425

Summary: Failed to create logging-deployer route with external certificate
Product: OpenShift Container Platform Reporter: Ruben Romero Montes <rromerom>
Component: InstallerAssignee: Jeff Cantrill <jcantril>
Status: CLOSED ERRATA QA Contact: Xia Zhao <xiazhao>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.4.1CC: aos-bugs, george.goh, jcantril, jokerman, mmccomas, myllynen, pruan, pweil
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-12 19:01:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ruben Romero Montes 2017-02-08 15:42:15 UTC
Description of problem:
When deploying the EFK stack with a custom certificate the route is created with the wrong CA even though it has been provided. Then the validation of the certificate fails.

Version-Release number of selected component (if applicable):
OCP 3.3.x does not fail because the router does not validate the certificates by default but the CA is wrong as well.
OCP 3.4.x

Steps to Reproduce:
1. oc create secret generic logging-deployer --from-file kibana.crt=./kibana.cloudapps.example.com.crt --from-file kibana.key=./kibana.cloudapps.example.com.key
2. oc create configmap logging-deployer --from-literal kibana-hostname=kibana.apps.example.com --from-literal public-master-url=https://master.example.com:8443 --from-literal es-cluster-size=1 --from-literal es-instance-ram=1G
3. oc new-app logging-deployer-template

Actual results:
Route logging-kibana -> ExtendedValidationFailed
spec.tls.certificate: Invalid value: "\u003ccertificate data\u003e": error verifying certificate: x509: certificate signed by unknown authority

The caCertificate is equals to the destinationCACertificate, which is the Openshift CA.
This caCertificate is not the one used to generate the certificate provided.


Expected results:
The expected behaviour would be to not use any caCertificate. Maybe this is mandatory.

Additional info:
I have also tried to provide the kibana.ca.crt file but I don't know why it is not being used. See the following command:
oc create secret generic logging-deployer --from-file kibana.crt=./kibana.cloudapps.example.com.crt --from-file kibana.key=./kibana.cloudapps.example.com.key --from-file kibana.ca.crt=./kibana.cloudapps.example.com.ca.crt

And the entry in the logs:
+ kibana_keys=
+ '[' -e /etc/deploy/scratch/kibana.crt ']'
+ kibana_keys='--cert=/etc/deploy/scratch/kibana.crt --key=/etc/deploy/scratch/kibana.key'
+ oc create route reencrypt --service=logging-kibana --hostname=kibana.apps.example.com --dest-ca-cert=/etc/deploy/scratch/ca.crt --ca-cert=/etc/deploy/scratch/ca.crt --cert=/etc/deploy/scratch/kibana.crt --key=/etc/deploy/scratch/kibana.key
route "logging-kibana" created

Comment 1 Jeff Cantrill 2017-02-09 18:32:09 UTC
fix https://github.com/openshift/openshift-ansible/pull/3318

Comment 2 openshift-github-bot 2017-02-10 16:26:31 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/f1e622ae0acb52fbd7ecd1b07b26b1b6884deb65
bug 1420425. Allow setting of public facing certs for kibana in openshift_logging role

https://github.com/openshift/openshift-ansible/commit/5ee790032dae79a6dd588f53b24932248e00f9ec
Merge pull request #3318 from jcantrill/bz_1420425_external_log_certs

bug 1420425. Allow setting of public facing certs for kibana in opens…

Comment 4 Xia Zhao 2017-03-14 05:45:55 UTC
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1431935

Comment 5 Xia Zhao 2017-03-16 09:10:49 UTC
Verify failed with openshift-ansible-3.5.35-1.git.0.7aa4728.el7.noarch, Issue got reproduced:

# oc get route
NAME             HOST/PORT                  PATH      SERVICES         PORT      TERMINATION          WILDCARD
logging-kibana   ExtendedValidationFailed             logging-kibana   <all>     reencrypt/Redirect   None

# oc describe route logging-kibana
...
Requested Host:	kibana.0316-ydc.qe.rhcloud.com
		  rejected by router router: ExtendedValidationFailed (6 minutes ago)
		    
  - spec.tls.caCertificate: Invalid value: "redacted ca certificate data": failed to parse CA certificate: Could not read any certificates
  - spec.tls.certificate: Invalid value: "redacted certificate data": error verifying certificate: x509: certificate signed by unknown authority
...

Test steps:

On master, generate the certs and keys:
# grep subdomain /etc/origin/master/master-config.yaml
  subdomain:  "0316-ydc.qe.rhcloud.com"
# openshift admin ca create-server-cert --key=a.key --cert=a.crt --hostnames='*.0316-ydc.qe.rhcloud.com' --signer-cert=/etc/origin/master/ca.crt --signer-key=/etc/origin/master/ca.key --signer-serial=/etc/origin/master/ca.serial.txt

Then copy files a.crt, a.key and the /etc/origin/master/admin.kubeconfig to /root/ directory on ansible control machine

In inventory file, define these 3 parameters:
openshift_logging_kibana_cert=/root/a.crt
openshift_logging_kibana_key=/root/a.key
openshift_logging_kibana_ca=/root/admin.kubeconfig

Then deploy logging 3.5.0:
$ ansible-playbook -i ~/inventory_logging -vvv /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/openshift_logging.yml

Test env:
# openshift version
openshift v3.5.0.53
kubernetes v1.5.2+43a9be4
etcd 3.1.0

# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-13.el7.x86_64
 Go version:      go1.7.4
 Git commit:      3a094bd/1.12.6
 Built:           Fri Mar  3 15:47:24 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-13.el7.x86_64
 Go version:      go1.7.4
 Git commit:      3a094bd/1.12.6
 Built:           Fri Mar  3 15:47:24 2017
 OS/Arch:         linux/amd64

Comment 6 Xia Zhao 2017-03-16 10:00:04 UTC
Please kindly ignore comment #5 , just realized that I used wrong CA there.

After changing the CA file used, it passed verification, the route is back:
openshift_logging_kibana_ca=/etc/origin/master/ca.crt

# oc get route
NAME             HOST/PORT                        PATH      SERVICES         PORT      TERMINATION          WILDCARD
logging-kibana   kibana.0316-ydc.qe.rhcloud.com             logging-kibana   <all>     reencrypt/Redirect   None

# oc describe route logging-kibana
...
Requested Host:		kibana.0316-ydc.qe.rhcloud.com
			  exposed on router router 42 seconds ago
Path:			<none>
TLS Termination:	reencrypt
Insecure Policy:	Redirect
Endpoint Port:		<all endpoint ports>
...

And kibana route is accessible with log entries on its UI. Reset to verified with openshift-ansible-3.5.35-1.git.0.7aa4728.el7.noarch.

Comment 8 George Goh 2017-04-12 11:36:22 UTC
I also hit this issue on OCP 3.3, and I ran the following patch commands to clear the certificates:

oc patch -n logging route/logging-kibana -p '{"spec":{"tls":{"certificate":""}}}' 

oc patch -n logging route/logging-kibana-ops -p '{"spec":{"tls":{"certificate":""}}}'

This allows the route to be served again.

Note that this is a workaround until the bug is fixed.

Comment 9 errata-xmlrpc 2017-04-12 19:01:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903

Comment 10 George Goh 2017-04-13 05:57:24 UTC
(In reply to George Goh from comment #8)
> I also hit this issue on OCP 3.3, and I ran the following patch commands to
> clear the certificates:
> 
> oc patch -n logging route/logging-kibana -p
> '{"spec":{"tls":{"certificate":""}}}' 
> 
> oc patch -n logging route/logging-kibana-ops -p
> '{"spec":{"tls":{"certificate":""}}}'
> 
> This allows the route to be served again.
> 
> Note that this is a workaround until the bug is fixed.

This should have read:

oc patch -n logging route/logging-kibana -p '{"spec":{"tls":{"caCertificate":""}}}' 

oc patch -n logging route/logging-kibana-ops -p '{"spec":{"tls":{"caCertificate":""}}}'