Bugzilla (bugzilla.redhat.com) will be under maintenance for infrastructure upgrades and will not be unavailable on July 31st between 12:30 AM - 05:30 AM UTC. We appreciate your understanding and patience. You can follow status.redhat.com for details.
Bug 1420425 - Failed to create logging-deployer route with external certificate
Summary: Failed to create logging-deployer route with external certificate
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.4.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Jeff Cantrill
QA Contact: Xia Zhao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-02-08 15:42 UTC by Ruben Romero Montes
Modified: 2017-07-24 14:11 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
Environment:
Last Closed: 2017-04-12 19:01:06 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0903 0 normal SHIPPED_LIVE OpenShift Container Platform atomic-openshift-utils bug fix and enhancement 2017-04-12 22:45:42 UTC

Description Ruben Romero Montes 2017-02-08 15:42:15 UTC
Description of problem:
When deploying the EFK stack with a custom certificate the route is created with the wrong CA even though it has been provided. Then the validation of the certificate fails.

Version-Release number of selected component (if applicable):
OCP 3.3.x does not fail because the router does not validate the certificates by default but the CA is wrong as well.
OCP 3.4.x

Steps to Reproduce:
1. oc create secret generic logging-deployer --from-file kibana.crt=./kibana.cloudapps.example.com.crt --from-file kibana.key=./kibana.cloudapps.example.com.key
2. oc create configmap logging-deployer --from-literal kibana-hostname=kibana.apps.example.com --from-literal public-master-url=https://master.example.com:8443 --from-literal es-cluster-size=1 --from-literal es-instance-ram=1G
3. oc new-app logging-deployer-template

Actual results:
Route logging-kibana -> ExtendedValidationFailed
spec.tls.certificate: Invalid value: "\u003ccertificate data\u003e": error verifying certificate: x509: certificate signed by unknown authority

The caCertificate is equals to the destinationCACertificate, which is the Openshift CA.
This caCertificate is not the one used to generate the certificate provided.


Expected results:
The expected behaviour would be to not use any caCertificate. Maybe this is mandatory.

Additional info:
I have also tried to provide the kibana.ca.crt file but I don't know why it is not being used. See the following command:
oc create secret generic logging-deployer --from-file kibana.crt=./kibana.cloudapps.example.com.crt --from-file kibana.key=./kibana.cloudapps.example.com.key --from-file kibana.ca.crt=./kibana.cloudapps.example.com.ca.crt

And the entry in the logs:
+ kibana_keys=
+ '[' -e /etc/deploy/scratch/kibana.crt ']'
+ kibana_keys='--cert=/etc/deploy/scratch/kibana.crt --key=/etc/deploy/scratch/kibana.key'
+ oc create route reencrypt --service=logging-kibana --hostname=kibana.apps.example.com --dest-ca-cert=/etc/deploy/scratch/ca.crt --ca-cert=/etc/deploy/scratch/ca.crt --cert=/etc/deploy/scratch/kibana.crt --key=/etc/deploy/scratch/kibana.key
route "logging-kibana" created

Comment 1 Jeff Cantrill 2017-02-09 18:32:09 UTC
fix https://github.com/openshift/openshift-ansible/pull/3318

Comment 2 openshift-github-bot 2017-02-10 16:26:31 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/f1e622ae0acb52fbd7ecd1b07b26b1b6884deb65
bug 1420425. Allow setting of public facing certs for kibana in openshift_logging role

https://github.com/openshift/openshift-ansible/commit/5ee790032dae79a6dd588f53b24932248e00f9ec
Merge pull request #3318 from jcantrill/bz_1420425_external_log_certs

bug 1420425. Allow setting of public facing certs for kibana in opens…

Comment 4 Xia Zhao 2017-03-14 05:45:55 UTC
Blocked by https://bugzilla.redhat.com/show_bug.cgi?id=1431935

Comment 5 Xia Zhao 2017-03-16 09:10:49 UTC
Verify failed with openshift-ansible-3.5.35-1.git.0.7aa4728.el7.noarch, Issue got reproduced:

# oc get route
NAME             HOST/PORT                  PATH      SERVICES         PORT      TERMINATION          WILDCARD
logging-kibana   ExtendedValidationFailed             logging-kibana   <all>     reencrypt/Redirect   None

# oc describe route logging-kibana
...
Requested Host:	kibana.0316-ydc.qe.rhcloud.com
		  rejected by router router: ExtendedValidationFailed (6 minutes ago)
		    
  - spec.tls.caCertificate: Invalid value: "redacted ca certificate data": failed to parse CA certificate: Could not read any certificates
  - spec.tls.certificate: Invalid value: "redacted certificate data": error verifying certificate: x509: certificate signed by unknown authority
...

Test steps:

On master, generate the certs and keys:
# grep subdomain /etc/origin/master/master-config.yaml
  subdomain:  "0316-ydc.qe.rhcloud.com"
# openshift admin ca create-server-cert --key=a.key --cert=a.crt --hostnames='*.0316-ydc.qe.rhcloud.com' --signer-cert=/etc/origin/master/ca.crt --signer-key=/etc/origin/master/ca.key --signer-serial=/etc/origin/master/ca.serial.txt

Then copy files a.crt, a.key and the /etc/origin/master/admin.kubeconfig to /root/ directory on ansible control machine

In inventory file, define these 3 parameters:
openshift_logging_kibana_cert=/root/a.crt
openshift_logging_kibana_key=/root/a.key
openshift_logging_kibana_ca=/root/admin.kubeconfig

Then deploy logging 3.5.0:
$ ansible-playbook -i ~/inventory_logging -vvv /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/openshift_logging.yml

Test env:
# openshift version
openshift v3.5.0.53
kubernetes v1.5.2+43a9be4
etcd 3.1.0

# docker version
Client:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-13.el7.x86_64
 Go version:      go1.7.4
 Git commit:      3a094bd/1.12.6
 Built:           Fri Mar  3 15:47:24 2017
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 Package version: docker-common-1.12.6-13.el7.x86_64
 Go version:      go1.7.4
 Git commit:      3a094bd/1.12.6
 Built:           Fri Mar  3 15:47:24 2017
 OS/Arch:         linux/amd64

Comment 6 Xia Zhao 2017-03-16 10:00:04 UTC
Please kindly ignore comment #5 , just realized that I used wrong CA there.

After changing the CA file used, it passed verification, the route is back:
openshift_logging_kibana_ca=/etc/origin/master/ca.crt

# oc get route
NAME             HOST/PORT                        PATH      SERVICES         PORT      TERMINATION          WILDCARD
logging-kibana   kibana.0316-ydc.qe.rhcloud.com             logging-kibana   <all>     reencrypt/Redirect   None

# oc describe route logging-kibana
...
Requested Host:		kibana.0316-ydc.qe.rhcloud.com
			  exposed on router router 42 seconds ago
Path:			<none>
TLS Termination:	reencrypt
Insecure Policy:	Redirect
Endpoint Port:		<all endpoint ports>
...

And kibana route is accessible with log entries on its UI. Reset to verified with openshift-ansible-3.5.35-1.git.0.7aa4728.el7.noarch.

Comment 8 George Goh 2017-04-12 11:36:22 UTC
I also hit this issue on OCP 3.3, and I ran the following patch commands to clear the certificates:

oc patch -n logging route/logging-kibana -p '{"spec":{"tls":{"certificate":""}}}' 

oc patch -n logging route/logging-kibana-ops -p '{"spec":{"tls":{"certificate":""}}}'

This allows the route to be served again.

Note that this is a workaround until the bug is fixed.

Comment 9 errata-xmlrpc 2017-04-12 19:01:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0903

Comment 10 George Goh 2017-04-13 05:57:24 UTC
(In reply to George Goh from comment #8)
> I also hit this issue on OCP 3.3, and I ran the following patch commands to
> clear the certificates:
> 
> oc patch -n logging route/logging-kibana -p
> '{"spec":{"tls":{"certificate":""}}}' 
> 
> oc patch -n logging route/logging-kibana-ops -p
> '{"spec":{"tls":{"certificate":""}}}'
> 
> This allows the route to be served again.
> 
> Note that this is a workaround until the bug is fixed.

This should have read:

oc patch -n logging route/logging-kibana -p '{"spec":{"tls":{"caCertificate":""}}}' 

oc patch -n logging route/logging-kibana-ops -p '{"spec":{"tls":{"caCertificate":""}}}'


Note You need to log in before you can comment on or make changes to this bug.