Bug 1500471 - 3.6.1 White spaces in the cert prevents Origin Metrics from starting
Summary: 3.6.1 White spaces in the cert prevents Origin Metrics from starting
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Hawkular
Version: 3.6.1
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.6.z
Assignee: Juraci Paixão Kröhling
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On: 1471251 1503450
Blocks: 1500464
TreeView+ depends on / blocked
 
Reported: 2017-10-10 16:06 UTC by Juraci Paixão Kröhling
Modified: 2020-12-14 10:27 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
When either a certificate within the chain at `serviceaccount/ca.crt` or any of the certificates within the provided truststore file contain a white space after the `BEGIN CERTIFICATE` declaration, the Java keytool rejects the certificate with an error, causing Origin Metrics to fail to start. As a workaround, Origin Metrics will now attempt to remove the spaces before feeding the certificate to the Keytool, but admins should make sure their certificates don't contain such spaces.
Clone Of: 1471251
Environment:
Last Closed: 2017-12-07 07:12:13 UTC
Target Upstream Version:


Attachments (Terms of Use)
hawkular-metrics 3.6 pod log (233.13 KB, text/plain)
2017-10-18 12:41 UTC, Junqi Zhao
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3389 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Enterprise security, bug fix, and enhancement update 2017-12-07 12:09:10 UTC

Comment 2 Junqi Zhao 2017-10-18 12:40:52 UTC
Tested with metrics-hawkular-metrics:v3.6.173.0.56-1
env:
# openshift version
openshift v3.6.173.0.56
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


Although metrics sanity testing passed, but throws out exception continuously, this does not the same with metrics 3.5, see the attached file
********************************************************************************
 [org.openshift.ping.common.stream.TokenStreamProvider] (thread-2,ee,hawkular-metrics-n15zd) Could not create trust manager for /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----  
	at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
	at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
	at org.openshift.ping.common.stream.TokenStreamProvider.configureCaCert(TokenStreamProvider.java:73)
	at org.openshift.ping.common.stream.TokenStreamProvider.getSSLSocketFactory(TokenStreamProvider.java:106)
	at org.openshift.ping.common.stream.TokenStreamProvider.openStream(TokenStreamProvider.java:49)
	at org.openshift.ping.common.stream.OpenStream.call(OpenStream.java:25)
	at org.openshift.ping.common.stream.OpenStream.call(OpenStream.java:7)
	at org.openshift.ping.common.Utils.execute(Utils.java:210)
	at org.openshift.ping.common.Utils.openStream(Utils.java:50)
	at org.openshift.ping.kube.Client.getNode(Client.java:84)
	at org.openshift.ping.kube.Client.getPods(Client.java:90)
	at org.openshift.ping.kube.KubePing.doReadAll(KubePing.java:196)
	at org.openshift.ping.common.OpenshiftPing.readAll(OpenshiftPing.java:249)
	at org.openshift.ping.common.OpenshiftPing.sendMcastDiscoveryRequest(OpenshiftPing.java:201)
	at org.jgroups.protocols.PING.sendDiscoveryRequest(PING.java:62)
	at org.jgroups.protocols.PING.findMembers(PING.java:32)
	at org.jgroups.protocols.Discovery.findMembers(Discovery.java:244)
	at org.jgroups.protocols.Discovery.down(Discovery.java:388)
	at org.openshift.ping.common.OpenshiftPing.down(OpenshiftPing.java:196)
	at org.jgroups.protocols.MERGE3$InfoSender.run(MERGE3.java:381)
	at org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:291)
	at org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:325)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----  
	at sun.security.provider.X509Factory.checkHeaderFooter(X509Factory.java:646)
	at sun.security.provider.X509Factory.readOneBlock(X509Factory.java:636)
	at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:96) 
************************************************************************
Steps:
1. Change to "-----BEGIN CERTIFICATE-----  "(two spaces in the end) of /etc/origin/master/ca-bundle.crt.
2. Restart server and deploy metrics 3.6
3. #oc rsh ${HAWKULAR_METRICS_PODS};
4. Sanity testing of Metrics, it works well.

Comment 3 Junqi Zhao 2017-10-18 12:41:22 UTC
Created attachment 1340164 [details]
hawkular-metrics 3.6 pod log

Comment 4 Juraci Paixão Kröhling 2017-10-18 13:09:48 UTC
The cause is similar to BZ 1503462 : another Java component of the stack complains about the extra spaces. As the path is fixed to a specific cert file, we cannot easily fix this on our side:

https://github.com/jboss-openshift/openshift-ping/blob/master/kube/src/main/java/org/openshift/ping/kube/KubePing.java#L91

If we *need* to apply the workaround on a scenario involving this component, a new BZ should be opened.

Comment 5 Junqi Zhao 2017-10-19 05:56:35 UTC
(In reply to Juraci Paixão Kröhling from comment #4)
> The cause is similar to BZ 1503462 : another Java component of the stack
> complains about the extra spaces. As the path is fixed to a specific cert
> file, we cannot easily fix this on our side:
> 
> https://github.com/jboss-openshift/openshift-ping/blob/master/kube/src/main/
> java/org/openshift/ping/kube/KubePing.java#L91
> 
> If we *need* to apply the workaround on a scenario involving this component,
> a new BZ should be opened.

Opened BZ to track: https://bugzilla.redhat.com/show_bug.cgi?id=1503931

Comment 6 Junqi Zhao 2017-10-19 06:01:45 UTC
env and steps please see Comment 2, the exception mentioned in Comment 2 does not affect metrics function. See Comment 5

Comment 9 errata-xmlrpc 2017-12-07 07:12:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389


Note You need to log in before you can comment on or make changes to this bug.