Bug 1500471

Summary: 3.6.1 White spaces in the cert prevents Origin Metrics from starting
Product: OpenShift Container Platform Reporter: Juraci Paixão Kröhling <jcosta>
Component: HawkularAssignee: Juraci Paixão Kröhling <jcosta>
Status: CLOSED ERRATA QA Contact: Junqi Zhao <juzhao>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.6.1CC: aos-bugs, cbucur, erich, erjones, hgomes, jcantril, jcosta, juzhao, mwringe, pweil, stwalter
Target Milestone: ---   
Target Release: 3.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
When either a certificate within the chain at `serviceaccount/ca.crt` or any of the certificates within the provided truststore file contain a white space after the `BEGIN CERTIFICATE` declaration, the Java keytool rejects the certificate with an error, causing Origin Metrics to fail to start. As a workaround, Origin Metrics will now attempt to remove the spaces before feeding the certificate to the Keytool, but admins should make sure their certificates don't contain such spaces.
 Story Points: ---
Clone Of: 1471251 Environment:
Last Closed: 2017-12-07 07:12:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1471251, 1503450    
Bug Blocks: 1500464    
Attachments:
Description Flags
hawkular-metrics 3.6 pod log none

Comment 2 Junqi Zhao 2017-10-18 12:40:52 UTC 
Tested with metrics-hawkular-metrics:v3.6.173.0.56-1
env:
# openshift version
openshift v3.6.173.0.56
kubernetes v1.6.1+5115d708d7
etcd 3.2.1


Although metrics sanity testing passed, but throws out exception continuously, this does not the same with metrics 3.5, see the attached file
********************************************************************************
 [org.openshift.ping.common.stream.TokenStreamProvider] (thread-2,ee,hawkular-metrics-n15zd) Could not create trust manager for /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----  
	at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:110)
	at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
	at org.openshift.ping.common.stream.TokenStreamProvider.configureCaCert(TokenStreamProvider.java:73)
	at org.openshift.ping.common.stream.TokenStreamProvider.getSSLSocketFactory(TokenStreamProvider.java:106)
	at org.openshift.ping.common.stream.TokenStreamProvider.openStream(TokenStreamProvider.java:49)
	at org.openshift.ping.common.stream.OpenStream.call(OpenStream.java:25)
	at org.openshift.ping.common.stream.OpenStream.call(OpenStream.java:7)
	at org.openshift.ping.common.Utils.execute(Utils.java:210)
	at org.openshift.ping.common.Utils.openStream(Utils.java:50)
	at org.openshift.ping.kube.Client.getNode(Client.java:84)
	at org.openshift.ping.kube.Client.getPods(Client.java:90)
	at org.openshift.ping.kube.KubePing.doReadAll(KubePing.java:196)
	at org.openshift.ping.common.OpenshiftPing.readAll(OpenshiftPing.java:249)
	at org.openshift.ping.common.OpenshiftPing.sendMcastDiscoveryRequest(OpenshiftPing.java:201)
	at org.jgroups.protocols.PING.sendDiscoveryRequest(PING.java:62)
	at org.jgroups.protocols.PING.findMembers(PING.java:32)
	at org.jgroups.protocols.Discovery.findMembers(Discovery.java:244)
	at org.jgroups.protocols.Discovery.down(Discovery.java:388)
	at org.openshift.ping.common.OpenshiftPing.down(OpenshiftPing.java:196)
	at org.jgroups.protocols.MERGE3$InfoSender.run(MERGE3.java:381)
	at org.jgroups.util.TimeScheduler3$Task.run(TimeScheduler3.java:291)
	at org.jgroups.util.TimeScheduler3$RecurringTask.run(TimeScheduler3.java:325)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Illegal header: -----BEGIN CERTIFICATE-----  
	at sun.security.provider.X509Factory.checkHeaderFooter(X509Factory.java:646)
	at sun.security.provider.X509Factory.readOneBlock(X509Factory.java:636)
	at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:96) 
************************************************************************
Steps:
1. Change to "-----BEGIN CERTIFICATE-----  "(two spaces in the end) of /etc/origin/master/ca-bundle.crt.
2. Restart server and deploy metrics 3.6
3. #oc rsh ${HAWKULAR_METRICS_PODS};
4. Sanity testing of Metrics, it works well.
Comment 3 Junqi Zhao 2017-10-18 12:41:22 UTC 
Created attachment 1340164 [details]
hawkular-metrics 3.6 pod log
Comment 4 Juraci Paixão Kröhling 2017-10-18 13:09:48 UTC 
The cause is similar to BZ 1503462 : another Java component of the stack complains about the extra spaces. As the path is fixed to a specific cert file, we cannot easily fix this on our side:

https://github.com/jboss-openshift/openshift-ping/blob/master/kube/src/main/java/org/openshift/ping/kube/KubePing.java#L91

If we *need* to apply the workaround on a scenario involving this component, a new BZ should be opened.
Comment 5 Junqi Zhao 2017-10-19 05:56:35 UTC 
(In reply to Juraci Paixão Kröhling from comment #4)
> The cause is similar to BZ 1503462 : another Java component of the stack
> complains about the extra spaces. As the path is fixed to a specific cert
> file, we cannot easily fix this on our side:
> 
> https://github.com/jboss-openshift/openshift-ping/blob/master/kube/src/main/
> java/org/openshift/ping/kube/KubePing.java#L91
> 
> If we *need* to apply the workaround on a scenario involving this component,
> a new BZ should be opened.

Opened BZ to track: https://bugzilla.redhat.com/show_bug.cgi?id=1503931
Comment 6 Junqi Zhao 2017-10-19 06:01:45 UTC 
env and steps please see Comment 2, the exception mentioned in Comment 2 does not affect metrics function. See Comment 5
Comment 9 errata-xmlrpc 2017-12-07 07:12:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3389