Bug 1548619

Summary: Jenkins configuration not parseable when KUBE_CA containers brackets
Product: OpenShift Container Platform Reporter: Robert Bost <rbost>
Component: ImageAssignee: Gabe Montero <gmontero>
Status: CLOSED ERRATA QA Contact: Dongbo Yan <dyan>
Severity: low Docs Contact:
Priority: unspecified    
Version: 3.7.0CC: aos-bugs, gmontero, jokerman, mmccomas, rbost, wzheng, xiuwang
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Feature: strip Bag Attributes and like preceding the BEGIN CERTIFICATE line in certificates mounted into the pod; generally validate the certificate for proper format Reason: Jenkins would fail to parse certificates with Bag Attributes preceding the BEGIN CERTIFICATE line and fail to start since the openshift jenkins image adds such certificate to the kubernetes cloud configuration Result: Jenkins can now start when such certificates are introduced
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:09:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Robert Bost 2018-02-24 02:02:47 UTC
Description of problem: The KUBE_CA is injected into Jenkins' /var/lib/jenkins/config.xml file without an sanitizing for XML. If the certificate contains brackets (see below for example), it causes Jenkins to fail to start. 

Example certificate:

Bag Attributes
    localKeyID: 57 4A 32 C7 C2 B8 9E D9 E0 61 13 0C 9D 2A F9 6C A6 17 43 A0 
    friendlyName: My Certificate
subject=/C=XX/L=Default City/O=Default Company Ltd
issuer=/C=XX/L=Default City/O=Default Company Ltd
-----BEGIN CERTIFICATE-----
MIIFWjCCA0KgAwIBAgIJALNholsrU2lpMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNV
BAYTAlhYMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0RlZmF1bHQg
...
-----END CERTIFICATE-----

Jenkins failure:
java.io.IOException: Unable to read /var/lib/jenkins/config.xml
	at hudson.XmlFile.unmarshal(XmlFile.java:161)
	at jenkins.model.Jenkins.loadConfig(Jenkins.java:3055)
	at jenkins.model.Jenkins.access$1200(Jenkins.java:307)
	at jenkins.model.Jenkins$16.run(Jenkins.java:3073)
	at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
	at jenkins.model.Jenkins$7.runTask(Jenkins.java:1089)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.thoughtworks.xstream.converters.ConversionException:  : end tag name </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud> must match start tag name <Empty> from line 158 (position: TEXT seen ...ut>\n    </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>... @207:63)  :  : end tag name </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud> must match start tag name <Empty> from line 158 (position: TEXT seen ...ut>\n    </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>... @207:63) 


Version-Release number of selected component (if applicable): https://access.redhat.com/containers/#/registry.access.redhat.com/openshift3/jenkins-2-rhel7/images/v3.6.173.0.96-11


Actual results: Jenkins fails to start correctly.


Expected results: The certificate should be sanitized before being injected into XML.

Comment 3 Gabe Montero 2018-02-26 15:53:36 UTC
Robert - can you clarify exactly *how* the cert was injected into the jenkins config ?

From the stack trace, I'm getting the implication that it was injected for the k8s plugin, and hence, perhaps injected via the k8s config panels.

If so, then this needs to be filed as a jira in jenkins-land against the k8s plugin.

But please clarify exactly how it was injected and we can go from there.

thanks

Comment 4 Robert Bost 2018-02-26 16:56:06 UTC
This is the line where the concerning certificate is read in:

  https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/kube-slave-common.sh#L76

And here is where that certificate is plugged into Jenkins XML configuration:

  https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/kube-slave-common.sh#L146

Do you think this should be filed in a jira?

Comment 6 Gabe Montero 2018-02-26 21:09:56 UTC
Assuming openssl is available in our image, or could be made available, perhaps 'openssl verify <env var contents stored in a temp file>'

Pending other suggestions I'll give that a go when I circle to this.

Comment 7 Robert Bost 2018-02-27 01:26:24 UTC
(In reply to Gabe Montero from comment #6)
> Assuming openssl is available in our image, or could be made available,
> perhaps 'openssl verify <env var contents stored in a temp file>'
> 
> Pending other suggestions I'll give that a go when I circle to this.

I believe an `openssl verify` would still succeed even with the 'annotations'  (I don't know what they are actually called) above the real certificate content.

I think to strip the them out you would have to do `openssl x509 -in ...`:

# head cert.pem 
Bag Attributes
    localKeyID: 57 4A 32 C7 C2 B8 9E D9 E0 61 13 0C 9D 2A F9 6C A6 17 43 A0 
    friendlyName: My Certificate
subject=/C=XX/L=Default City/O=Default Company Ltd
issuer=/C=XX/L=Default City/O=Default Company Ltd
-----BEGIN CERTIFICATE-----
MIIFWjCCA0KgAwIBAgIJALNholsrU2lpMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNV
BAYTAlhYMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0RlZmF1bHQg
Q29tcGFueSBMdGQwHhcNMTgwMjI0MDE1ODM1WhcNMTkwMjI0MDE1ODM1WjBCMQsw
CQYDVQQGEwJYWDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZh
...

# openssl x509 -in cert.pem 
-----BEGIN CERTIFICATE-----
MIIFWjCCA0KgAwIBAgIJALNholsrU2lpMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNV
BAYTAlhYMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0RlZmF1bHQg
Q29tcGFueSBMdGQwHhcNMTgwMjI0MDE1ODM1WhcNMTkwMjI0MDE1ODM1WjBCMQsw
CQYDVQQGEwJYWDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZh
...

Comment 8 Robert Bost 2018-03-20 18:07:35 UTC
Workaround is to remove any attributes from the certificate files that would be recognized as XML. Then redeploy the certificates following: https://docs.openshift.com/container-platform/3.7/install_config/redeploying_certificates.html#redeploying-all-certificates-current-ca

Comment 9 Gabe Montero 2018-03-21 00:55:34 UTC
PR https://github.com/openshift/jenkins/pull/555 should provide the verification previously discussed here

Comment 10 Robert Bost 2018-03-21 17:52:14 UTC
I don't think https://github.com/openshift/jenkins/pull/555 will resolve the issue in this bz. A certificate with the weird 'annotations' would still pass.

For example:

# openssl x509 -in test.pem -noout
# echo $?
0
# cat test.pem 
Bag Attributes
    localKeyID: 57 4A 32 C7 C2 B8 9E D9 E0 61 13 0C 9D 2A F9 6C A6 17 43 A0 
    friendlyName: My <Weird> Certificate
subject=/C=XX/L=Default City/O=Default Company Ltd
issuer=/C=XX/L=Default City/O=Default Company Ltd
-----BEGIN CERTIFICATE-----
MIIEBDCCAuygAwIBAgIBATANBgkqhkiG9w0BAQQFADBcMQswCQYDVQQGEwJVUzEL
MAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIYXQx
DDAKBgNVBAsMA0FQUzEOMAwGA1UEAwwFcmJvc3QwHhcNMTcxMTA5MTUzODE3WhcN
MjcxMTA3MTUzODE3WjBMMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTkMxEDAOBgNV
BAoMB1JlZCBIYXQxDDAKBgNVBAsMA0FQUzEQMA4GA1UEAwwHQ2xpZW50MTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMQ4VctXYG8tozF77flbHe/YlX0q
Fo62lo4J/uRdnzjVFta/Lo6MedldnW/Wb2IrcODEOTPtqVAuvy9ag66HnWaO/DuW
C9HhyrsvzQR2oKHP7cIM7u4yI/xaeAWepBb/H8/oFBVxPTYViNn8zCW30BrUneh8
IYNY+mrl3uz5p2dq9oRW8HpE3RlArfMuCz5Cra+gQ5IdnsffaOqH8u54hluF4oOD
9nzHVSAnoMYs3g9/nt3DfgZ2uCbRQVqmEjDPuFyiRqRI2FK6b4t4JHzhBadiMSJB
/9ACFAaLaMe4xVWuODAZ2vEUuFDjmcoEL1cxw7PmGUxycIJ2lXJtjjVDK9ECAwEA
AaOB4DCB3TAJBgNVHRMEAjAAMB0GA1UdDgQWBBTJYU6/mDmFlQqUkgmmk120lB/J
ezCBjgYDVR0jBIGGMIGDgBQWLe5i+iMmbnI3vuI3n6X3tpbaC6FgpF4wXDELMAkG
A1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQK
DAdSZWQgSGF0MQwwCgYDVQQLDANBUFMxDjAMBgNVBAMMBXJib3N0ggkA9Iru65rm
mkwwEwYDVR0lBAwwCgYIKwYBBQUHAwIwCwYDVR0PBAQDAgeAMA0GCSqGSIb3DQEB
BAUAA4IBAQAGPpH0U1OzQEGyZCogodj8Jak+QyrFYa7/Kt3Wmguxytj3La+Sgu7Q
RO3QCEm/wMfdjxhcoi8V+0nw/DwWa/ajC+pAgLLyw0OkzvSSgnzIDMzOUeMNulAZ
8U3V0ibaEOMt10xnI4dUr/jRiUg27f7SsmQm6BFlud3fSuTI/Wha8yZ51sGVY/LJ
R5t1WKw3g3aMXHUkKwajRR5X6/DS9dm7pKh+MOXMWJLS1iR01KZ/0YXCi6duQ+sY
sAcLrwqzg+PbMe+44DqFec7PPDRBtHR3j7FzIveJ5BRznemC5ZpU3nsBtB8cOGWw
9DSiXyMBKEin6iHBwq5RsCU/N8YeK6MQ
-----END CERTIFICATE-----

I was thinking of doing something more like:

    # do not dump contents of cert here, but allow any error messages to be seen
    local crt_contents=$(openssl x509 -in $KUBE_CA)
    if [ $? -eq 1 ] ; then
      ....
    
Above would serve as a validation of certificate and sanitization of any stray fields. 


What do you think?

Comment 11 Gabe Montero 2018-03-21 19:10:25 UTC
Yep see what you are saying Robert ... think you are right.

I'll test the change out and assuming things are good, will update the PR.

And certainly feel free to comment in the PR directly.

Comment 12 Gabe Montero 2018-03-21 19:15:20 UTC
Wait ... if openssl x509 -in test.pem -noout had a zero RC,

won't  local crt_contents=$(openssl x509 -in $KUBE_CA) as well if the file KUBE_CA points to have the same contents?

what I am missing?

Comment 13 Gabe Montero 2018-03-21 19:18:39 UTC
Oh I see ... it strips the bag attributes

yeah in my testing I also did things like manipulate the body, etc. to get the errors and my have lost some of the results in the churn

updating the PR

Comment 14 Gabe Montero 2018-03-21 19:46:32 UTC
PR https://github.com/openshift/jenkins/pull/556 will do the last bit Robert and I  discussed here ... since the discussion was not in the PR, the prior PR got merged before we finished

Comment 15 Robert Bost 2018-03-21 20:33:58 UTC
Sorry about that, I will make sure to comment directly on PR's in the future :)

Comment 16 Gabe Montero 2018-03-21 21:53:00 UTC
OK looks like the PR has passed tests and will merge

QA should pick up a v3.10 openshift rhel image from brew that has commit 9fd08cb49568baaa1ece8b218b309b395e76b4c6 to verify this defect

Comment 19 XiuJuan Wang 2018-04-09 09:26:34 UTC
Could reproduce this issue with old jenkins images(below v3.10)

And verified this bug using jenkins-2-rhel7:v3.10.0 image.

1.Use jenkins image as s2i builder image, 

Source code: https://github.com/xiuwang/jenkins-certificate#add_verification_for_KUBE_CA_contents

KUBE_CA containers brackets

2.After the new image deployed, ssh into jenkins pod, to check if the certificate is correct
Jenkins pod is running meanwhile certificate is correct

3.Change to use branch invalid_x509_certificate which include a invalid x509 certificate to build a new jenkins image.

4.After step 3 image deployed, check jenkins pod logs.

Jenkins pod is running, and there is warnning in jenkins log
-------------jenkins log-----------------
140011190269856:error:0906D064:PEM routines:PEM_read_bio:bad base64 decode:pem_lib.c:824:
The file referenced by the KUBE_CA environment variable does not have a valid x509 certificate.
You will need to manually configure the server certificate used by the kubernetes-plugin.

Comment 21 errata-xmlrpc 2018-07-30 19:09:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816