Robert - can you clarify exactly *how* the cert was injected into the jenkins config ?

From the stack trace, I'm getting the implication that it was injected for the k8s plugin, and hence, perhaps injected via the k8s config panels.

If so, then this needs to be filed as a jira in jenkins-land against the k8s plugin.

But please clarify exactly how it was injected and we can go from there.

thanks

This is the line where the concerning certificate is read in:

  https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/kube-slave-common.sh#L76

And here is where that certificate is plugged into Jenkins XML configuration:

  https://github.com/openshift/jenkins/blob/master/2/contrib/jenkins/kube-slave-common.sh#L146

Do you think this should be filed in a jira?

Assuming openssl is available in our image, or could be made available, perhaps 'openssl verify <env var contents stored in a temp file>'

Pending other suggestions I'll give that a go when I circle to this.

(In reply to Gabe Montero from comment #6)
> Assuming openssl is available in our image, or could be made available,
> perhaps 'openssl verify <env var contents stored in a temp file>'
> 
> Pending other suggestions I'll give that a go when I circle to this.

I believe an `openssl verify` would still succeed even with the 'annotations'  (I don't know what they are actually called) above the real certificate content.

I think to strip the them out you would have to do `openssl x509 -in ...`:

# head cert.pem 
Bag Attributes
    localKeyID: 57 4A 32 C7 C2 B8 9E D9 E0 61 13 0C 9D 2A F9 6C A6 17 43 A0 
    friendlyName: My Certificate
subject=/C=XX/L=Default City/O=Default Company Ltd
issuer=/C=XX/L=Default City/O=Default Company Ltd
-----BEGIN CERTIFICATE-----
MIIFWjCCA0KgAwIBAgIJALNholsrU2lpMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNV
BAYTAlhYMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0RlZmF1bHQg
Q29tcGFueSBMdGQwHhcNMTgwMjI0MDE1ODM1WhcNMTkwMjI0MDE1ODM1WjBCMQsw
CQYDVQQGEwJYWDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZh
...

# openssl x509 -in cert.pem 
-----BEGIN CERTIFICATE-----
MIIFWjCCA0KgAwIBAgIJALNholsrU2lpMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNV
BAYTAlhYMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0RlZmF1bHQg
Q29tcGFueSBMdGQwHhcNMTgwMjI0MDE1ODM1WhcNMTkwMjI0MDE1ODM1WjBCMQsw
CQYDVQQGEwJYWDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZh
...

Workaround is to remove any attributes from the certificate files that would be recognized as XML. Then redeploy the certificates following: https://docs.openshift.com/container-platform/3.7/install_config/redeploying_certificates.html#redeploying-all-certificates-current-ca

PR https://github.com/openshift/jenkins/pull/555 should provide the verification previously discussed here

I don't think https://github.com/openshift/jenkins/pull/555 will resolve the issue in this bz. A certificate with the weird 'annotations' would still pass.

For example:

# openssl x509 -in test.pem -noout
# echo $?
0
# cat test.pem 
Bag Attributes
    localKeyID: 57 4A 32 C7 C2 B8 9E D9 E0 61 13 0C 9D 2A F9 6C A6 17 43 A0 
    friendlyName: My <Weird> Certificate
subject=/C=XX/L=Default City/O=Default Company Ltd
issuer=/C=XX/L=Default City/O=Default Company Ltd
-----BEGIN CERTIFICATE-----
MIIEBDCCAuygAwIBAgIBATANBgkqhkiG9w0BAQQFADBcMQswCQYDVQQGEwJVUzEL
MAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIYXQx
DDAKBgNVBAsMA0FQUzEOMAwGA1UEAwwFcmJvc3QwHhcNMTcxMTA5MTUzODE3WhcN
MjcxMTA3MTUzODE3WjBMMQswCQYDVQQGEwJVUzELMAkGA1UECAwCTkMxEDAOBgNV
BAoMB1JlZCBIYXQxDDAKBgNVBAsMA0FQUzEQMA4GA1UEAwwHQ2xpZW50MTCCASIw
DQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMQ4VctXYG8tozF77flbHe/YlX0q
Fo62lo4J/uRdnzjVFta/Lo6MedldnW/Wb2IrcODEOTPtqVAuvy9ag66HnWaO/DuW
C9HhyrsvzQR2oKHP7cIM7u4yI/xaeAWepBb/H8/oFBVxPTYViNn8zCW30BrUneh8
IYNY+mrl3uz5p2dq9oRW8HpE3RlArfMuCz5Cra+gQ5IdnsffaOqH8u54hluF4oOD
9nzHVSAnoMYs3g9/nt3DfgZ2uCbRQVqmEjDPuFyiRqRI2FK6b4t4JHzhBadiMSJB
/9ACFAaLaMe4xVWuODAZ2vEUuFDjmcoEL1cxw7PmGUxycIJ2lXJtjjVDK9ECAwEA
AaOB4DCB3TAJBgNVHRMEAjAAMB0GA1UdDgQWBBTJYU6/mDmFlQqUkgmmk120lB/J
ezCBjgYDVR0jBIGGMIGDgBQWLe5i+iMmbnI3vuI3n6X3tpbaC6FgpF4wXDELMAkG
A1UEBhMCVVMxCzAJBgNVBAgMAk5DMRAwDgYDVQQHDAdSYWxlaWdoMRAwDgYDVQQK
DAdSZWQgSGF0MQwwCgYDVQQLDANBUFMxDjAMBgNVBAMMBXJib3N0ggkA9Iru65rm
mkwwEwYDVR0lBAwwCgYIKwYBBQUHAwIwCwYDVR0PBAQDAgeAMA0GCSqGSIb3DQEB
BAUAA4IBAQAGPpH0U1OzQEGyZCogodj8Jak+QyrFYa7/Kt3Wmguxytj3La+Sgu7Q
RO3QCEm/wMfdjxhcoi8V+0nw/DwWa/ajC+pAgLLyw0OkzvSSgnzIDMzOUeMNulAZ
8U3V0ibaEOMt10xnI4dUr/jRiUg27f7SsmQm6BFlud3fSuTI/Wha8yZ51sGVY/LJ
R5t1WKw3g3aMXHUkKwajRR5X6/DS9dm7pKh+MOXMWJLS1iR01KZ/0YXCi6duQ+sY
sAcLrwqzg+PbMe+44DqFec7PPDRBtHR3j7FzIveJ5BRznemC5ZpU3nsBtB8cOGWw
9DSiXyMBKEin6iHBwq5RsCU/N8YeK6MQ
-----END CERTIFICATE-----

I was thinking of doing something more like:

    # do not dump contents of cert here, but allow any error messages to be seen
    local crt_contents=$(openssl x509 -in $KUBE_CA)
    if [ $? -eq 1 ] ; then
      ....
    
Above would serve as a validation of certificate and sanitization of any stray fields. 


What do you think?

Yep see what you are saying Robert ... think you are right.

I'll test the change out and assuming things are good, will update the PR.

And certainly feel free to comment in the PR directly.

Wait ... if openssl x509 -in test.pem -noout had a zero RC,

won't  local crt_contents=$(openssl x509 -in $KUBE_CA) as well if the file KUBE_CA points to have the same contents?

what I am missing?

Oh I see ... it strips the bag attributes

yeah in my testing I also did things like manipulate the body, etc. to get the errors and my have lost some of the results in the churn

updating the PR

PR https://github.com/openshift/jenkins/pull/556 will do the last bit Robert and I  discussed here ... since the discussion was not in the PR, the prior PR got merged before we finished

Sorry about that, I will make sure to comment directly on PR's in the future :)

OK looks like the PR has passed tests and will merge

QA should pick up a v3.10 openshift rhel image from brew that has commit 9fd08cb49568baaa1ece8b218b309b395e76b4c6 to verify this defect

Could reproduce this issue with old jenkins images(below v3.10)

And verified this bug using jenkins-2-rhel7:v3.10.0 image.

1.Use jenkins image as s2i builder image, 

Source code: https://github.com/xiuwang/jenkins-certificate#add_verification_for_KUBE_CA_contents

KUBE_CA containers brackets

2.After the new image deployed, ssh into jenkins pod, to check if the certificate is correct
Jenkins pod is running meanwhile certificate is correct

3.Change to use branch invalid_x509_certificate which include a invalid x509 certificate to build a new jenkins image.

4.After step 3 image deployed, check jenkins pod logs.

Jenkins pod is running, and there is warnning in jenkins log
-------------jenkins log-----------------
140011190269856:error:0906D064:PEM routines:PEM_read_bio:bad base64 decode:pem_lib.c:824:
The file referenced by the KUBE_CA environment variable does not have a valid x509 certificate.
You will need to manually configure the server certificate used by the kubernetes-plugin.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816