Bug 1399022 - New Jenkins app is not accessible due to a routing failure caused by TLS certificate being invalid
Summary: New Jenkins app is not accessible due to a routing failure caused by TLS cert...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Routing
Version: 3.x
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Abhishek Gupta
QA Contact: zhaliu
URL:
Whiteboard:
Depends On: 1413863
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-28 05:02 UTC by Kenichi
Modified: 2017-04-05 20:53 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-05 20:53:39 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
jenkins pod logs (404.74 KB, text/plain)
2017-01-10 05:38 UTC, zhaliu
no flags Details
deploymentconfig (7.99 KB, text/plain)
2017-01-10 05:39 UTC, zhaliu
no flags Details
jenkins-yasun-int.1ec1.dev-preview-int.openshiftapps.com.har (43.12 KB, text/plain)
2017-01-12 10:18 UTC, zhaliu
no flags Details
jenkins-jjj.1ec1.dev-preview-int.openshiftapps.com.har (43.01 KB, text/plain)
2017-01-12 10:20 UTC, zhaliu
no flags Details

Description Kenichi 2016-11-28 05:02:13 UTC
Description of problem:


Version-Release number of selected component (if applicable):

Openshift next gen developer preview

How reproducible:
always


Steps to Reproduce:
1. login https://console.preview.openshift.com
2. add to project -> Browse catalog -> Instant Apps -> jenkins-persistent
3. Using default configuration, click Create.


Actual results:

It seems that Jenkins service has started (from its log) but access to the url gets:


Application is not available

The application is currently not serving requests at this endpoint. It may not have been started or is still starting.

On Applications -> Routes, There is a warning icon with an error.

Jenkins Requested host jenkins-kenkenpa-project.44fs.preview.openshiftapps.com was rejected by the router. Reason: - spec.tls.certificate: Invalid value: "-----BEGIN CERTIFICATE-----\nMIIDIjCCAgqgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBoTELMAkGA1UEBhMCVVMx\nCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl\nZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0ExGjAYBgNVBAMMEXd3\ndy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFtcGxlQGV4YW1wbGUu\nY29tMB4XDTE1MDExMjE0MTk0MVoXDTE2MDExMjE0MTk0MVowfDEYMBYGA1UEAwwP\nd3d3LmV4YW1wbGUuY29tMQswCQYDVQQIDAJTQzELMAkGA1UEBhMCVVMxIjAgBgkq\nhkiG9w0BCQEWE2V4YW1wbGVAZXhhbXBsZS5jb20xEDAOBgNVBAoMB0V4YW1wbGUx\nEDAOBgNVBAsMB0V4YW1wbGUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMrv\ngu6ZTTefNN7jjiZbS/xvQjyXjYMN7oVXv76jbX8gjMOmg9m0xoVZZFAE4XyQDuCm\n47VRx5Qrf/YLXmB2VtCFvB0AhXr5zSeWzPwaAPrjA4ebG+LUo24ziS8KqNxrFs1M\nmNrQUgZyQC6XIe1JHXc9t+JlL5UZyZQC1IfaJulDAgMBAAGjDTALMAkGA1UdEwQC\nMAAwDQYJKoZIhvcNAQEFBQADggEBAFCi7ZlkMnESvzlZCvv82Pq6S46AAOTPXdFd\nTMvrh12E1sdVALF1P1oYFJzG1EiZ5ezOx88fEDTW+Lxb9anw5/KJzwtWcfsupf1m\nV7J0D3qKzw5C1wjzYHh9/Pz7B1D0KthQRATQCfNf8s6bbFLaw/dmiIUhHLtIH5Qc\nyfrejTZbOSP77z8NOWir+BWWgIDDB2//3AkDIQvT20vmkZRhkqSdT7et4NmXOX/j\njhPti4b2Fie0LeuvgaOdKjCpQQNrYthZHXeVlOLRhMTSk3qUczenkKTOhvP7IS9q\n+Dzv5hqgSfvMG392KWh5f8xXfJNs4W5KLbZyl901MeReiLrPH3w=\n-----END CERTIFICATE-----": error verifying certificate: x509: certificate has expired or is not yet valid. 

The certificate looks dummy.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=SC, L=Default City, O=Default Company Ltd, OU=Test CA, CN=www.exampleca.com/emailAddress=example
        Validity
            Not Before: Jan 12 14:19:41 2015 GMT
            Not After : Jan 12 14:19:41 2016 GMT
        Subject: CN=www.example.com, ST=SC, C=US/emailAddress=example, O=Example, OU=Example
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Modulus:
                    00:ca:ef:82:ee:99:4d:37:9f:34:de:e3:8e:26:5b:
                    4b:fc:6f:42:3c:97:8d:83:0d:ee:85:57:bf:be:a3:
                    6d:7f:20:8c:c3:a6:83:d9:b4:c6:85:59:64:50:04:
                    e1:7c:90:0e:e0:a6:e3:b5:51:c7:94:2b:7f:f6:0b:
                    5e:60:76:56:d0:85:bc:1d:00:85:7a:f9:cd:27:96:
                    cc:fc:1a:00:fa:e3:03:87:9b:1b:e2:d4:a3:6e:33:
                    89:2f:0a:a8:dc:6b:16:cd:4c:98:da:d0:52:06:72:
                    40:2e:97:21:ed:49:1d:77:3d:b7:e2:65:2f:95:19:
                    c9:94:02:d4:87:da:26:e9:43
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha1WithRSAEncryption
         50:a2:ed:99:64:32:71:12:bf:39:59:0a:fb:fc:d8:fa:ba:4b:
         8e:80:00:e4:cf:5d:d1:5d:4c:cb:eb:87:5d:84:d6:c7:55:00:
         b1:75:3f:5a:18:14:9c:c6:d4:48:99:e5:ec:ce:c7:cf:1f:10:
         34:d6:f8:bc:5b:f5:a9:f0:e7:f2:89:cf:0b:56:71:fb:2e:a5:
         fd:66:57:b2:74:0f:7a:8a:cf:0e:42:d7:08:f3:60:78:7d:fc:
         fc:fb:07:50:f4:2a:d8:50:44:04:d0:09:f3:5f:f2:ce:9b:6c:
         52:da:c3:f7:66:88:85:21:1c:bb:48:1f:94:1c:c9:fa:de:8d:
         36:5b:39:23:fb:ef:3f:0d:39:68:ab:f8:15:96:80:80:c3:07:
         6f:ff:dc:09:03:21:0b:d3:db:4b:e6:91:94:61:92:a4:9d:4f:
         b7:ad:e0:d9:97:39:7f:e3:8e:13:ed:8b:86:f6:16:27:b4:2d:
         eb:af:81:a3:9d:2a:30:a9:41:03:6b:62:d8:59:1d:77:95:94:
         e2:d1:84:c4:d2:93:7a:94:73:37:a7:90:a4:ce:86:f3:fb:21:
         2f:6a:f8:3c:ef:e6:1a:a0:49:fb:cc:1b:7f:76:29:68:79:7f:
         cc:57:7c:93:6c:e1:6e:4a:2d:b6:72:97:dd:35:31:e4:5e:88:
         ba:cf:1f:7c

Expected results:

Jenkins service is up running.


Additional info:

Comment 1 Ben Parees 2016-11-28 12:48:49 UTC
The jenkins template doesn't include a certificate anymore, so either online needs to update to a new template, or something else is misconfigured in online such that the router certs are bad:

https://github.com/openshift/origin/blob/master/examples/jenkins/jenkins-persistent-template.json

Comment 2 Abhishek Gupta 2016-11-29 19:53:01 UTC
This has been fixed in DevPreview INT already (3.4) and will be applied to DevPrevew Prod as a hotfix.

Comment 3 zhaliu 2017-01-09 10:27:05 UTC
Verified in the INT environment, the route can be accessed, but it prompt "java.lang.Exception: State is invalid" when having been tried to authorize that with github.

The full log is attached.

Comment 5 Miciah Dashiel Butler Masters 2017-01-09 21:37:11 UTC
I cannot reproduce the problem.  Here are the steps I followed:

1. Log in to dev-preview-int at <https://console.preview.openshift.com>.
2. At the console, click "New Project", enter a valid name, display name, and description, and click "Create".
3. Enter "jenkins" into the search field and click "Select" under "Jenkins (Persistent)".
4. Leave the default configuration values and click "Create".
5. On the command line, run `while :; do oc logs dc/jenkins -f; sleep 1; done`.
6. Wait for the "INFO: Jenkins is fully up and running" log message (it appears after about 46 seconds).
7. Back in the console, navigate "Go to overview" -> "Applications" -> "Routes" and click the route.
8. Click "Login with OpenShift" at the prompt from Jenkins.
9. Click "Login with github" at the prompt from OAuth.
10. If prompted, grant the GitHub access permissions requested by Jenkins (the prompt was skipped the second time I ran through these steps).
11. In Jenkins, navigate "Manage Jenkins" -> "System Log" -> "All Jenkins Logs".

Based on comment 3, I expect to see a Java exception and backtrace in the `oc logs` output after Step 10, but I see no errors the `oc logs` output, nor in the logs displayed by Step 11.

Note that the original description mentions an "Instant Apps" choice when browsing the catalog, but I see no such option, so instead I typed "jenkins" into the search field in order to get the "Jenkins (Persistent)" option for the template named "jenkins-persistent".  Is it possible you got a different template and image? The "jenkins-persistent" template I got uses the following image:

% oc get dc/jenkins --template=$'{{(index .spec.template.spec.containers 0).image}}\n'
registry.ops.openshift.com/openshift3/jenkins-2-rhel7@sha256:27c8e7dd29e5663d9e11ffb573f49c9c9aaac547d657e3f1f78a5cecfa824544

For the record, I see the following on "?" -> "About":

    Version
    
    OpenShift Master:
        v3.4.0.38 (online version 3.4.0.11)
    Kubernetes Master:
        v1.4.0+776c994

3.4.0.11 was built and deployed 2017-01-04, as was the image deployed by the jenkins-persistent template, so it looks like we should be using the same template and image.

Perhaps there some relevant difference between our Dev Preview or GitHub credentials, but before we look into that, let's check some simpler things.

Can you confirm that we ware using the same image and steps?

Do you see the problem consistently if you run through the above steps 2 or 3 times, or is it intermittent?

Comment 6 zhaliu 2017-01-10 05:31:48 UTC
All steps I followed are same as your steps , and the jenkins image is same too.
However, today I try a few times , the problem occurred again, so it is intermittent.

After clicking "LOGIN WITH GITHUB" the page "Authorize Access" will be open, at which point the problem appears. We find if the "Authorize Access" can be shown successfully it will work well.

The dc json file and the pod's log is saved and attached. I find some warnings about websocket, which maybe results the problem. 

Would you help check the log? it maybe is helpful to find the the reason.

Comment 7 zhaliu 2017-01-10 05:38:41 UTC
Created attachment 1238978 [details]
jenkins pod logs

Comment 8 zhaliu 2017-01-10 05:39:15 UTC
Created attachment 1238979 [details]
deploymentconfig

Comment 9 zhaliu 2017-01-10 05:44:06 UTC
And please ignore the information about user "ychww" in the jenkins logs , it is another github user just for testing.

Comment 10 Gabe Montero 2017-01-11 20:18:28 UTC
Miciah asked for an assist.  I don't have a conclusion, but can possible progress the debug / diagnosis some.

Regarding the exception in #Comment 3 and log in #Comment 4, here is the relevant stack trace snippet:

java.lang.Exception: State is invalid
	at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83)
	at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:106)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doFinishLogin(OpenShiftOAuth2SecurityRealm.java:742)


The relevant piece of the code in the plugin is the `doFinishLogin` method here:  https://github.com/openshift/jenkins-openshift-login-plugin/blob/master/src/main/java/org/openshift/jenkins/plugins/openshiftlogin/OAuthSession.java#L96-L116

This is the end of the http redirect ping pong flow that occurs between Jenkins and OpenShift Oauth support running in the OpenShift master.

In `doFinishLogin` the "State is invalid" is getting cited when the state in the auth code response does not equal the state set in the auth code request earlier in the oauth redirect ping pong flow that occurs.  As an FYI, the state on the auth code request is set here:  https://github.com/openshift/jenkins-openshift-login-plugin/blob/master/src/main/java/org/openshift/jenkins/plugins/openshiftlogin/OAuthSession.java#L88

The auth code request state (the `uuid` field) is state stored in the HttpSession in Jenkins for the given login attempt when the redirect ping/pong starts.  The redirect back to Jenkins should result in that HttpSession being accessed.

On the surface, it would appear diagnosis in minimally needed on the openshift oauth side.

I've cc:ed Jordon, Mo, and Clayton to chime in.  I suspect the next course of debug is to analyze the master logs at the time of the failure and see what is going on in the OAuth layer when this occurs.  Does that sound right?  I also wonder if the identity provider used for online (Miciah tells me it is "Keycloak") has any bearing here.

Comment 12 Jordan Liggitt 2017-01-11 20:32:31 UTC
can you capture all the network requests the browser sees (in chrome, view developer tools, network tab, check "preserve log", then walk through a login flow)

I'd want to see the original OAuth redirect from jenkins, all the intermediate requests, and the final return URL directing back to jenkins

Comment 13 Miciah Dashiel Butler Masters 2017-01-11 20:53:48 UTC
Zhao, could you gather the information requested by Jordan in comment 12? I still have not reproduced the failure (having tried with Chrome 57.0.2970 and Firefox 45.4.0).

Comment 14 zhaliu 2017-01-12 09:42:05 UTC
I have gathered the log with developer tools in version 53.0.2785.143 chrome . The logs are saved with har format and have been attached. There are two log files ,which belong to different projects.

Comment 15 zhaliu 2017-01-12 10:18:30 UTC
Created attachment 1239840 [details]
jenkins-yasun-int.1ec1.dev-preview-int.openshiftapps.com.har

Comment 16 zhaliu 2017-01-12 10:20:10 UTC
Created attachment 1239841 [details]
jenkins-jjj.1ec1.dev-preview-int.openshiftapps.com.har

Comment 17 Gabe Montero 2017-01-12 14:50:34 UTC
Took a peek at the attachment from #Comment 15 and the URL that came back from openshift oauth was indicative of an error on the oauth provider running in openshift master:

"https://jenkins-yasun-int.1ec1.dev-preview-int.openshiftapps.com/securityRealm/finishLogin?error=server_error&error_description=The+authorization+server+encountered+an+unexpected+condition+that+prevented+it+from+fulfilling+the+request."

Comment 18 Gabe Montero 2017-01-12 14:51:31 UTC
Same looking URL for the attachment from #Comment 16

Comment 19 Mo 2017-01-12 14:53:39 UTC
The master's log will be more useful here as that error message serves as a general mask for a variety of different internal oauth errors.

Comment 30 Miciah Dashiel Butler Masters 2017-01-17 17:32:15 UTC
The latest problem appears be the same problem reported in bug 1413863.

Comment 31 Miciah Dashiel Butler Masters 2017-01-19 16:24:36 UTC
Bug 1413863 has been VERIFIED, so the issue in comment 28 should be resolved.

Comment 32 Abhishek Gupta 2017-02-13 18:19:16 UTC
Moving to QA based on comment above.

Comment 33 Bing Li 2017-02-14 03:03:56 UTC
Jenkins app can be accessed normally using image:

sha256:66eea68ebc701a13527a4172feb2c4c0c9acda09efaa59821393498121d97ae0

Move to verified, thanks!


Note You need to log in before you can comment on or make changes to this bug.