Description of problem: Version-Release number of selected component (if applicable): Openshift next gen developer preview How reproducible: always Steps to Reproduce: 1. login https://console.preview.openshift.com 2. add to project -> Browse catalog -> Instant Apps -> jenkins-persistent 3. Using default configuration, click Create. Actual results: It seems that Jenkins service has started (from its log) but access to the url gets: Application is not available The application is currently not serving requests at this endpoint. It may not have been started or is still starting. On Applications -> Routes, There is a warning icon with an error. Jenkins Requested host jenkins-kenkenpa-project.44fs.preview.openshiftapps.com was rejected by the router. Reason: - spec.tls.certificate: Invalid value: "-----BEGIN CERTIFICATE-----\nMIIDIjCCAgqgAwIBAgIBATANBgkqhkiG9w0BAQUFADCBoTELMAkGA1UEBhMCVVMx\nCzAJBgNVBAgMAlNDMRUwEwYDVQQHDAxEZWZhdWx0IENpdHkxHDAaBgNVBAoME0Rl\nZmF1bHQgQ29tcGFueSBMdGQxEDAOBgNVBAsMB1Rlc3QgQ0ExGjAYBgNVBAMMEXd3\ndy5leGFtcGxlY2EuY29tMSIwIAYJKoZIhvcNAQkBFhNleGFtcGxlQGV4YW1wbGUu\nY29tMB4XDTE1MDExMjE0MTk0MVoXDTE2MDExMjE0MTk0MVowfDEYMBYGA1UEAwwP\nd3d3LmV4YW1wbGUuY29tMQswCQYDVQQIDAJTQzELMAkGA1UEBhMCVVMxIjAgBgkq\nhkiG9w0BCQEWE2V4YW1wbGVAZXhhbXBsZS5jb20xEDAOBgNVBAoMB0V4YW1wbGUx\nEDAOBgNVBAsMB0V4YW1wbGUwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAMrv\ngu6ZTTefNN7jjiZbS/xvQjyXjYMN7oVXv76jbX8gjMOmg9m0xoVZZFAE4XyQDuCm\n47VRx5Qrf/YLXmB2VtCFvB0AhXr5zSeWzPwaAPrjA4ebG+LUo24ziS8KqNxrFs1M\nmNrQUgZyQC6XIe1JHXc9t+JlL5UZyZQC1IfaJulDAgMBAAGjDTALMAkGA1UdEwQC\nMAAwDQYJKoZIhvcNAQEFBQADggEBAFCi7ZlkMnESvzlZCvv82Pq6S46AAOTPXdFd\nTMvrh12E1sdVALF1P1oYFJzG1EiZ5ezOx88fEDTW+Lxb9anw5/KJzwtWcfsupf1m\nV7J0D3qKzw5C1wjzYHh9/Pz7B1D0KthQRATQCfNf8s6bbFLaw/dmiIUhHLtIH5Qc\nyfrejTZbOSP77z8NOWir+BWWgIDDB2//3AkDIQvT20vmkZRhkqSdT7et4NmXOX/j\njhPti4b2Fie0LeuvgaOdKjCpQQNrYthZHXeVlOLRhMTSk3qUczenkKTOhvP7IS9q\n+Dzv5hqgSfvMG392KWh5f8xXfJNs4W5KLbZyl901MeReiLrPH3w=\n-----END CERTIFICATE-----": error verifying certificate: x509: certificate has expired or is not yet valid. The certificate looks dummy. Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, ST=SC, L=Default City, O=Default Company Ltd, OU=Test CA, CN=www.exampleca.com/emailAddress=example Validity Not Before: Jan 12 14:19:41 2015 GMT Not After : Jan 12 14:19:41 2016 GMT Subject: CN=www.example.com, ST=SC, C=US/emailAddress=example, O=Example, OU=Example Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (1024 bit) Modulus: 00:ca:ef:82:ee:99:4d:37:9f:34:de:e3:8e:26:5b: 4b:fc:6f:42:3c:97:8d:83:0d:ee:85:57:bf:be:a3: 6d:7f:20:8c:c3:a6:83:d9:b4:c6:85:59:64:50:04: e1:7c:90:0e:e0:a6:e3:b5:51:c7:94:2b:7f:f6:0b: 5e:60:76:56:d0:85:bc:1d:00:85:7a:f9:cd:27:96: cc:fc:1a:00:fa:e3:03:87:9b:1b:e2:d4:a3:6e:33: 89:2f:0a:a8:dc:6b:16:cd:4c:98:da:d0:52:06:72: 40:2e:97:21:ed:49:1d:77:3d:b7:e2:65:2f:95:19: c9:94:02:d4:87:da:26:e9:43 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Signature Algorithm: sha1WithRSAEncryption 50:a2:ed:99:64:32:71:12:bf:39:59:0a:fb:fc:d8:fa:ba:4b: 8e:80:00:e4:cf:5d:d1:5d:4c:cb:eb:87:5d:84:d6:c7:55:00: b1:75:3f:5a:18:14:9c:c6:d4:48:99:e5:ec:ce:c7:cf:1f:10: 34:d6:f8:bc:5b:f5:a9:f0:e7:f2:89:cf:0b:56:71:fb:2e:a5: fd:66:57:b2:74:0f:7a:8a:cf:0e:42:d7:08:f3:60:78:7d:fc: fc:fb:07:50:f4:2a:d8:50:44:04:d0:09:f3:5f:f2:ce:9b:6c: 52:da:c3:f7:66:88:85:21:1c:bb:48:1f:94:1c:c9:fa:de:8d: 36:5b:39:23:fb:ef:3f:0d:39:68:ab:f8:15:96:80:80:c3:07: 6f:ff:dc:09:03:21:0b:d3:db:4b:e6:91:94:61:92:a4:9d:4f: b7:ad:e0:d9:97:39:7f:e3:8e:13:ed:8b:86:f6:16:27:b4:2d: eb:af:81:a3:9d:2a:30:a9:41:03:6b:62:d8:59:1d:77:95:94: e2:d1:84:c4:d2:93:7a:94:73:37:a7:90:a4:ce:86:f3:fb:21: 2f:6a:f8:3c:ef:e6:1a:a0:49:fb:cc:1b:7f:76:29:68:79:7f: cc:57:7c:93:6c:e1:6e:4a:2d:b6:72:97:dd:35:31:e4:5e:88: ba:cf:1f:7c Expected results: Jenkins service is up running. Additional info:
The jenkins template doesn't include a certificate anymore, so either online needs to update to a new template, or something else is misconfigured in online such that the router certs are bad: https://github.com/openshift/origin/blob/master/examples/jenkins/jenkins-persistent-template.json
This has been fixed in DevPreview INT already (3.4) and will be applied to DevPrevew Prod as a hotfix.
Verified in the INT environment, the route can be accessed, but it prompt "java.lang.Exception: State is invalid" when having been tried to authorize that with github. The full log is attached.
I cannot reproduce the problem. Here are the steps I followed: 1. Log in to dev-preview-int at <https://console.preview.openshift.com>. 2. At the console, click "New Project", enter a valid name, display name, and description, and click "Create". 3. Enter "jenkins" into the search field and click "Select" under "Jenkins (Persistent)". 4. Leave the default configuration values and click "Create". 5. On the command line, run `while :; do oc logs dc/jenkins -f; sleep 1; done`. 6. Wait for the "INFO: Jenkins is fully up and running" log message (it appears after about 46 seconds). 7. Back in the console, navigate "Go to overview" -> "Applications" -> "Routes" and click the route. 8. Click "Login with OpenShift" at the prompt from Jenkins. 9. Click "Login with github" at the prompt from OAuth. 10. If prompted, grant the GitHub access permissions requested by Jenkins (the prompt was skipped the second time I ran through these steps). 11. In Jenkins, navigate "Manage Jenkins" -> "System Log" -> "All Jenkins Logs". Based on comment 3, I expect to see a Java exception and backtrace in the `oc logs` output after Step 10, but I see no errors the `oc logs` output, nor in the logs displayed by Step 11. Note that the original description mentions an "Instant Apps" choice when browsing the catalog, but I see no such option, so instead I typed "jenkins" into the search field in order to get the "Jenkins (Persistent)" option for the template named "jenkins-persistent". Is it possible you got a different template and image? The "jenkins-persistent" template I got uses the following image: % oc get dc/jenkins --template=$'{{(index .spec.template.spec.containers 0).image}}\n' registry.ops.openshift.com/openshift3/jenkins-2-rhel7@sha256:27c8e7dd29e5663d9e11ffb573f49c9c9aaac547d657e3f1f78a5cecfa824544 For the record, I see the following on "?" -> "About": Version OpenShift Master: v3.4.0.38 (online version 3.4.0.11) Kubernetes Master: v1.4.0+776c994 3.4.0.11 was built and deployed 2017-01-04, as was the image deployed by the jenkins-persistent template, so it looks like we should be using the same template and image. Perhaps there some relevant difference between our Dev Preview or GitHub credentials, but before we look into that, let's check some simpler things. Can you confirm that we ware using the same image and steps? Do you see the problem consistently if you run through the above steps 2 or 3 times, or is it intermittent?
All steps I followed are same as your steps , and the jenkins image is same too. However, today I try a few times , the problem occurred again, so it is intermittent. After clicking "LOGIN WITH GITHUB" the page "Authorize Access" will be open, at which point the problem appears. We find if the "Authorize Access" can be shown successfully it will work well. The dc json file and the pod's log is saved and attached. I find some warnings about websocket, which maybe results the problem. Would you help check the log? it maybe is helpful to find the the reason.
Created attachment 1238978 [details] jenkins pod logs
Created attachment 1238979 [details] deploymentconfig
And please ignore the information about user "ychww" in the jenkins logs , it is another github user just for testing.
Miciah asked for an assist. I don't have a conclusion, but can possible progress the debug / diagnosis some. Regarding the exception in #Comment 3 and log in #Comment 4, here is the relevant stack trace snippet: java.lang.Exception: State is invalid at org.kohsuke.stapler.HttpResponses.error(HttpResponses.java:83) at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:106) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doFinishLogin(OpenShiftOAuth2SecurityRealm.java:742) The relevant piece of the code in the plugin is the `doFinishLogin` method here: https://github.com/openshift/jenkins-openshift-login-plugin/blob/master/src/main/java/org/openshift/jenkins/plugins/openshiftlogin/OAuthSession.java#L96-L116 This is the end of the http redirect ping pong flow that occurs between Jenkins and OpenShift Oauth support running in the OpenShift master. In `doFinishLogin` the "State is invalid" is getting cited when the state in the auth code response does not equal the state set in the auth code request earlier in the oauth redirect ping pong flow that occurs. As an FYI, the state on the auth code request is set here: https://github.com/openshift/jenkins-openshift-login-plugin/blob/master/src/main/java/org/openshift/jenkins/plugins/openshiftlogin/OAuthSession.java#L88 The auth code request state (the `uuid` field) is state stored in the HttpSession in Jenkins for the given login attempt when the redirect ping/pong starts. The redirect back to Jenkins should result in that HttpSession being accessed. On the surface, it would appear diagnosis in minimally needed on the openshift oauth side. I've cc:ed Jordon, Mo, and Clayton to chime in. I suspect the next course of debug is to analyze the master logs at the time of the failure and see what is going on in the OAuth layer when this occurs. Does that sound right? I also wonder if the identity provider used for online (Miciah tells me it is "Keycloak") has any bearing here.
can you capture all the network requests the browser sees (in chrome, view developer tools, network tab, check "preserve log", then walk through a login flow) I'd want to see the original OAuth redirect from jenkins, all the intermediate requests, and the final return URL directing back to jenkins
Zhao, could you gather the information requested by Jordan in comment 12? I still have not reproduced the failure (having tried with Chrome 57.0.2970 and Firefox 45.4.0).
I have gathered the log with developer tools in version 53.0.2785.143 chrome . The logs are saved with har format and have been attached. There are two log files ,which belong to different projects.
Created attachment 1239840 [details] jenkins-yasun-int.1ec1.dev-preview-int.openshiftapps.com.har
Created attachment 1239841 [details] jenkins-jjj.1ec1.dev-preview-int.openshiftapps.com.har
Took a peek at the attachment from #Comment 15 and the URL that came back from openshift oauth was indicative of an error on the oauth provider running in openshift master: "https://jenkins-yasun-int.1ec1.dev-preview-int.openshiftapps.com/securityRealm/finishLogin?error=server_error&error_description=The+authorization+server+encountered+an+unexpected+condition+that+prevented+it+from+fulfilling+the+request."
Same looking URL for the attachment from #Comment 16
The master's log will be more useful here as that error message serves as a general mask for a variety of different internal oauth errors.
The latest problem appears be the same problem reported in bug 1413863.
Bug 1413863 has been VERIFIED, so the issue in comment 28 should be resolved.
Moving to QA based on comment above.
Jenkins app can be accessed normally using image: sha256:66eea68ebc701a13527a4172feb2c4c0c9acda09efaa59821393498121d97ae0 Move to verified, thanks!