Bug 773202

Summary: Jenkins build hang there after move jenkins app form one node to another node.
Product: OKD Reporter: Johnny Liu <jialiu>
Component: ocAssignee: Bill DeCoste <wdecoste>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 1.xCC: dmcphers
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-01-16 21:11:44 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Johnny Liu 2012-01-11 09:12:20 UTC
Description of problem:
Jenkins build hang there after move jenkins app form one node to another node.

Version-Release number of selected component (if applicable):
rhc-cartridge-jenkins-1.4-0.84.5-1.el6_2.noarch
jenkins-plugin-openshift-0.4.16-1.el6_2.x86_64
rhc-cartridge-jenkins-client-1.4-0.18.4-1.el6_2.noarch


How reproducible:
Always

Steps to Reproduce:
1. Setup a mult node test env.
2. Create a jenkins app and a php app individually.
3. Log into libra server, move the jenkins app to another node using rhc-admin-move command
4. Embed jenkins client to this php app.
5. Do some change in php app git repo, and run git push to trigger jenkins build.
  
Actual results:
Jenkins job hang there for ever.

Expected results:
After move jenkins app from one node to another, jenkins build should be working fine.

Additional info:
After move the jenkins app back to the previous node, jenkins build job is finished successfully (by checking this job's status in jenkins app's web UI).

Comment 1 Dan McPherson 2012-01-11 18:11:52 UTC
Jenkins will only work on one of the nodes in a two node devenv setup by default.  You'll need to change the broker system in the jenkins config (brokerHost in config.xml or Jenkins UI -> Manage Jenkins -> Configure System -> Cloud -> OpenShift Broker Hostname to point to the ip of your broker system (will be 1 of the two devenvs).


You can also fix this ahead of time before creating jenkins by setting libra_server in node.conf to point to you broker system.  I've fixed this in the multinode setup doc to indicate this.

Comment 2 Dan McPherson 2012-01-12 06:56:34 UTC
Hit another issue...  it seems the java client doesn't allow for self signed certs like the jenkins plugin used to:

Jan 12, 2012 1:42:11 AM hudson.plugins.openshift.OpenShiftCloud provision
SEVERE: Exception caught trying to provision
java.io.IOException: com.openshift.express.client.OpenShiftEndpointException: Could not get user info for user "{0}" at "{1}"
	at hudson.plugins.openshift.OpenShiftCloud.getUserInfo(OpenShiftCloud.java:349)
	at hudson.plugins.openshift.OpenShiftCloud.shouldProvision(OpenShiftCloud.java:242)
	at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:437)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:203)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:54)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:286)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:534)
	at java.util.TimerThread.run(Timer.java:484)
Caused by: com.openshift.express.client.OpenShiftEndpointException: Could not get user info for user "{0}" at "{1}"
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:366)
	at com.openshift.express.client.OpenShiftService.getUserInfo(OpenShiftService.java:115)
	at hudson.plugins.openshift.OpenShiftCloud.getUserInfo(OpenShiftCloud.java:344)
	... 8 more
Caused by: com.openshift.express.internal.client.httpclient.HttpClientException: java.io.IOException: HTTPS hostname wrong:  should be <10.123.101.31>
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:137)
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.post(UrlConnectionHttpClient.java:81)
	at com.openshift.express.client.OpenShiftService.sendRequest(OpenShiftService.java:357)
	... 10 more
Caused by: java.io.IOException: HTTPS hostname wrong:  should be <10.123.101.31>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:529)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:454)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1139)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:397)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
	at com.openshift.express.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:124)


We got this even with ignoreBrokerCertCheck=true



Note to recreate simply change brokerHost to the internal ip address of the server or set libra_server to the internal ip address in node.conf before creating jenkins.

Comment 3 Bill DeCoste 2012-01-12 13:46:07 UTC
I'm looking at the first issue and am unable to reproduce the 2nd issue. 

On issue #2, that is not the self-signed certificate exception. This error means that the hostname in the server certificate does not match the hostname that is in the URL. I created a php app and a jenkins app and pointed jenkins brokerHost to the local IP (10.198.49.220) of a new devenv (ami-93ed3afa). The php app built without error.

Comment 4 Dan McPherson 2012-01-12 14:53:11 UTC
Hi Bill,

   It's the error we got by pointing the broker to the internal ip.  Note that it was with the previous stage build that we got this error (Johnny was getting it working before migrating and moving).  Is it possible this changed since the last release?

  I'll give it a shot not and see if I can recreate on the latest.

  I am not sure what you mean by issue 1 vs 2.  The only known issue at this point is the exception from comment 2.


Johnny,

  Did you verify if it was still broken after the upgrade talking to the internal ip?

Comment 5 Bill DeCoste 2012-01-12 15:26:54 UTC
OK. I tested the issue in comment #2 and was not able to recreate when pointing brokerHost to the internal IP, but I will  keep trying.

Comment 6 Bill DeCoste 2012-01-12 15:39:36 UTC
Nothing has changed in several sprints except the conversion from url to hostname/port

Comment 7 Dan McPherson 2012-01-12 16:29:05 UTC
Bill and I have both tried with the latest builds using using internal ip and it seems to work fine.

I retried with the stage 110 build to use internal ip and it failed for me as well.  It appears to be an issue that was fixed.  So this should work to upgrade everything and then build.

Comment 8 Johnny Liu 2012-01-13 06:38:37 UTC
Verified this bug with latest build (devenv_215) with set libra_server to internal ip, and PASSED.