Bug 1029828

Summary: Jenkins server does not work after moved
Product: OpenShift Online Reporter: Meng Bo <bmeng>
Component: PodAssignee: Dan McPherson <dmcphers>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: dmcphers
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-24 03:29:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meng Bo 2013-11-13 10:19:15 UTC
Description of problem:
Setup multi-node env and create district on it, add two node to the district.
Create jenkins server in the district. 
After move the jenkins server from one node to the other, build app with jenkins will always fail. For both existed app with jenkins added before move and new created app after move.

Version-Release number of selected component (if applicable):
devenv_4027

How reproducible:
always

Steps to Reproduce:
1.Setup multi-node env with district on it
2.Create jenkins server and create app with jenkins added
3.Move the jenkins server to other node
#oo-admin-move --gear_uuid 5282eb7f51b0a03198000001 -i ip-10-151-71-11
4.Try to build the existed app with jenkins
5.Create new app with jenkins added and build

Actual results:
Jenkins build will always fail for step4 and step5.

Expected results:
Should be able to build with jenkins after move.

Additional info:
The broker_host has been set to the broker IP in node.conf.
PUBLIC_HOSTNAME=ec2-107-22-83-54.compute-1.amazonaws.com
PUBLIC_IP=107.22.83.54
BROKER_HOST=10.151.120.101

Jenkins log for the failure.

Nov 13, 2013 4:51:47 AM hudson.plugins.openshift.OpenShiftCloud provision
INFO: Provisioning new node for workload = 2 and label = app5-build
Nov 13, 2013 4:51:47 AM hudson.plugins.openshift.OpenShiftCloud getOpenShiftConnection
INFO: Initiating Java Client Service - Configured for OpenShift Server https://localhost
Nov 13, 2013 4:51:47 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/api
Nov 13, 2013 4:51:47 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/user
Nov 13, 2013 4:51:47 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user". Will retry 4 more times before canceling build.
Nov 13, 2013 4:51:52 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/user
Nov 13, 2013 4:51:52 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user". Will retry 3 more times before canceling build.
Nov 13, 2013 4:51:57 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/user
Nov 13, 2013 4:51:57 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user". Will retry 2 more times before canceling build.
Nov 13, 2013 4:52:02 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/user
Nov 13, 2013 4:52:02 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user". Will retry 1 more times before canceling build.
Nov 13, 2013 4:52:07 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/user
Nov 13, 2013 4:52:07 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user". Will retry 0 more times before canceling build.
Nov 13, 2013 4:52:12 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Cancelling build due to earlier exceptions
com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user"
	at com.openshift.internal.client.RestService.request(RestService.java:132)
	at com.openshift.internal.client.RestService.request(RestService.java:109)
	at com.openshift.internal.client.RestService.request(RestService.java:92)
	at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:139)
	at com.openshift.internal.client.APIResource$GetUserRequest.execute(APIResource.java:245)
	at com.openshift.internal.client.APIResource.getUser(APIResource.java:100)
	at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:863)
	at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:440)
	at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:402)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:264)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:347)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: com.openshift.internal.client.httpclient.NotFoundException: {"api_version":1.2,"data":null,"messages":[{"exit_code":101,"field":null,"index":null,"severity":"error","text":"Application '52834a6a51b0a0d7f1000339' not found."}],"status":"not_found","supported_api_versions":[1.0,1.1,1.2,1.3,1.4,1.5,1.6],"type":null,"version":"1.2"}

	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:159)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:134)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:114)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.get(UrlConnectionHttpClient.java:84)
	at com.openshift.internal.client.RestService.request(RestService.java:175)
	at com.openshift.internal.client.RestService.request(RestService.java:123)
	... 14 more
Caused by: java.io.FileNotFoundException: https://localhost/broker/rest/user
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:130)
	... 18 more
Nov 13, 2013 4:52:12 AM hudson.plugins.openshift.OpenShiftCloud cancelItem
INFO: Cancelling Item 
Nov 13, 2013 4:52:12 AM hudson.plugins.openshift.OpenShiftCloud cancelItem
WARNING: Build app5-build app5bldr has been canceled

Comment 1 Dan McPherson 2013-11-13 16:11:18 UTC
The issue here is the java client looks for /etc/openshift/express.conf for it's preferred settings.  If you move that file or change localhost in it the problem will not occur.  I have fixed the devenv steps to create a multinode devenv with this step:

https://github.com/openshift/li/pull/2148

Comment 3 Meng Bo 2013-11-14 06:09:32 UTC
Tried with the latest devenv script.

The jenkins build will be success after moved the jenkins server.

Nov 14, 2013 1:06:24 AM hudson.plugins.openshift.OpenShiftCloud getOpenShiftConnection
INFO: Initiating Java Client Service - Configured for OpenShift Server https://10.147.162.122
Nov 14, 2013 1:06:24 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://10.147.162.122/broker/rest/api
Nov 14, 2013 1:06:25 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://10.147.162.122/broker/rest/user
Nov 14, 2013 1:06:25 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://10.147.162.122/broker/rest/domains
Nov 14, 2013 1:06:25 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://10.147.162.122/broker/rest/domain/bmengdev/applications?include=cartridges
Nov 14, 2013 1:06:25 AM hudson.plugins.openshift.OpenShiftCloud hasCapacity
INFO: Checking capacity
Nov 14, 2013 1:06:25 AM hudson.plugins.openshift.OpenShiftCloud hasCapacity
INFO: User has consumed 2 of 50 gears.

Move bug to verified.