Bug 1012303 - [fork][origin_runtime_208] Jenkins builds always fail with "NotFoundOpenShiftException" for both scaling and non-scaling apps in multi-node env
[fork][origin_runtime_208] Jenkins builds always fail with "NotFoundOpenShift...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Mrunal Patel
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-26 04:59 EDT by Zhe Wang
Modified: 2015-05-14 19:29 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-10-17 09:31:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Zhe Wang 2013-09-26 04:59:59 EDT
Description of problem:
Given a multi-node env, for both scaling and non-scaling apps, the jenkins builds always fail with NotFoundOpenShiftException. On the other hand, there is no such problem when testing against a single node env of fork_ami_208_864.

This could be caused by that the jenkins plugin referrs to "localhost" while the jenkins server could locate in different nodes from the broker.

Version-Release number of selected component (if applicable):
fork_ami_208_864 (multi-node env)

How reproducible:
always

Steps to Reproduce:
1. create an app with jenkins-client added
2. trigger a Jenkins build
3. launch an instance (single node) and test against it with Step 1 ~ 2.

Actual results:
In step 2, the build failed with the following errors in the jenkins log:

Sep 26, 2013 3:32:56 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://localhost/broker/rest/user
Sep 26, 2013 3:32:56 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user". Will retry 0 more times before canceling build.
com.openshift.client.NotFoundOpenShiftException: Could not find any OpenShift resource at "https://localhost/broker/rest/user"
	at com.openshift.internal.client.RestService.request(RestService.java:128)
	at com.openshift.internal.client.RestService.request(RestService.java:111)
	at com.openshift.internal.client.RestService.request(RestService.java:93)
	at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:128)
	at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:123)
	at com.openshift.internal.client.APIResource$GetUserRequest.execute(APIResource.java:243)
	at com.openshift.internal.client.APIResource.getUser(APIResource.java:98)
	at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:859)
	at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:436)
	at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:401)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:264)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:347)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
Caused by: com.openshift.internal.client.httpclient.NotFoundException: {"api_version":1.2,"data":null,"messages":[{"exit_code":101,"field":null,"severity":"error","text":"Application '5243df223e44f6d65e000710' not found."}],"status":"not_found","supported_api_versions":[1.0,1.1,1.2,1.3,1.4,1.5,1.6],"type":null,"version":"1.2"}

	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:233)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.write(UrlConnectionHttpClient.java:207)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.get(UrlConnectionHttpClient.java:115)
	at com.openshift.internal.client.RestService.request(RestService.java:167)
	at com.openshift.internal.client.RestService.request(RestService.java:122)
	... 15 more
Caused by: java.io.FileNotFoundException: https://localhost/broker/rest/user
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1624)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
	at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.write(UrlConnectionHttpClient.java:203)
	... 18 more
Sep 26, 2013 3:33:01 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Cancelling build due to earlier exceptions
Sep 26, 2013 3:33:01 AM hudson.plugins.openshift.OpenShiftCloud cancelItem
INFO: Cancelling Item 
Sep 26, 2013 3:33:02 AM hudson.plugins.openshift.OpenShiftCloud cancelItem
WARNING: Build py27-build py27bldr has been canceled

On the other hand, the jenkins build in the single node instance succeeded.


Expected results:
The jenkins build should be successful in multi-node env.

Additional info:
Comment 1 Mrunal Patel 2013-09-26 15:31:51 EDT
This is a general issue with /etc/openshift/node/env/OPENSHIFT_BROKER_HOST not set correctly when adding a node to a multi node setup.

https://github.com/openshift/li/pull/1929
Comment 2 Zhe Wang 2013-09-27 03:38:24 EDT
The jenkins build succeeded after exporting the OPENSHIFT_BROKER_HOST env in nodes, and here is the corresponding jenkins log:

INFO: Requesting POST with protocol 1.2 on https://10.204.62.44/broker/rest/domains/dev3837tst/applications
Sep 27, 2013 3:33:49 AM hudson.plugins.openshift.OpenShiftSlave createApp
INFO: Stopping application on builder gear ...
Sep 27, 2013 3:33:49 AM com.openshift.internal.client.RestService request
INFO: Requesting POST with protocol 1.2 on https://10.204.62.44/broker/rest/domains/dev3837tst/applications/py27bldr/events
Sep 27, 2013 3:33:51 AM hudson.plugins.openshift.OpenShiftSlave connect
INFO: Connecting to slave py27bldr...
Sep 27, 2013 3:33:51 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://10.204.62.44/broker/rest/domains/dev3837tst/applications/py27bldr/gear_groups

Move the bug to VERIFIED.

Note You need to log in before you can comment on or make changes to this bug.