Bug 1124122 - Failed to create medium/large gear size app due to "No district nodes available" on INT
Summary: Failed to create medium/large gear size app due to "No district nodes availab...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 2.x
Assignee: Rajat Chopra
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-29 02:47 UTC by chunchen
Modified: 2016-09-30 02:15 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-10 00:49:53 UTC


Attachments (Terms of Use)

Description chunchen 2014-07-29 02:47:40 UTC
Description of problem:
It's failed to create medium/large gear size app due to "No district nodes available" on INT.

Version-Release number of selected component (if applicable):
INT(devenv_5018)

How reproducible:
always

Steps to Reproduce:
1. Create a medium/large app
rhc app create cdiy diy-0.1 -g medium --no-git

Actual results:
Application Options
-------------------
Domain:     intm
Cartridges: diy-0.1
Gear Size:  medium
Scaling:    no

Creating application 'cdiy' ... 
Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: e792cb80b23b6c8aa54da1c19c984511

Expected results:
Should create medium/large gear size app on INT successfully.

Additional info:

Comment 1 Liang Xia 2014-07-29 04:59:39 UTC
Failed to jenkins build if jenkins server is of gear size medium/large due to "No district nodes available. Please try again and contact support if the issue persists. Reference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4"

Following is jenkins.log :
Jul 29, 2014 12:43:27 AM hudson.plugins.openshift.OpenShiftCloud provision
INFO: Provisioning new node for workload = 2 and label = jbossas71s-build in domain lxiamigrate
Jul 29, 2014 12:43:27 AM hudson.plugins.openshift.OpenShiftCloud getOpenShiftConnection
INFO: Initiating Java Client Service - Configured for OpenShift Server https://int.openshift.redhat.com
Jul 29, 2014 12:43:27 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/api
Jul 29, 2014 12:43:28 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/user
Jul 29, 2014 12:43:28 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domains
Jul 29, 2014 12:43:28 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges
Jul 29, 2014 12:43:32 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: a1e2e22ae893af7ea2892becde4dd3c0
. Will retry 4 more times before canceling build.
Jul 29, 2014 12:43:37 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges
Jul 29, 2014 12:43:42 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: 85b3322347b00623788da155c13917f1
. Will retry 3 more times before canceling build.
Jul 29, 2014 12:43:47 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges
Jul 29, 2014 12:43:51 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: 7834c4ea570c995a21e0757b40ba5cba
. Will retry 2 more times before canceling build.
Jul 29, 2014 12:43:56 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges
Jul 29, 2014 12:44:01 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: 1719cb1b18b4b25a8adaf208efb77b96
. Will retry 1 more times before canceling build.
Jul 29, 2014 12:44:06 AM com.openshift.internal.client.RestService request
INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges
Jul 29, 2014 12:44:11 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4
. Will retry 0 more times before canceling build.
Jul 29, 2014 12:44:16 AM hudson.plugins.openshift.OpenShiftCloud provision
WARNING: Cancelling build due to earlier exceptions
com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. 
Reference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4
        at com.openshift.internal.client.RestService.request(RestService.java:136)
        at com.openshift.internal.client.RestService.request(RestService.java:103)
        at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:155)
        at com.openshift.internal.client.DomainResource$ListApplicationsRequest.execute(DomainResource.java:446)
        at com.openshift.internal.client.DomainResource.loadApplications(DomainResource.java:279)
        at com.openshift.internal.client.DomainResource.getOrLoadApplications(DomainResource.java:266)
        at com.openshift.internal.client.DomainResource.getApplications(DomainResource.java:261)
        at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:877)
        at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:451)
        at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:413)
        at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
        at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
        at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368)
        at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.openshift.internal.client.httpclient.InternalServerErrorException: {"api_version":1.2,"data":null,"messages":[{"exit_code":140,"field":null,"index":null,"severity":"error","text":"Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. \nReference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4"}],"status":"internal_server_error","supported_api_versions":[1.0,1.1,1.2,1.3,1.4,1.5,1.6,1.7],"type":null,"version":"1.2"}
        at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:167)
        at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:148)
        at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:123)
        at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.get(UrlConnectionHttpClient.java:82)
        at com.openshift.internal.client.RestService.request(RestService.java:175)
        at com.openshift.internal.client.RestService.request(RestService.java:123)
        ... 20 more
Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
        at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:144)
        ... 24 more
Jul 29, 2014 12:44:16 AM hudson.plugins.openshift.OpenShiftCloud cancelItem
INFO: Cancelling Item 
Jul 29, 2014 12:44:16 AM hudson.plugins.openshift.OpenShiftCloud cancelItem
WARNING: Build jbossas71s-build jbossas71sbldr has been canceled

Comment 3 chunchen 2014-07-30 02:04:20 UTC
Checked on INT(devenv_5024), this issue is fixed, could you please help to change status to ON_QA, then i will verify it, and please refer to the following results:

1) Create a medium gear size app
[chunchen@F17-CCY test]$ rhc app create cdiy diy-0.1 -g medium --no-git
Application Options
-------------------
Domain:     intm
Cartridges: diy-0.1
Gear Size:  medium
Scaling:    no

Creating application 'cdiy' ... done

  Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on OpenShift.

Waiting for your DNS name to be available ... done

Your application 'cdiy' is now available.

  URL:        http://cdiy-intm.int.rhcloud.com/
  SSH to:     53d8514a03ef6425d2000154@cdiy-intm.int.rhcloud.com
  Git remote: ssh://53d8514a03ef6425d2000154@cdiy-intm.int.rhcloud.com/~/git/cdiy.git/

Run 'rhc show-app cdiy' for more details about your app.

2) Create a large gear size app
[chunchen@F17-CCY test]$ rhc app create cdiy2 diy-0.1 -g large --no-git
Application Options
-------------------
Domain:     intm
Cartridges: diy-0.1
Gear Size:  large
Scaling:    no

Creating application 'cdiy2' ... done

  Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on OpenShift.

Waiting for your DNS name to be available ... done

Your application 'cdiy2' is now available.

  URL:        http://cdiy2-intm.int.rhcloud.com/
  SSH to:     53d8516f03ef64e23300066b@cdiy2-intm.int.rhcloud.com
  Git remote: ssh://53d8516f03ef64e23300066b@cdiy2-intm.int.rhcloud.com/~/git/cdiy2.git/

Run 'rhc show-app cdiy2' for more details about your app.

3) Show medium/large gear size app
[chunchen@F17-CCY test]$ rhc app show cdiy2
cdiy2 @ http://cdiy2-intm.int.rhcloud.com/ (uuid: 53d8516f03ef64e23300066b)
---------------------------------------------------------------------------
  Domain:     intm
  Created:    9:59 AM
  Gears:      1 (defaults to large)
  Git URL:    ssh://53d8516f03ef64e23300066b@cdiy2-intm.int.rhcloud.com/~/git/cdiy2.git/
  SSH:        53d8516f03ef64e23300066b@cdiy2-intm.int.rhcloud.com
  Deployment: auto (on git push)

  diy-0.1 (Do-It-Yourself 0.1)
  ----------------------------
    Gears: 1 large

Comment 4 Rajat Chopra 2014-07-30 03:57:17 UTC
On QA as per comment#3

Comment 5 chunchen 2014-07-30 10:31:41 UTC
According to comment#3, mark it as VERIFIED.


Note You need to log in before you can comment on or make changes to this bug.