Description of problem: It's failed to create medium/large gear size app due to "No district nodes available" on INT. Version-Release number of selected component (if applicable): INT(devenv_5018) How reproducible: always Steps to Reproduce: 1. Create a medium/large app rhc app create cdiy diy-0.1 -g medium --no-git Actual results: Application Options ------------------- Domain: intm Cartridges: diy-0.1 Gear Size: medium Scaling: no Creating application 'cdiy' ... Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: e792cb80b23b6c8aa54da1c19c984511 Expected results: Should create medium/large gear size app on INT successfully. Additional info:
Failed to jenkins build if jenkins server is of gear size medium/large due to "No district nodes available. Please try again and contact support if the issue persists. Reference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4" Following is jenkins.log : Jul 29, 2014 12:43:27 AM hudson.plugins.openshift.OpenShiftCloud provision INFO: Provisioning new node for workload = 2 and label = jbossas71s-build in domain lxiamigrate Jul 29, 2014 12:43:27 AM hudson.plugins.openshift.OpenShiftCloud getOpenShiftConnection INFO: Initiating Java Client Service - Configured for OpenShift Server https://int.openshift.redhat.com Jul 29, 2014 12:43:27 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/api Jul 29, 2014 12:43:28 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/user Jul 29, 2014 12:43:28 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domains Jul 29, 2014 12:43:28 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges Jul 29, 2014 12:43:32 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: a1e2e22ae893af7ea2892becde4dd3c0 . Will retry 4 more times before canceling build. Jul 29, 2014 12:43:37 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges Jul 29, 2014 12:43:42 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: 85b3322347b00623788da155c13917f1 . Will retry 3 more times before canceling build. Jul 29, 2014 12:43:47 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges Jul 29, 2014 12:43:51 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: 7834c4ea570c995a21e0757b40ba5cba . Will retry 2 more times before canceling build. Jul 29, 2014 12:43:56 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges Jul 29, 2014 12:44:01 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: 1719cb1b18b4b25a8adaf208efb77b96 . Will retry 1 more times before canceling build. Jul 29, 2014 12:44:06 AM com.openshift.internal.client.RestService request INFO: Requesting GET with protocol 1.2 on https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges Jul 29, 2014 12:44:11 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Caught com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4 . Will retry 0 more times before canceling build. Jul 29, 2014 12:44:16 AM hudson.plugins.openshift.OpenShiftCloud provision WARNING: Cancelling build due to earlier exceptions com.openshift.client.OpenShiftEndpointException: Could not request https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges: Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. Reference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4 at com.openshift.internal.client.RestService.request(RestService.java:136) at com.openshift.internal.client.RestService.request(RestService.java:103) at com.openshift.internal.client.AbstractOpenShiftResource$ServiceRequest.execute(AbstractOpenShiftResource.java:155) at com.openshift.internal.client.DomainResource$ListApplicationsRequest.execute(DomainResource.java:446) at com.openshift.internal.client.DomainResource.loadApplications(DomainResource.java:279) at com.openshift.internal.client.DomainResource.getOrLoadApplications(DomainResource.java:266) at com.openshift.internal.client.DomainResource.getApplications(DomainResource.java:261) at hudson.plugins.openshift.OpenShiftCloud.getSlaves(OpenShiftCloud.java:877) at hudson.plugins.openshift.OpenShiftCloud.provisionSlave(OpenShiftCloud.java:451) at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:413) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.openshift.internal.client.httpclient.InternalServerErrorException: {"api_version":1.2,"data":null,"messages":[{"exit_code":140,"field":null,"index":null,"severity":"error","text":"Unable to complete the requested operation due to: No district nodes available. Please try again and contact support if the issue persists. \nReference ID: b8de9f5ae5f213e20e2b0dc4d0890ac4"}],"status":"internal_server_error","supported_api_versions":[1.0,1.1,1.2,1.3,1.4,1.5,1.6,1.7],"type":null,"version":"1.2"} at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.createException(UrlConnectionHttpClient.java:167) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:148) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:123) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.get(UrlConnectionHttpClient.java:82) at com.openshift.internal.client.RestService.request(RestService.java:175) at com.openshift.internal.client.RestService.request(RestService.java:123) ... 20 more Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: https://int.openshift.redhat.com/broker/rest/domain/lxiamigrate/applications?include=cartridges at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254) at com.openshift.internal.client.httpclient.UrlConnectionHttpClient.request(UrlConnectionHttpClient.java:144) ... 24 more Jul 29, 2014 12:44:16 AM hudson.plugins.openshift.OpenShiftCloud cancelItem INFO: Cancelling Item Jul 29, 2014 12:44:16 AM hudson.plugins.openshift.OpenShiftCloud cancelItem WARNING: Build jbossas71s-build jbossas71sbldr has been canceled
Checked on INT(devenv_5024), this issue is fixed, could you please help to change status to ON_QA, then i will verify it, and please refer to the following results: 1) Create a medium gear size app [chunchen@F17-CCY test]$ rhc app create cdiy diy-0.1 -g medium --no-git Application Options ------------------- Domain: intm Cartridges: diy-0.1 Gear Size: medium Scaling: no Creating application 'cdiy' ... done Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on OpenShift. Waiting for your DNS name to be available ... done Your application 'cdiy' is now available. URL: http://cdiy-intm.int.rhcloud.com/ SSH to: 53d8514a03ef6425d2000154.rhcloud.com Git remote: ssh://53d8514a03ef6425d2000154.rhcloud.com/~/git/cdiy.git/ Run 'rhc show-app cdiy' for more details about your app. 2) Create a large gear size app [chunchen@F17-CCY test]$ rhc app create cdiy2 diy-0.1 -g large --no-git Application Options ------------------- Domain: intm Cartridges: diy-0.1 Gear Size: large Scaling: no Creating application 'cdiy2' ... done Disclaimer: This is an experimental cartridge that provides a way to try unsupported languages, frameworks, and middleware on OpenShift. Waiting for your DNS name to be available ... done Your application 'cdiy2' is now available. URL: http://cdiy2-intm.int.rhcloud.com/ SSH to: 53d8516f03ef64e23300066b.rhcloud.com Git remote: ssh://53d8516f03ef64e23300066b.rhcloud.com/~/git/cdiy2.git/ Run 'rhc show-app cdiy2' for more details about your app. 3) Show medium/large gear size app [chunchen@F17-CCY test]$ rhc app show cdiy2 cdiy2 @ http://cdiy2-intm.int.rhcloud.com/ (uuid: 53d8516f03ef64e23300066b) --------------------------------------------------------------------------- Domain: intm Created: 9:59 AM Gears: 1 (defaults to large) Git URL: ssh://53d8516f03ef64e23300066b.rhcloud.com/~/git/cdiy2.git/ SSH: 53d8516f03ef64e23300066b.rhcloud.com Deployment: auto (on git push) diy-0.1 (Do-It-Yourself 0.1) ---------------------------- Gears: 1 large
On QA as per comment#3
According to comment#3, mark it as VERIFIED.