Description of problem: Setup a multi node env, small node in small district (broker), medium node in medium district (node), c9 node in c9 district (node). Tried to create application on medium node and c9 node, all met "Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity.. Reference ID: fa1265f8c9dae0249c8882062c71cbdf" error. It is ok when creating app uisng rhc client, this issue only exists on website. Version-Release number of selected component (if applicable): devenv_3677 How reproducible: always Steps to Reproduce: 1.Setup multi node env with small,medium,c9 node 2.Try to create application on meidum and c9 node 3. Actual results: Met "Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity.. Reference ID: 10c3fd88622b9e04ecc0e336723b3b3e" error. site log is as following: 2013-08-20 05:48:38.716 [DEBUG] Server error: : #0: Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity.. Reference ID: 10c3fd88622b9e04ecc0e336723b3b3e (pid:2749) 2013-08-20 05:48:38.717 [DEBUG] Found errors on the response object: {:base=>["Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity..\nReference ID: 10c3fd88622b9e04ecc0e336723b3b3e"]} (pid:2749) 2013-08-20 05:48:38.718 [DEBUG] #<ActiveResource::Errors:0x000000057afb20 @base=#<Application:0x0000000578c620 @as=#<WebUser::Mock:0x000000055fd818 @rhlogin="aaaaaaa", @errors=#<ActiveModel::Errors:0x000000055fd070 @base=#<WebUser::Mock:0x000000055fd818 ...>, @messages={}>, @ticket="YWFhYWFhYQ==", @streamline_type=:full, @api_ticket="8fc10f51fafd04ed2df1af4dd916a78664ee1158b83515c1e347f3a80c228319">, @attributes={"cartridges"=>["php-5.3"], "name"=>"asdf", "initial_git_url"=>"", "gear_profile"=>"medium", "scale"=>"false"}, @prefix_options={:domain_id=>"rtyrty3534"}, @persisted=false, @messages=nil, @errors=#<ActiveResource::Errors:0x000000057afb20 ...>, @changed_attributes={"name"=>nil}, @remote_errors=#<ActiveResource::ServerError: Failed. Response code = 500. Response message = .>, @validation_context=nil, @connection=#<RestApi::UserAwareConnection:0x000000057af4e0 @password=nil, @user=nil, @uri_parser=#<URI::Parser:0x000000057af4b8>, @site=#<URI::HTTPS:0x00000003a7a050 URL:https://localhost/broker/rest>, @format=#<RestApi::OpenshiftJsonFormat:0x000000032859f8 @root_attrs=[]>, @connection=#<ActiveResource::PersistentConnection:0x00000004ddbcc0 @password=nil, @user=nil, @uri_parser=#<URI::Parser:0x00000004ddbc98>, @site=#<URI::HTTPS:0x00000003a7a050 URL:https://localhost/broker/rest>, @format=#<RestApi::OpenshiftJsonFormat:0x000000032859f8 @root_attrs=[]>, @idle_timeout=4, @read_timeout=240, @open_timeout=3, @connection_name="rest_api", @default_header={}, @http=#<Net::HTTP::Persistent:0x000000054d49c8 @name="rest_api", @debug_output=nil, @proxy_uri=nil, @headers={}, @override_headers={}, @http_versions={"localhost:443"=>"1.1"}, @keep_alive=30, @open_timeout=3, @read_timeout=240, @idle_timeout=4, @socket_options=[[6, 1, 1]], @generation_key=:net_http_persistent_rest_api_generations, @ssl_generation_key=:net_http_persistent_rest_api_ssl_generations, @request_key=:net_http_persistent_rest_api_requests, @timeout_key=:net_http_persistent_rest_api_timeouts, @certificate=nil, @ca_file=nil, @private_key=nil, @ssl_version=nil, @verify_callback=nil, @verify_mode=0, @cert_store=nil, @generation=0, @ssl_generation=1, @reuse_ssl_sessions=true, @retry_change_requests=false>>, @as=#<WebUser::Mock:0x000000055fd818 @rhlogin="aaaaaaa", @errors=#<ActiveModel::Errors:0x000000055fd070 @base=#<WebUser::Mock:0x000000055fd818 ...>, @messages={}>, @ticket="YWFhYWFhYQ==", @streamline_type=:full, @api_ticket="8fc10f51fafd04ed2df1af4dd916a78664ee1158b83515c1e347f3a80c228319">, @default_header={}>>, @messages={:base=>["Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity..\nReference ID: 10c3fd88622b9e04ecc0e336723b3b3e"], :domain_id=>[], :scale=>[]}, @codes={:base=>[0]}> (pid:2749) 2013-08-20 05:48:38.727 [INFO ] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.13.2/app/views/applications/_name.html.haml (0.4ms) (pid:2749) 2013-08-20 05:48:38.738 [INFO ] Rendered application_types/_application_type_notifications.html.haml (0.1ms) (pid:2749) 2013-08-20 05:48:38.741 [INFO ] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.13.2/app/views/application_types/show.html.haml within layouts/console (21.8ms) (pid:2749) 2013-08-20 05:48:38.743 [INFO ] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.13.2/app/views/layouts/_head.html.haml (1.4ms) (pid:2749) 2013-08-20 05:48:38.745 [INFO ] Rendered layouts/console/_stylesheets.html.haml (1.4ms) (pid:2749) 2013-08-20 05:48:38.750 [INFO ] Rendered layouts/console/_header.html.haml (4.1ms) (pid:2749) 2013-08-20 05:48:38.751 [INFO ] Rendered layouts/_footer.html.haml (0.3ms) (pid:2749) 2013-08-20 05:48:38.752 [INFO ] Rendered layouts/console/_javascripts.html.haml (0.5ms) (pid:2749) 2013-08-20 05:48:38.752 [INFO ] Completed 200 OK in 2807ms (Views: 34.1ms | OpenShift API: 2742.5ms) (pid:2749) 2013-08-20 05:48:39.127 [INFO ] Started GET "/app/status/status.js?id=outage" for 66.187.233.202 at 2013-08-20 05:48:39 -0400 (pid:2749) 2013-08-20 05:48:39.187 [DEBUG] Cookies are disabled for this request by StatusApp::IgnoredCookies (pid:2749) Expected results: Should be able to create application on medium and c9 node from website. Additional info:
oo-stats result: [root@ip-10-145-185-23 ~]# oo-stats --------------------- Profile 'c9' summary: --------------------- District count : 1 District capacity : 6000 Dist avail capacity : 5999 Dist avail uids : 5999 Lowest dist usage pct : 0.01666666666666572 Highest dist usage pct : 0.01666666666666572 Avg dist usage pct : 0.01666666666666572 Nodes count : 1 Nodes active : 1 Gears total count : 1 Gears active count : 1 Available active gears : 3071 Effective available gears : 3071 Districts: Name Nodes DistAvailCapacity GearsActive EffectiveAvailGears AvgActiveUsagePct ----- ----- ----------------- ----------- ------------------- -------------------- dist3 1 5999 1 3071 0.032552083333333336 ------------------------- Profile 'medium' summary: ------------------------- District count : 1 District capacity : 6000 Dist avail capacity : 5998 Dist avail uids : 5998 Lowest dist usage pct : 0.03333333333333144 Highest dist usage pct : 0.03333333333333144 Avg dist usage pct : 0.03333333333333144 Nodes count : 1 Nodes active : 1 Gears total count : 2 Gears active count : 2 Available active gears : 38 Effective available gears : 38 Districts: Name Nodes DistAvailCapacity GearsActive EffectiveAvailGears AvgActiveUsagePct ----- ----- ----------------- ----------- ------------------- ----------------- dist2 1 5998 2 38 5.0 ------------------------ Profile 'small' summary: ------------------------ District count : 1 District capacity : 6000 Dist avail capacity : 6000 Dist avail uids : 6000 Lowest dist usage pct : 0.0 Highest dist usage pct : 0.0 Avg dist usage pct : 0.0 Nodes count : 1 Nodes active : 1 Gears total count : 0 Gears active count : 0 Available active gears : 90 Effective available gears : 90 Districts: Name Nodes DistAvailCapacity GearsActive EffectiveAvailGears AvgActiveUsagePct ----- ----- ----------------- ----------- ------------------- ----------------- dist1 1 6000 0 90 0.0 ------------------------ Summary for all systems: ------------------------ Districts : 3 Nodes : 3 Profiles : 3
Not sure what this has to do with UI - need more debugging info about the scenario. Can you provide the broker log near the tsack trace (including the broker production log dump of the create application request).
Will need the broker development.log output from around this error.
Hi, Abhishek Eventually it's not a bug, sorry for having wasted your time. With normal steps on devenv-stage_448, we can not recreate the bug as well. And then I doubt yujzhang(PTO today) is trying to creating applications from the web console of the medium or c9 nodes but not from the broker, then the error is recreated: Unable to complete the requested operation due to: uid could not be reserved in target district '521574e34eae18433d000001'. Please ensure the target district has available capacity.. Reference ID: 921da8cd25e45d352e41ba318de7351e But don't think it's a valid scenario, so going to close it.
The broker service was running on the node and using the console on that machine to create an application results in this error. The broker is connected to the local node and local mongo. The district_uuid is fetched from the node using facter and an attempt is made to reserve the UID within this district. The issue is that the query is made on the local mongo which is not expected to have any district data at all since this machine is the node and not the "broker".