Bug 998902 - Can not create application on medium node in medium district and c9 node in c9 district from website.
Can not create application on medium node in medium district and c9 node in c...
Status: CLOSED NOTABUG
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-20 06:18 EDT by Yujie Zhang
Modified: 2015-05-14 20:20 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-08-21 22:58:41 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Yujie Zhang 2013-08-20 06:18:23 EDT
Description of problem:

Setup a multi node env, small node in small district (broker), medium node in medium district (node), c9 node in c9 district (node). Tried to create application on medium node and c9 node, all met "Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity.. Reference ID: fa1265f8c9dae0249c8882062c71cbdf" error.

It is ok when creating app uisng rhc client, this issue only exists on website.

Version-Release number of selected component (if applicable):
devenv_3677

How reproducible:
always

Steps to Reproduce:
1.Setup multi node env with small,medium,c9 node
2.Try to create application on meidum and c9 node
3.

Actual results:

Met "Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'. Please ensure the target district has available capacity.. Reference ID: 10c3fd88622b9e04ecc0e336723b3b3e" error.

site log is as following:

2013-08-20 05:48:38.716 [DEBUG] Server error: : #0: Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'.  Please ensure the target district has available capacity..
Reference ID: 10c3fd88622b9e04ecc0e336723b3b3e (pid:2749)
2013-08-20 05:48:38.717 [DEBUG] Found errors on the response object: {:base=>["Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'.  Please ensure the target district has available capacity..\nReference ID: 10c3fd88622b9e04ecc0e336723b3b3e"]} (pid:2749)
2013-08-20 05:48:38.718 [DEBUG] #<ActiveResource::Errors:0x000000057afb20 @base=#<Application:0x0000000578c620 @as=#<WebUser::Mock:0x000000055fd818 @rhlogin="aaaaaaa", @errors=#<ActiveModel::Errors:0x000000055fd070 @base=#<WebUser::Mock:0x000000055fd818 ...>, @messages={}>, @ticket="YWFhYWFhYQ==", @streamline_type=:full, @api_ticket="8fc10f51fafd04ed2df1af4dd916a78664ee1158b83515c1e347f3a80c228319">, @attributes={"cartridges"=>["php-5.3"], "name"=>"asdf", "initial_git_url"=>"", "gear_profile"=>"medium", "scale"=>"false"}, @prefix_options={:domain_id=>"rtyrty3534"}, @persisted=false, @messages=nil, @errors=#<ActiveResource::Errors:0x000000057afb20 ...>, @changed_attributes={"name"=>nil}, @remote_errors=#<ActiveResource::ServerError: Failed.  Response code = 500.  Response message = .>, @validation_context=nil, @connection=#<RestApi::UserAwareConnection:0x000000057af4e0 @password=nil, @user=nil, @uri_parser=#<URI::Parser:0x000000057af4b8>, @site=#<URI::HTTPS:0x00000003a7a050 URL:https://localhost/broker/rest>, @format=#<RestApi::OpenshiftJsonFormat:0x000000032859f8 @root_attrs=[]>, @connection=#<ActiveResource::PersistentConnection:0x00000004ddbcc0 @password=nil, @user=nil, @uri_parser=#<URI::Parser:0x00000004ddbc98>, @site=#<URI::HTTPS:0x00000003a7a050 URL:https://localhost/broker/rest>, @format=#<RestApi::OpenshiftJsonFormat:0x000000032859f8 @root_attrs=[]>, @idle_timeout=4, @read_timeout=240, @open_timeout=3, @connection_name="rest_api", @default_header={}, @http=#<Net::HTTP::Persistent:0x000000054d49c8 @name="rest_api", @debug_output=nil, @proxy_uri=nil, @headers={}, @override_headers={}, @http_versions={"localhost:443"=>"1.1"}, @keep_alive=30, @open_timeout=3, @read_timeout=240, @idle_timeout=4, @socket_options=[[6, 1, 1]], @generation_key=:net_http_persistent_rest_api_generations, @ssl_generation_key=:net_http_persistent_rest_api_ssl_generations, @request_key=:net_http_persistent_rest_api_requests, @timeout_key=:net_http_persistent_rest_api_timeouts, @certificate=nil, @ca_file=nil, @private_key=nil, @ssl_version=nil, @verify_callback=nil, @verify_mode=0, @cert_store=nil, @generation=0, @ssl_generation=1, @reuse_ssl_sessions=true, @retry_change_requests=false>>, @as=#<WebUser::Mock:0x000000055fd818 @rhlogin="aaaaaaa", @errors=#<ActiveModel::Errors:0x000000055fd070 @base=#<WebUser::Mock:0x000000055fd818 ...>, @messages={}>, @ticket="YWFhYWFhYQ==", @streamline_type=:full, @api_ticket="8fc10f51fafd04ed2df1af4dd916a78664ee1158b83515c1e347f3a80c228319">, @default_header={}>>, @messages={:base=>["Unable to complete the requested operation due to: uid could not be reserved in target district '781742592509949570449408'.  Please ensure the target district has available capacity..\nReference ID: 10c3fd88622b9e04ecc0e336723b3b3e"], :domain_id=>[], :scale=>[]}, @codes={:base=>[0]}> (pid:2749)
2013-08-20 05:48:38.727 [INFO ] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.13.2/app/views/applications/_name.html.haml (0.4ms) (pid:2749)
2013-08-20 05:48:38.738 [INFO ] Rendered application_types/_application_type_notifications.html.haml (0.1ms) (pid:2749)
2013-08-20 05:48:38.741 [INFO ] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.13.2/app/views/application_types/show.html.haml within layouts/console (21.8ms) (pid:2749)
2013-08-20 05:48:38.743 [INFO ] Rendered /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-console-1.13.2/app/views/layouts/_head.html.haml (1.4ms) (pid:2749)
2013-08-20 05:48:38.745 [INFO ] Rendered layouts/console/_stylesheets.html.haml (1.4ms) (pid:2749)
2013-08-20 05:48:38.750 [INFO ] Rendered layouts/console/_header.html.haml (4.1ms) (pid:2749)
2013-08-20 05:48:38.751 [INFO ] Rendered layouts/_footer.html.haml (0.3ms) (pid:2749)
2013-08-20 05:48:38.752 [INFO ] Rendered layouts/console/_javascripts.html.haml (0.5ms) (pid:2749)
2013-08-20 05:48:38.752 [INFO ] Completed 200 OK in 2807ms (Views: 34.1ms | OpenShift API: 2742.5ms) (pid:2749)
2013-08-20 05:48:39.127 [INFO ] Started GET "/app/status/status.js?id=outage" for 66.187.233.202 at 2013-08-20 05:48:39 -0400 (pid:2749)
2013-08-20 05:48:39.187 [DEBUG] Cookies are disabled for this request by StatusApp::IgnoredCookies (pid:2749)


Expected results:

Should be able to create application on medium and c9 node from website.

Additional info:
Comment 1 Yujie Zhang 2013-08-20 06:19:35 EDT
oo-stats result:

[root@ip-10-145-185-23 ~]# oo-stats 
---------------------
Profile 'c9' summary:
---------------------
            District count : 1
         District capacity : 6000
       Dist avail capacity : 5999
           Dist avail uids : 5999
     Lowest dist usage pct : 0.01666666666666572
    Highest dist usage pct : 0.01666666666666572
        Avg dist usage pct : 0.01666666666666572
               Nodes count : 1
              Nodes active : 1
         Gears total count : 1
        Gears active count : 1
    Available active gears : 3071
 Effective available gears : 3071
Districts:
 Name Nodes DistAvailCapacity GearsActive EffectiveAvailGears    AvgActiveUsagePct
----- ----- ----------------- ----------- ------------------- --------------------
dist3     1              5999           1                3071 0.032552083333333336
-------------------------
Profile 'medium' summary:
-------------------------
            District count : 1
         District capacity : 6000
       Dist avail capacity : 5998
           Dist avail uids : 5998
     Lowest dist usage pct : 0.03333333333333144
    Highest dist usage pct : 0.03333333333333144
        Avg dist usage pct : 0.03333333333333144
               Nodes count : 1
              Nodes active : 1
         Gears total count : 2
        Gears active count : 2
    Available active gears : 38
 Effective available gears : 38
Districts:
 Name Nodes DistAvailCapacity GearsActive EffectiveAvailGears AvgActiveUsagePct
----- ----- ----------------- ----------- ------------------- -----------------
dist2     1              5998           2                  38               5.0
------------------------
Profile 'small' summary:
------------------------
            District count : 1
         District capacity : 6000
       Dist avail capacity : 6000
           Dist avail uids : 6000
     Lowest dist usage pct : 0.0
    Highest dist usage pct : 0.0
        Avg dist usage pct : 0.0
               Nodes count : 1
              Nodes active : 1
         Gears total count : 0
        Gears active count : 0
    Available active gears : 90
 Effective available gears : 90
Districts:
 Name Nodes DistAvailCapacity GearsActive EffectiveAvailGears AvgActiveUsagePct
----- ----- ----------------- ----------- ------------------- -----------------
dist1     1              6000           0                  90               0.0
------------------------
Summary for all systems:
------------------------
 Districts : 3
     Nodes : 3
  Profiles : 3
Comment 2 Clayton Coleman 2013-08-21 13:26:44 EDT
Not sure what this has to do with UI - need more debugging info about the scenario.  Can you provide the broker log near the tsack trace (including the broker production log dump of the create application request).
Comment 3 Abhishek Gupta 2013-08-21 16:33:52 EDT
Will need the broker development.log output from around this error.
Comment 4 Xiaoli Tian 2013-08-21 22:58:41 EDT
Hi, Abhishek

Eventually it's not a  bug, sorry for having wasted your time.

With normal steps on devenv-stage_448, we can not recreate the bug as well.

And then I doubt yujzhang(PTO today) is trying to creating applications from the web console of the medium or c9 nodes but not from the broker,  then the error is recreated: 
Unable to complete the requested operation due to: uid could not be reserved in target district '521574e34eae18433d000001'. Please ensure the target district has available capacity.. Reference ID: 921da8cd25e45d352e41ba318de7351e

But don't think it's a valid scenario, so going to close it.
Comment 5 Abhishek Gupta 2013-08-21 23:30:47 EDT
The broker service was running on the node and using the console on that machine to create an application results in this error. The broker is connected to the local node and local mongo. 

The district_uuid is fetched from the node using facter and an attempt is made to reserve the UID within this district. The issue is that the query is made on the local mongo which is not expected to have any district data at all since this machine is the node and not the "broker".

Note You need to log in before you can comment on or make changes to this bug.