Bug 1004686

Summary: The count of nodes for every district will be inconsistent in suggestions
Product: OpenShift Online Reporter: zhaozhanqi <zzhao>
Component: WebsiteAssignee: Luke Meyer <lmeyer>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 2.xCC: xtian, yujzhang
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-17 13:28:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description zhaozhanqi 2013-09-05 08:47:48 UTC 
Description of problem:
create one district (dist1) and add two nodes to dist1. if node_needs is greater than suggest_node_district_per* district_exist, will give suggestions to add district. The following is the suggestions:
<--
Create 11 new districts.

oo-admin-ctl-district -c create -p 'small' -n <name>

Then add 1 node to each new district.

oo-admin-ctl-district -c add-node -n '<district_name>' -i <server_identity>

Add 1 node to district dist1, which according to an active gear percent of 5% has room for 1 more node.

oo-admin-ctl-district -c add-node -n 'dist1' -i <server_identity>

-->

The dist1 will have 3 nodes(exist+add), and other districts only have 1 node.

Version-Release number of selected component (if applicable):
fork_ami_origin_ui_81_821

How reproducible:
always

Steps to Reproduce:
1. create one district and add two nodes to it
2. set 'GEAR_EXPECTED_ACTIVE_PERCENT=5' and 'GEAR_UP_THRESHOLD=1000, small: 1000, large: 750'
3. login on admin-console --suggestion

Actual results:

Add capacity for small gear profile
Gear profile small has capacity for 0 active gears, which is below the threshold of 1000.

Create 12 nodes with profile small to increase the active gear capacity. This suggestion is based on:

1000 gears needed to both reach the threshold and meet the configured GEAR_UP_SIZE of this profile.
The estimated maximum active gears of 90 for nodes in this profile.
Create 11 new districts.

oo-admin-ctl-district -c create -p 'small' -n <name>
Then add 1 node to each new district.
oo-admin-ctl-district -c add-node -n '<district_name>' -i <server_identity>
Add 1 node to district dist1, which according to an active gear percent of 5% has room for 1 more node.

oo-admin-ctl-district -c add-node -n 'dist1' -i <server_identity>

Expected results:

First three district,each distrcts consist of 3 nodes
the last one consist of 2 nodes

Additional info:
Comment 1 Luke Meyer 2013-09-05 21:31:09 UTC 
This is surprising, but currently as designed. I'm considering changing that design - it shouldn't be hard.

I'm glad you noticed, you're paying good attention. My thought was that we should discourage people from filling their districts up before they have a good idea of what their active percentage is. So, it suggests creating districts half-full, and tries to add nodes evenly to existing districts rather than filling them up.

In the case of 3-node districts, "half full" is one node. That's certainly a surprising result if you didn't write the logic like me.

As I've since lowered the default active percentage to 5, meaning by default suggesting fairly small districts, it would probably make more sense to stop suggesting creating them half-full. I'll do that if it's not too tough. Talking through this helped me decide.
Comment 2 openshift-github-bot 2013-10-16 19:11:48 UTC 
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/b781203ede25ec5130bb2bdc441193f3f4282a24
<capacity suggestions> bug 1004686

https://bugzilla.redhat.com/show_bug.cgi?id=1004686
Testers noted that the fill algorithm for putting new nodes in districts
was kind of weird. It was attempting to keep from filling districts
prematurely, but this was the wrong approach. What we do now is default
to a low active percentage setting that will result in a fairly small
district size target. Then, we suggest filling existing districts before
creating new districts as full as needed.
Comment 3 Yujie Zhang 2013-10-17 07:39:05 UTC 
Tested on devenv_3907, steps as following:
1. Created 1 district with 2 small nodes in the district.
2. Set 'GEAR_EXPECTED_ACTIVE_PERCENT=5' and 'GEAR_UP_THRESHOLD=1000, small: 1000, large: 750'
3. Login on admin-console --suggestion

The suggestion is as following:

 Add capacity for small gear profile

Gear profile small has capacity for 180 active gears, which is below the threshold of 1000.

Create 10 nodes with profile small to increase the active gear capacity. This suggestion is based on:

    820 gears needed to both reach the threshold and meet the configured GEAR_UP_SIZE of this profile.
    The estimated maximum active gears of 90 for nodes in this profile.

Add 1 node to district dist_small, which according to an active gear percent of 5% has room for 1 more node.

oo-admin-ctl-district -c add-node -n 'dist_small' -i <server_identity>

Create 3 new districts.

oo-admin-ctl-district -c create -p 'small' -n <name>

Then add 3 nodes to each new district.

oo-admin-ctl-district -c add-node -n '<district_name>' -i <server_identity>

The suggestion is as expected, so verify this bug, thanks!