Bug 835176

Summary: min/max gear restriction should be checked at the mongo level
Product: OKD Reporter: Dan McPherson <dmcphers>
Component: PodAssignee: Ravi Sankar <rpenta>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: medium    
Version: 2.xCC: jhou, mmcgrath, rmillner, rpenta, xtian
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: devenv_2737 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-14 00:48:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan McPherson 2012-06-25 18:24:56 UTC
Description of problem:

min/max gear checking is currently done in application code.  This creates race conditions for concurrent scaleup/scaledown requests.  The check should be done at the mongo level.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Set min or max gears allowed for a cart
2. Concurrently scale up and scale down an app with min/max gears
3.
  
Actual results:
Can have more or less gears than allowed by min/max gear settings


Expected results:
Should work as expected regardless of concurrency


Additional info:

Comment 1 Rob Millner 2012-06-26 20:54:49 UTC
An important caveat is that if there's an error or exception from the save method due to Mongodb; then any steps that had been taken to create a gear must be un-done before the call completes or a half-built gear could be left behind.

Comment 2 John Poelstra 2012-07-09 16:11:05 UTC
fixed on friday, pull requesting coming today

Comment 3 Ravi Sankar 2012-07-10 16:52:23 UTC
Fixed in git rev 0a45d8f6f86ddfdd.
Testing must be done by firing multiple scale-up/scale-down events in parallel and make sure number of application gears is consistent with consumed gears.

Comment 4 Xiaoli Tian 2012-07-11 11:26:04 UTC
Tested this on devenv_1884, this bug should be partly fixed, test with the following steps:

1. Change min/max gears for php to 2 /3 
	Scaling:
	  Min: 2
	  Max: 3
2. Create a scaling php app, then it should have already consumed 3 gears (2 web cartridge gears and 1 haproxy gear)

3. Open 3 terminals to scale up at the same time:

   2 of them return success, 1 of them return:
"Failed to add event scale-up to application scalphpapp1 due to: Cannot scale up beyond maximum gear limit '3' in app scalphpapp1.","field":null}],

4. Check the apps's gear_groups, the max gear is 3.
data": [
    {
      "gears": [
        {
          "id": "eb7b4164dccc4f36887c2d3bfd5fcc55",
          "state": "started"
        }
      ],
      "cartridges": [
        {
          "name": "haproxy-1.4"
        },
        {
          "name": "php-5.3"
        }
      ],
      "name": "@@app/comp-proxy/cart-php-5.3",
      "gear_profile": "small"
    },
    {
      "gears": [
        {
          "id": "8c528a9e5036480ca986989faa505afe",
          "state": "started"
        },
        {
          "id": "d6ec9c2ad95d4ca4a8766cc4b9633e97",
          "state": "started"
        },
        {
          "id": "4075a090a7f04d18a957702ce28d5290",
          "state": "started"
        }
      ],
      "cartridges": [
        {
          "name": "php-5.3"
        }
      ],
      "name": "@@app/comp-web/cart-php-5.3",
      "gear_profile": "small"
    }
  ],


5. Check consumed gears:
	irb(main):004:0> user1.consumed_gears
	=> 5
	irb(main):005:0> user1.applications.length
	=> 1
	irb(main):006:0> user1.applications[0].group_instances[1].gears.length
	=> 1
	irb(main):007:0> user1.applications[0].group_instances[0].gears.length
	=> 3

Then the consumed_gears is larger than the actual gears, the consumed gears should be 4 as well.

Comment 5 Mike McGrath 2012-10-10 18:56:41 UTC
Likely fixed in model refactor, please close when it is merged.

Comment 6 Dan McPherson 2013-01-29 00:07:08 UTC
Fixed with model refactor.

Comment 7 Jianwei Hou 2013-01-30 08:27:04 UTC
Verified on devenv_2744

1. Edit /usr/libexec/openshift/cartridges/php-5.3/info/manifest.yml, set scaling:
Scaling:
  Min: 2
  Max: 3 

2. Restart broker, create a scalable php application
3. Check consumed gears
libra_rs:PRIMARY> db.cloud_users.find({"login":"jhou+1"},{consumed_gears:1})
{ "_id" : ObjectId("5108d2a310c4fee306000001"), "consumed_gears" : 1 }

As we see, the min/max gears checking is done at mongo level

4. Concurrently scale up 3 gears for the app, and check consumed gears
libra_rs:PRIMARY> db.cloud_users.find({"login":"jhou+1"},{consumed_gears:1})
{ "_id" : ObjectId("5108d2a310c4fee306000001"), "consumed_gears" : 4 }

No errors.