Bug 835176 - min/max gear restriction should be checked at the mongo level
min/max gear restriction should be checked at the mongo level
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Pod (Show other bugs)
2.x
Unspecified Unspecified
medium Severity low
: ---
: ---
Assigned To: Ravi Sankar
libra bugs
: Triaged
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-06-25 14:24 EDT by Dan McPherson
Modified: 2015-05-14 21:59 EDT (History)
5 users (show)

See Also:
Fixed In Version: devenv_2737
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-13 19:48:32 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dan McPherson 2012-06-25 14:24:56 EDT
Description of problem:

min/max gear checking is currently done in application code.  This creates race conditions for concurrent scaleup/scaledown requests.  The check should be done at the mongo level.


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Set min or max gears allowed for a cart
2. Concurrently scale up and scale down an app with min/max gears
3.
  
Actual results:
Can have more or less gears than allowed by min/max gear settings


Expected results:
Should work as expected regardless of concurrency


Additional info:
Comment 1 Rob Millner 2012-06-26 16:54:49 EDT
An important caveat is that if there's an error or exception from the save method due to Mongodb; then any steps that had been taken to create a gear must be un-done before the call completes or a half-built gear could be left behind.
Comment 2 John Poelstra 2012-07-09 12:11:05 EDT
fixed on friday, pull requesting coming today
Comment 3 Ravi Sankar 2012-07-10 12:52:23 EDT
Fixed in git rev 0a45d8f6f86ddfdd.
Testing must be done by firing multiple scale-up/scale-down events in parallel and make sure number of application gears is consistent with consumed gears.
Comment 4 Xiaoli Tian 2012-07-11 07:26:04 EDT
Tested this on devenv_1884, this bug should be partly fixed, test with the following steps:

1. Change min/max gears for php to 2 /3 
	Scaling:
	  Min: 2
	  Max: 3
2. Create a scaling php app, then it should have already consumed 3 gears (2 web cartridge gears and 1 haproxy gear)

3. Open 3 terminals to scale up at the same time:

   2 of them return success, 1 of them return:
"Failed to add event scale-up to application scalphpapp1 due to: Cannot scale up beyond maximum gear limit '3' in app scalphpapp1.","field":null}],

4. Check the apps's gear_groups, the max gear is 3.
data": [
    {
      "gears": [
        {
          "id": "eb7b4164dccc4f36887c2d3bfd5fcc55",
          "state": "started"
        }
      ],
      "cartridges": [
        {
          "name": "haproxy-1.4"
        },
        {
          "name": "php-5.3"
        }
      ],
      "name": "@@app/comp-proxy/cart-php-5.3",
      "gear_profile": "small"
    },
    {
      "gears": [
        {
          "id": "8c528a9e5036480ca986989faa505afe",
          "state": "started"
        },
        {
          "id": "d6ec9c2ad95d4ca4a8766cc4b9633e97",
          "state": "started"
        },
        {
          "id": "4075a090a7f04d18a957702ce28d5290",
          "state": "started"
        }
      ],
      "cartridges": [
        {
          "name": "php-5.3"
        }
      ],
      "name": "@@app/comp-web/cart-php-5.3",
      "gear_profile": "small"
    }
  ],


5. Check consumed gears:
	irb(main):004:0> user1.consumed_gears
	=> 5
	irb(main):005:0> user1.applications.length
	=> 1
	irb(main):006:0> user1.applications[0].group_instances[1].gears.length
	=> 1
	irb(main):007:0> user1.applications[0].group_instances[0].gears.length
	=> 3

Then the consumed_gears is larger than the actual gears, the consumed gears should be 4 as well.
Comment 5 Mike McGrath 2012-10-10 14:56:41 EDT
Likely fixed in model refactor, please close when it is merged.
Comment 6 Dan McPherson 2013-01-28 19:07:08 EST
Fixed with model refactor.
Comment 7 Jianwei Hou 2013-01-30 03:27:04 EST
Verified on devenv_2744

1. Edit /usr/libexec/openshift/cartridges/php-5.3/info/manifest.yml, set scaling:
Scaling:
  Min: 2
  Max: 3 

2. Restart broker, create a scalable php application
3. Check consumed gears
libra_rs:PRIMARY> db.cloud_users.find({"login":"jhou+1@redhat.com"},{consumed_gears:1})
{ "_id" : ObjectId("5108d2a310c4fee306000001"), "consumed_gears" : 1 }

As we see, the min/max gears checking is done at mongo level

4. Concurrently scale up 3 gears for the app, and check consumed gears
libra_rs:PRIMARY> db.cloud_users.find({"login":"jhou+1@redhat.com"},{consumed_gears:1})
{ "_id" : ObjectId("5108d2a310c4fee306000001"), "consumed_gears" : 4 }

No errors.

Note You need to log in before you can comment on or make changes to this bug.