Bug 989941 - After haproxy's manifest.yml is modified, the min gears of any web-cartridge can be increased to be greater than its max gears
After haproxy's manifest.yml is modified, the min gears of any web-cartridge ...
Product: OpenShift Online
Classification: Red Hat
Component: Pod (Show other bugs)
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Abhishek Gupta
libra bugs
Depends On:
  Show dependency treegraph
Reported: 2013-07-30 03:55 EDT by Jianwei Hou
Modified: 2015-05-14 20:19 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2014-04-24 17:34:33 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Jianwei Hou 2013-07-30 03:55:52 EDT
Description of problem:
It's a rare situation, but it makes the min and max gears of a cartridge looks strange and inconsistent. After setting the Max gears of haproxy's manifest.yml, create one scalable application of any cartridge, and set its min/max gears with cartridge-scale via rhc, the min gears of the web cartridge can be set to a value greater than the max gears.

Version-Release number of selected component (if applicable):
On devenv_3580

How reproducible:

Steps to Reproduce:
1. On node, edit haproxy's manifest.yml in /var/lib/openshift/.cartridge_repository/redhat-haproxy/0.0.4/metadata, update the value of max, Save and clear broker cache. 
Here is the detail:
      Min: 1
      Max: 2
      Multiplier: 1

2. Create one scalable application
rhc app create r19s ruby-1.9 -s --no-git --no-dns
3. Set min gears to 3 and max gears to -1 of ruby-1.9 cartridge
rhc cartridge-scale ruby-1.9 -a r19s --min 3 --max -1
4. rhc app-show r19s
rhc app-show r19s --gears
Actual results:
Can not scale up this app via RESTAPI, but can increase its gears by setting min gears of ruby-1.9 cartridge, and the min gears is even greater than the max gears. The app is consuming 3 gears when it only allows 2 gears.

# rhc app-show r19s
r19s @ http://r19s-jhou.dev.rhcloud.com/ (uuid: 907252348802326916497408)
  Domain:  jhou
  Created: 3:13 PM
  Gears:   3 (defaults to small)
  Git URL: ssh://907252348802326916497408@r19s-jhou.dev.rhcloud.com/~/git/r19s.git/
  SSH:     907252348802326916497408@r19s-jhou.dev.rhcloud.com

  ruby-1.9 (Ruby 1.9)
    Scaling: x3 (minimum: 3, maximum: 2) on small gears

  haproxy-1.4 (OpenShift Web Balancer)
    Scaling: x3 (minimum: 3, maximum: 2) on small gears

# rhc app-show --gears r19s
ID                       State   Cartridges           Size  SSH URL
------------------------ ------- -------------------- ----- ----------------------------------------------------------------------
907252348802326916497408 started ruby-1.9 haproxy-1.4 small 907252348802326916497408@r19s-jhou.dev.rhcloud.com
860993208439630341341184 started ruby-1.9 haproxy-1.4 small 860993208439630341341184@860993208439630341341184-jhou.dev.rhcloud.com
51f767d84495fd1615000001 started ruby-1.9 haproxy-1.4 small 51f767d84495fd1615000001@51f767d84495fd1615000001-jhou.dev.rhcloud.com

Expected results:
The min gears can not be greater than the max gears

Additional info:
Comment 1 Rajat Chopra 2013-07-30 15:35:25 EDT
By setting the multiplier to be 1, haproxy no longer becomes a sparse cart. And then, if it has to co-locate with the framework cartridge, then both the scaling policies must match.
The broker should not allow two cartridges with different scaling min/max to sit with each other unless one of them is a sparse cartridge (i.e. multiplier != 1).

Lowering the severity as this does not affect the current release (with unmodified haproxy cartridge).
Comment 5 Abhishek Gupta 2014-03-26 21:16:01 EDT
Proposed fix --> https://github.com/openshift/origin-server/pull/5087
Comment 6 openshift-github-bot 2014-03-27 01:19:06 EDT
Commit pushed to master at https://github.com/openshift/origin-server

Bug 989941: preventing colocation of cartridges that independently scale
Comment 7 Jianwei Hou 2014-03-27 22:09:57 EDT
Verified on devenv_4594, the logs have been moved to /var/log/openshift/broker

[root@ip-10-231-17-137 haproxy]# oo-admin-upgrade upgrade-node --version 2.0.42
Upgrader started with options: {:version=>"2.0.42", :ignore_cartridge_version=>false, :target_server_identity=>nil, :upgrade_position=>1, :num_upgraders=>1, :max_threads=>12, :gear_whitelist=>[], :num_tries=>2}
Building new upgrade queues and cluster metadata
Getting all active gears...
Getting all logins...
Writing 5 entries to gear queue for node ip-10-231-17-137 at /var/log/openshift/broker/upgrade/gear_queue_ip-10-231-17-137
Writing node queue to /var/log/openshift/broker/upgrade/node_queue
Writing cluster metadata to /var/log/openshift/broker/upgrade/cluster_metadata
Loading cluster metadata from /var/log/openshift/broker/upgrade/cluster_metadata
Loading node queue from /var/log/openshift/broker/upgrade/node_queue
Upgrading node ip-10-231-17-137 from gear queue file at /var/log/openshift/broker/upgrade/gear_queue_ip-10-231-17-137
1 of 1 nodes completed
Writing updated node queue to /var/log/openshift/broker/upgrade/node_queue
# of users: 6
# of gears: 5
# of failures: 0
# of leftovers: 0
Gear counts per thread: [5]

Additional timings:

[root@ip-10-231-17-137 upgrade]# pwd
[root@ip-10-231-17-137 upgrade]# ls
cluster_metadata  node_queue  node_queue-2014-03-27-220658  upgrade_log_ip-10-231-17-137  upgrade_results_ip-10-231-17-137
Comment 8 Jianwei Hou 2014-03-28 05:46:24 EDT
Oops! Sorry for the wrong update.
This bug is verifeid on devenv-stage_775:

1. Set Max to 2 in haproxy's manifest, restart ruby193-mcollective and re-import the cartridge using oo-admin-ctl-cartridge
service ruby193-mocllective restart
oo-broker oo-admin-broker-cache -c
oo-broker oo-admin-ctl-cartridge -c delete haproxy-1.4
oo-broker oo-admin-ctl-cartridge -c import_node
oo-broker oo-admin-ctl-cartridge -c activate --name haproxy-1.4
2. Create a scaled php-5.3 application, After creation, the scaling info of the app shows the max gears for php-5.3 is not limited, max haproxy gears is 2
php1s @ http://php1s-jhou.dev.rhcloud.com/ (uuid: 5335382d599e2ea80f000058)
  Domain:     jhou
  Created:    4:51 PM
  Gears:      1 (defaults to small)
  Git URL:    ssh://5335382d599e2ea80f000058@php1s-jhou.dev.rhcloud.com/~/git/php1s.git/
  SSH:        5335382d599e2ea80f000058@php1s-jhou.dev.rhcloud.com
  Deployment: auto (on git push)

  haproxy-1.4 (Web Load Balancer)
    Scaling: x1 (minimum: 1, maximum: 2) on small gears

  php-5.3 (PHP 5.3)
    Scaling: x1 (minimum: 1, maximum: available) on small gears
3. Set min scale for php cartridge
-> % rhc cartridge-scale php-5.3 -a php1s --min 3 --max -1
Please sign in to start a new session to ec2-107-21-89-147.compute-1.amazonaws.com.
Password: ******
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for php-5.3 ... done

4. Show app scaling info
php1s @ http://php1s-jhou.dev.rhcloud.com/ (uuid: 5335382d599e2ea80f000058)
  Domain:     jhou
  Created:    4:51 PM
  Gears:      3 (defaults to small)
  Git URL:    ssh://5335382d599e2ea80f000058@php1s-jhou.dev.rhcloud.com/~/git/php1s.git/
  SSH:        5335382d599e2ea80f000058@php1s-jhou.dev.rhcloud.com
  Deployment: auto (on git push)

  haproxy-1.4 (Web Load Balancer)
    Scaling: x1 (minimum: 1, maximum: 2) on small gears

  php-5.3 (PHP 5.3)
    Scaling: x3 (minimum: 3, maximum: available) on small gears

This is working as expected, mark as verified.
Comment 9 Abhishek Gupta 2014-03-28 14:52:36 EDT
Note to QE: Did you set the multiplier in the haproxy cartridge to 1 as well? That is what determines that the cartridge is no longer sparse and hence is now being blocked from being located with the web_framework gear.
Comment 10 Jianwei Hou 2014-03-30 22:37:15 EDT
Thanks for the note, I'll add a test case for it as well.

Verified on devenv-stage_779
After setting multiplier to 1, the haproxy can not be located with web_framework gear.

<-------------cartridge detail from cartridgte_types in datastore ------------>
"text": "{\"Name\":\"haproxy\",\"Display-Name\":\"Web Load Balancer\",\"Version\":\"1.4\",\"Description\":\"Acts as a load balancer for your web cartridge and will automatically scale up to handle incoming traffic. Is automatically added to scaled applications when they are created and cannot be removed or added to an application after the fact.\",\"License\":\"GPLv2+\",\"License-Url\":\"http://www.gnu.org/licenses/gpl-2.0.html\",\"Categories\":[\"web_proxy\",\"scales\",\"embedded\"],\"Website\":\"http://haproxy.1wt.eu/\",\"Cartridge-Version\":\"0.0.14\",\"Provides\":[\"haproxy-1.4\",\"haproxy\",\"web_proxy\"],\"Vendor\":\"http://haproxy.1wt.eu/\",\"Cartridge-Vendor\":\"redhat\",\"Endpoints\":[{\"Private-IP-Name\":\"IP\",\"Private-Port-Name\":\"PORT\",\"Private-Port\":8080,\"Public-Port-Name\":\"PROXY_PORT\",\"Mappings\":[{\"Frontend\":\"\",\"Backend\":\"\",\"Options\":{\"target_update\":true,\"connections\":-1,\"websocket\":true}},{\"Frontend\":\"/health\",\"Backend\":\"/configuration/health\",\"Options\":{\"file\":true}}]},{\"Private-IP-Name\":\"STATUS_IP\",\"Private-Port-Name\":\"STATUS_PORT\",\"Private-Port\":8080,\"Mappings\":[{\"Frontend\":\"/haproxy-status\",\"Backend\":\"/\"},{\"Frontend\":\"/health\",\"Backend\":\"\",\"Options\":{\"health\":true}}]}],\"Configure-Order\":[\"web_framework\",\"web_proxy\"],\"Components\":{\"web_proxy\":{\"Publishes\":{\"get-balancer-connection-info\":{\"Type\":\"NET_TCP:http:http\",\"Required\":false},\"publish-haproxy-status-url\":{\"Type\":\"NET_TCP:haproxy-status-info\",\"Required\":false}},\"Subscribes\":{\"set-db-connection-info\":{\"Type\":\"NET_TCP:db:connection-info\",\"Required\":false},\"set-haproxy-status-url\":{\"Type\":\"NET_TCP:haproxy-status-info\",\"Required\":false}},\"Scaling\":{\"Min\":1,\"Max\":2,\"Min-Managed\":0,\"Multiplier\":1,\"Required\":null}}},\"Group-Overrides\":[{\"components\":[\"web_framework\",\"web_proxy\"]}]}",

0 % create-scalable-php-app php1s
Application Options
Domain:     jhou
Cartridges: php-5.3
Gear Size:  default
Scaling:    yes

Creating application 'php1s' ... Cartridges ["haproxy-1.4", "php-5.3"] cannot be grouped together as they scale individually

Note You need to log in before you can comment on or make changes to this bug.