Bug 1123353 - set-multiplier action should not be allowed on a non-HA scaling app.
Summary: set-multiplier action should not be allowed on a non-HA scaling app.
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 2.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Luke Meyer
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1123371 1124306
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-25 12:05 UTC by Johnny Liu
Modified: 2014-08-26 13:53 UTC (History)
6 users (show)

Fixed In Version: rubygem-openshift-origin-controller-1.23.10.9-1.el6op
Doc Type: Bug Fix
Doc Text:
When the 'oo-admin-ctl-app -c set-multiplier' command was run for the web framework cartridge of a scalable application with only one gear, the MongoDB record for the application was broken, making the application unscalable. This bug fix updates the application model code in the controller, and MongoDB records for application are no longer susceptible to this issue as a result.
Clone Of:
: 1123371 (view as bug list)
Environment:
Last Closed: 2014-08-26 13:53:01 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1095 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.1.5 bug fix and enhancement update 2014-08-26 17:51:34 UTC

Description Johnny Liu 2014-07-25 12:05:21 UTC
Description of problem:
Currently set-multiplier action is allowed to a non-HA scaling app, that would cause inconsistent behaviour. See the following for more details.

Version-Release number of selected component (if applicable):
puddle-2-1-2014-07-23

How reproducible:
Always

Steps to Reproduce:
1.Set up env, 1 broker + 2 nodes, and make sure ALLOW_MULTIPLE_HAPROXY_ON_NODE="false" in broker.conf
2.Create 2 scaling app.
$ rhc app create scaperlapp perl -s
$ rhc app create scaphp53app php-5.3 -s
3.To scaphp53app, do the following operation
# oo-admin-ctl-app -l jialiu -a scaphp53app -c set-multiplier --cartridge haproxy-1.4 --multiplier 1
4.To scaperlapp, do the following operation
$ rhc cartridge-scale perl-5.10 --min 2 -a scaperlapp
# oo-admin-ctl-app -l jialiu -a scaperlapp -c set-multiplier --cartridge haproxy-1.4 --multiplier 1
5.Check app info in server's mongodb.
6.Scale up for this two app.
$ rhc cartridge-scale perl-5.10 --min 4 -a scaperlapp
$ rhc cartridge-scale php-5.3 --min 4 -a scaphp53app

Actual results:
Step 5:
multiplier setting is not saved into mongo for scaperlapp, but saved for scaphp53app
ose:PRIMARY> db.applications.findOne({"name":"scaperlapp"},{group_overrides: 1})
{
	"_id" : ObjectId("53d235ccf13a79cd3c000098"),
	"group_overrides" : [
		{
			"components" : [
				{
					"comp" : "web_proxy",
					"cart" : "haproxy-1.4",
					"cart_id" : ObjectId("53a02edebe739b79db000004")
				},
				{
					"comp" : "perl-5.10",
					"cart" : "perl-5.10",
					"cart_id" : ObjectId("53a02edebe739b79db00000e")
				}
			],
			"min_gears" : 2
		}
	]
}

ose:PRIMARY> db.applications.findOne({"name":"scaphp53app"},{group_overrides: 1})
{
	"_id" : ObjectId("53d23597f13a79cd3c00007a"),
	"group_overrides" : [
		{
			"components" : [
				{
					"comp" : "web_proxy",
					"cart" : "haproxy-1.4",
					"cart_id" : ObjectId("53a02edebe739b79db000004"),
					"multiplier" : 1
				},
				{
					"comp" : "php-5.3",
					"cart" : "php-5.3",
					"cart_id" : ObjectId("53a02edebe739b79db000002")
				}
			]
		}
	]
}
Setp 6:
scaperlapp is scaled up successfully, but failed for scaphp53app.
$ rhc cartridge-scale perl-5.10 --min 4 -a scaperlapp
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for perl-5.10 ... done

perl-5.10 (Perl 5.10)
---------------------
  Scaling: x4 (minimum: 4, maximum: available) on small gears

$ rhc cartridge-scale php-5.3 --min 4 -a scaphp53app
This operation will run until the application is at the minimum scale and may take several minutes.
Setting scale range for php-5.3 ... 
Unable to complete the requested operation due to: No nodes available. Please try again and contact support if the issue persists. 
Reference ID: b1e3cf972ab39524c7210fb866bc649c

Seem like broker is treating this app as HA scaling app, and take multiplier setting into consideration to select available node, and the error message is confused to user, will open another bug to tracking helpless error message issue.

Expected results:
If this ap is not a HA scaling app, set-multiplier action should be allowed, that would make everything happy, so such inconsistence.

Additional info:

Comment 1 Luke Meyer 2014-07-25 12:53:38 UTC
The multiplier doesn't really make sense if the app isn't HA. I'm curious what happens to non-HA apps when the default multiplier is set at the broker... probably nothing. So I'm wondering why setting the multiplier explicitly does anything.

Just to point it out above - the difference between the apps is that one was scaled to 2 gears before applying the multiplier. That one had no problems and didn't seem to even record the multiplier. I'm not sure why the one-gear app recorded the multiplier, and I think the "no nodes available" message is probably misleading.

Comment 2 Brenton Leanhardt 2014-07-29 20:05:13 UTC
We'll need this PR too: https://github.com/openshift/origin-server/pull/5669

Comment 4 Luke Meyer 2014-08-14 12:09:11 UTC
commits cherry-picked from origin:

    commit 42d048dbb774b35c1895395851f0f145297fb88f
    Author: Abhishek Gupta <abhgupta>
    Date:   Mon Jul 28 13:16:01 2014 -0700
    
        Bug 1123371: Fixing issue with setting the cartridge multiplier

    commit 3b749fec58521344e1977af39fbaefc3bc0b2ddb
    Author: Abhishek Gupta <abhgupta>
    Date:   Tue Jul 29 12:06:14 2014 -0700
    
        Bug 1124306: Handling nil min/max scale values for cartridge

Comment 8 Ma xiaoqiang 2014-08-18 06:16:15 UTC
Check on puddle [2.1.z/2014-08-15.1]

1.Create two scalable apps
#rhc app create scaperlapp perl -s -g medium
#rhc app create scaphp53app php-5.3 -s -g medium 

2.set multiplier for these two apps
#oo-admin-ctl-app -l xiaom -a scaphp53app -c set-multiplier --cartridge haproxy-1.4 --multiplier 1
#rhc cartridge-scale perl-5.10 --min 2 -a scaperlapp
#oo-admin-ctl-app -l xiaom -a scaperlapp -c set-multiplier --cartridge haproxy-1.4 --multiplier 1

3. Check info of apps
>db.applications.findOne({"name":"scaphp53app"},{group_overrides: 1})
>db.applications.findOne({"name":"scaperlapp"},{group_overrides: 1})
 
Output:
db.applications.findOne({"name":"scaphp53app"},{group_overrides: 1})
{
        "_id" : ObjectId("53f19565db26c808750000c5"),
        "group_overrides" : [
                {
                        "components" : [
                                {
                                        "comp" : "web_proxy",
                                        "cart" : "haproxy-1.4",
                                        "cart_id" : ObjectId("53f16722db26c80275000004"),
                                        "multiplier" : 1
                                },
                                {
                                        "comp" : "php-5.3",
                                        "cart" : "php-5.3",
                                        "cart_id" : ObjectId("53f16722db26c80275000002")
                                }
                        ]
                }
        ]
}
> db.applications.findOne({"name":"scaperlapp"},{group_overrides: 1})
{
        "_id" : ObjectId("53f194fadb26c8087500008f"),
        "group_overrides" : [
                {
                        "components" : [
                                {
                                        "comp" : "web_proxy",
                                        "cart" : "haproxy-1.4",
                                        "cart_id" : ObjectId("53f16722db26c80275000004"),
                                        "multiplier" : 1
                                },
                                {
                                        "comp" : "perl-5.10",
                                        "cart" : "perl-5.10",
                                        "cart_id" : ObjectId("53f16723db26c8027500000e")
                                }
                        ],
                        "min_gears" : 2
                }
        ]
}
4. set 'ALLOW_MULTIPLE_HAPROXY_ON_NODE' to true on broker
#cat /etc/openshift/broker.conf
<--snip-->
ALLOW_MULTIPLE_HAPROXY_ON_NODE="true"
<--snip-->
5. Scale-up the apps
#rhc cartridge-scale perl-5.10 --min 4 -a scaperlapp
#rhc cartridge-scale php-5.3 --min 4 -a scaphp53app

The apps can be scaled up successfully.

Comment 9 Ma xiaoqiang 2014-08-19 01:43:19 UTC
Continue to do the next steps
6. Check the haproxy
#rhc app show  scaphp53app --gears
#rhc app show  scaperlapp  --gears
The scaled-up gears contains one  haproxy at least.
7.scale down the scaperlapp
#rhc cartridge-scale perl --min 1 -a scaperlapp
#rhc app scale-down -a scaperlapp
The 'scaperlapp' can be scaled down successfully.

So move this issue to VERIFIED

Comment 11 errata-xmlrpc 2014-08-26 13:53:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1095.html


Note You need to log in before you can comment on or make changes to this bug.