Bug 1027483

Summary:	get error 'failed to get application info from broker' when trying to scale up by flooding the app
Product:	OpenShift Online	Reporter:	Peter Ruan <pruan>
Component:	Containers	Assignee:	Mrunal Patel <mpatel>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	2.x	CC:	abhgupta, chunchen
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-01-24 03:28:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Peter Ruan 2013-11-06 22:49:30 UTC

Description of problem:
I get the error in the scale_events.log when trying for it to scale up by flooding it with packets.

I, [2013-11-06T17:22:32.247857 #27409]  INFO -- : GEAR_UP - capacity: 206.25% gear_count: 1 sessions: 33 up_thresh: 90.0%
I, [2013-11-06T17:24:43.145262 #27409]  INFO -- : GEAR_UP - add-gear: exit: 1  stdout: Failed to get application info from the broker.


Version-Release number of selected component (if applicable):
devenv_3997

How reproducible:
sometimes.

Steps to Reproduce:
1. rhc app create app1 jbosseap-6 -s
2. rhc cartridge add postgresql-8.4 -a app1
3. ab -n 20000 -c 33 http://<app_url>/    # app should have 3 gears after this
4. ab -n 20000 -c 44 http://<app_url>/    # app should have 4 gears after this

Actual results:
only 3 gears.
scale_event.log has the following:

I, [2013-11-06T17:22:32.247857 #27409]  INFO -- : GEAR_UP - capacity: 206.25% gear_count: 1 sessions: 33 up_thresh: 90.0%
I, [2013-11-06T17:24:43.145262 #27409]  INFO -- : GEAR_UP - add-gear: exit: 1  stdout: Failed to get application info from the broker.

Expected results:
4 gears.

Additional info:

Comment 1 Abhishek Gupta 2013-11-09 00:09:32 UTC

Mrunal: Please add the exception logging on the haproxy end and then send it over to QE as not reproducible. Would be best to let them run their tests again to get more information on this bug.

Comment 2 Mrunal Patel 2013-11-09 00:37:08 UTC

Abhishesk,
The logging that we have is sufficient.

From above 
----
I, [2013-11-06T17:24:43.145262 #27409]  INFO -- : GEAR_UP - add-gear: exit: 1  stdout: Failed to get application info from the broker.
----

QE, could you retest the bug? We were unable to reproduce the issue.

Comment 3 Mrunal Patel 2013-11-09 00:39:07 UTC

Disregard the above comment. I will add exception logging.

Comment 4 Mrunal Patel 2013-11-09 00:59:34 UTC

https://github.com/openshift/origin-server/pull/4143

Comment 5 chunchen 2013-11-11 06:39:01 UTC

It's fixed, verified on devenv_4016, please refer to the following results:

I, [2013-11-11T01:34:40.833123 #3108]  INFO -- : GEAR_UP - capacity: 93.75% gear_count: 2 sessions: 30 up_thresh: 90.0%
I, [2013-11-11T01:35:39.019541 #5365]  INFO -- : Starting haproxy_ctld