Bug 1026240

Summary:	Scaled up gears are in down status for scalable app on STG
Product:	OpenShift Online	Reporter:	Meng Bo <bmeng>
Component:	Containers	Assignee:	Jhon Honce <jhonce>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.x	CC:	agoldste
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-01-24 03:28:39 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Meng Bo 2013-11-04 09:15:31 UTC

Description of problem:
Create scalable app of any type, scale-up the app, check the haproxy-statue page, the new gear is in down status.

When scale-up the app to 3 gears, the app will become unavailable since the local-gear is also down.

When access the scaled-up gears' dns, it can be visited.

Version-Release number of selected component (if applicable):
STG (devenv-stage_549)

How reproducible:
always

Steps to Reproduce:
1.Create scalable app
2.Set the min gear number to scale the app up
3.Check the scaled-up gear

Actual results:
The gears are in down status and cannot be accessed. But can be visited via the gear dns directly.

Expected results:
Scaled-up gears should be available.

Additional info:
Found the following exception in haproxy_ctld.rb.log:

I, [2013-11-04T03:40:30.621166 #87713]  INFO -- : *** below you find the most recent exception thrown, this will be likely (but not certainly) the exception that made the application exit abnormally ***
E, [2013-11-04T03:40:30.621249 #87713] ERROR -- : Failed to get information from haproxy (Haproxy::ShouldRetry)
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:126:in `refresh'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:67:in `initialize'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:345:in `new'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:345:in `<top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:139:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld_daemon.rb:21:in `<main>'
I, [2013-11-04T03:40:30.621340 #87713]  INFO -- : *** below you find all exception objects found in memory, some of them may have been thrown in your application, others may just be in memory because they are standard exceptions ***
E, [2013-11-04T03:40:30.622267 #87713] ERROR -- : stream closed (IOError)

E, [2013-11-04T03:40:30.622370 #87713] ERROR -- : failed to allocate memory (NoMemoryError)

E, [2013-11-04T03:40:30.622461 #87713] ERROR -- : stack level too deep (SystemStackError)

E, [2013-11-04T03:40:30.622726 #87713] ERROR -- : exception reentered (fatal)

E, [2013-11-04T03:40:30.628289 #87713] ERROR -- : exit (SystemExit)
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:355:in `exit'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:355:in `rescue in <top (required)>'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:341:in `<top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:139:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld_daemon.rb:21:in `<main>'

Comment 1 Andy Goldstein 2013-11-04 12:57:07 UTC

I don't believe the errors listed in haproxy_ctld.rb.log are related - I see them in my app in prod.

I believe the issue is either an issue with iptables (port proxy) or the firewall. I tested creating an app in stg and my scaled up gear (gear-5277917e2587c84142000a5a-agoldste) is supposed to be accessible by haproxy at ex-std-node1.stg.rhcloud.com:35536, but it can't connect, which is why the gear is listed as DOWN in the haproxy-status page.

Comment 3 Andy Goldstein 2013-11-04 15:12:02 UTC

Please retest - this should be working now.

Comment 4 Meng Bo 2013-11-05 06:11:58 UTC

Checked on current STG (devenv-stage_552),

The scaled-up gears for scalable app work fine. 

Move bug to verified.