Bug 1026240 - Scaled up gears are in down status for scalable app on STG
Summary: Scaled up gears are in down status for scalable app on STG
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Online
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Jhon Honce
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-11-04 09:15 UTC by Meng Bo
Modified: 2015-05-14 23:32 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-01-24 03:28:39 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Meng Bo 2013-11-04 09:15:31 UTC
Description of problem:
Create scalable app of any type, scale-up the app, check the haproxy-statue page, the new gear is in down status.

When scale-up the app to 3 gears, the app will become unavailable since the local-gear is also down.

When access the scaled-up gears' dns, it can be visited.

Version-Release number of selected component (if applicable):
STG (devenv-stage_549)

How reproducible:
always

Steps to Reproduce:
1.Create scalable app
2.Set the min gear number to scale the app up
3.Check the scaled-up gear

Actual results:
The gears are in down status and cannot be accessed. But can be visited via the gear dns directly.

Expected results:
Scaled-up gears should be available.

Additional info:
Found the following exception in haproxy_ctld.rb.log:

I, [2013-11-04T03:40:30.621166 #87713]  INFO -- : *** below you find the most recent exception thrown, this will be likely (but not certainly) the exception that made the application exit abnormally ***
E, [2013-11-04T03:40:30.621249 #87713] ERROR -- : Failed to get information from haproxy (Haproxy::ShouldRetry)
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:126:in `refresh'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:67:in `initialize'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:345:in `new'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:345:in `<top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:139:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld_daemon.rb:21:in `<main>'
I, [2013-11-04T03:40:30.621340 #87713]  INFO -- : *** below you find all exception objects found in memory, some of them may have been thrown in your application, others may just be in memory because they are standard exceptions ***
E, [2013-11-04T03:40:30.622267 #87713] ERROR -- : stream closed (IOError)

E, [2013-11-04T03:40:30.622370 #87713] ERROR -- : failed to allocate memory (NoMemoryError)

E, [2013-11-04T03:40:30.622461 #87713] ERROR -- : stack level too deep (SystemStackError)

E, [2013-11-04T03:40:30.622726 #87713] ERROR -- : exception reentered (fatal)

E, [2013-11-04T03:40:30.628289 #87713] ERROR -- : exit (SystemExit)
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:355:in `exit'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:355:in `rescue in <top (required)>'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:341:in `<top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:139:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld_daemon.rb:21:in `<main>'

Comment 1 Andy Goldstein 2013-11-04 12:57:07 UTC
I don't believe the errors listed in haproxy_ctld.rb.log are related - I see them in my app in prod.

I believe the issue is either an issue with iptables (port proxy) or the firewall. I tested creating an app in stg and my scaled up gear (gear-5277917e2587c84142000a5a-agoldste) is supposed to be accessible by haproxy at ex-std-node1.stg.rhcloud.com:35536, but it can't connect, which is why the gear is listed as DOWN in the haproxy-status page.

Comment 3 Andy Goldstein 2013-11-04 15:12:02 UTC
Please retest - this should be working now.

Comment 4 Meng Bo 2013-11-05 06:11:58 UTC
Checked on current STG (devenv-stage_552),

The scaled-up gears for scalable app work fine. 

Move bug to verified.


Note You need to log in before you can comment on or make changes to this bug.