Bug 1026240 - Scaled up gears are in down status for scalable app on STG
Scaled up gears are in down status for scalable app on STG
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Jhon Honce
libra bugs
: TestBlocker
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-11-04 04:15 EST by Meng Bo
Modified: 2015-05-14 19:32 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-23 22:28:39 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Meng Bo 2013-11-04 04:15:31 EST
Description of problem:
Create scalable app of any type, scale-up the app, check the haproxy-statue page, the new gear is in down status.

When scale-up the app to 3 gears, the app will become unavailable since the local-gear is also down.

When access the scaled-up gears' dns, it can be visited.

Version-Release number of selected component (if applicable):
STG (devenv-stage_549)

How reproducible:
always

Steps to Reproduce:
1.Create scalable app
2.Set the min gear number to scale the app up
3.Check the scaled-up gear

Actual results:
The gears are in down status and cannot be accessed. But can be visited via the gear dns directly.

Expected results:
Scaled-up gears should be available.

Additional info:
Found the following exception in haproxy_ctld.rb.log:

I, [2013-11-04T03:40:30.621166 #87713]  INFO -- : *** below you find the most recent exception thrown, this will be likely (but not certainly) the exception that made the application exit abnormally ***
E, [2013-11-04T03:40:30.621249 #87713] ERROR -- : Failed to get information from haproxy (Haproxy::ShouldRetry)
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:126:in `refresh'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:67:in `initialize'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:345:in `new'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:345:in `<top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:139:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld_daemon.rb:21:in `<main>'
I, [2013-11-04T03:40:30.621340 #87713]  INFO -- : *** below you find all exception objects found in memory, some of them may have been thrown in your application, others may just be in memory because they are standard exceptions ***
E, [2013-11-04T03:40:30.622267 #87713] ERROR -- : stream closed (IOError)

E, [2013-11-04T03:40:30.622370 #87713] ERROR -- : failed to allocate memory (NoMemoryError)

E, [2013-11-04T03:40:30.622461 #87713] ERROR -- : stack level too deep (SystemStackError)

E, [2013-11-04T03:40:30.622726 #87713] ERROR -- : exception reentered (fatal)

E, [2013-11-04T03:40:30.628289 #87713] ERROR -- : exit (SystemExit)
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:355:in `exit'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:355:in `rescue in <top (required)>'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld.rb:341:in `<top (required)>'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:176:in `start_load'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:257:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:139:in `block in run'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions'
/opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:138:in `run'
/var/lib/openshift/52775d59dbd93ced67000002/haproxy/usr/bin/haproxy_ctld_daemon.rb:21:in `<main>'
Comment 1 Andy Goldstein 2013-11-04 07:57:07 EST
I don't believe the errors listed in haproxy_ctld.rb.log are related - I see them in my app in prod.

I believe the issue is either an issue with iptables (port proxy) or the firewall. I tested creating an app in stg and my scaled up gear (gear-5277917e2587c84142000a5a-agoldste) is supposed to be accessible by haproxy at ex-std-node1.stg.rhcloud.com:35536, but it can't connect, which is why the gear is listed as DOWN in the haproxy-status page.
Comment 3 Andy Goldstein 2013-11-04 10:12:02 EST
Please retest - this should be working now.
Comment 4 Meng Bo 2013-11-05 01:11:58 EST
Checked on current STG (devenv-stage_552),

The scaled-up gears for scalable app work fine. 

Move bug to verified.

Note You need to log in before you can comment on or make changes to this bug.