Bug 852215 - Ruby application reports error in logs on pre-spawn server processes on INT
Summary: Ruby application reports error in logs on pre-spawn server processes on INT
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OKD
Classification: Red Hat
Component: Containers
Version: 2.x
Hardware: Unspecified
OS: All
low
low
Target Milestone: ---
: ---
Assignee: Troy Dawson
QA Contact: libra bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-08-27 21:33 UTC by Ram Ranganathan
Modified: 2015-05-14 22:58 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-02-05 17:52:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ram Ranganathan 2012-08-27 21:33:32 UTC
Description of problem:
When you create a ruby 1.8 or 1.9 application on production there's a couple of errors in the logs that connection was refused -- see below for additional info.

The issue has to do with how Passenger starts up (pre-spawns) server processes.
What the pre-spawning does is send a HTTP request back to the app, so as to force apache/mod_passenger to start up a Rack process to serve the application.
There's a bug in the URI its connecting to which has 127.0.0.1 hardcoded in it.

Version-Release number of selected component (if applicable):
2.0.16

How reproducible:
Always on prod.

Steps to Reproduce:
1. rhc app create -a $app -t ruby-1.{8,9}
2. ssh into the app and look at the app logs.
3.
  
Actual results:
Prespawn errors.

Expected results:
No errors.

Additional info:

[Sun Aug 26 04:34:36 2012] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.12 configured -- resuming normal operations
/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `new'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `'

Comment 1 Ram Ranganathan 2012-08-27 21:44:50 UTC
As a workaround, if you still want the Rack server processes pre-started (what prespawn does), you can add a 
    $app/.openshift/action_hooks/post_start_$apptype 
hook that does something like 
     curl http://$app-$ns.rhcloud.com/ -o /dev/null.

Comment 2 Troy Dawson 2012-08-28 15:33:34 UTC
Patched packages have been built.  They are not yet in the libra candidate repo.

https://brewweb.devel.redhat.com/buildinfo?buildID=231060
https://brewweb.devel.redhat.com/buildinfo?buildID=231069

Comment 3 Troy Dawson 2012-08-29 22:30:38 UTC
The fixed packages are now in the libra candidate repo.  This can be tested on developement machines.

Comment 4 Ram Ranganathan 2012-09-05 06:29:44 UTC
Verified on devenv w/ a ruby 1.8 and ruby 1.9 apps on a new devenv launched today. Assigning on QA.

Comment 5 Jianwei Hou 2012-09-05 07:46:35 UTC
Verified on denenv_2114

Steps:
1. Create scalable/non-scalable ruby-1.8/ruby-1.9 applications
2. ssh into application, check log file under $appname/logs/

Result:
No 'connection refused' errors recorded in logs.

Comment 6 Wei Sun 2012-09-11 09:48:39 UTC
Reopen this bug,because this bug can be reproduced on INT(devenv_2143),but on devenv_2146,it can not be reproduced.


info:
[sunwei@dhcp-8-229 int]$ rhc-tail-files -a rack9test -d
Password: ******

Submitting form:
rhlogin: wsun+1
debug: true
Contacting https://int.openshift.redhat.com
Attempting to tail files: rack9test/logs/*
Use ctl + c to stop
ssh -t 159d24199cce431fa8508f28bd7ec45d.rhcloud.com 'tail rack9test/logs/*'
==> rack9test/logs/error_log-20120911-000000-EST <==
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `<main>'
/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `new'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `<main>'

==> rack9test/logs/access_log-20120911-000000-EST <==
203.114.244.88 - - [11/Sep/2012:05:30:22 -0400] "GET /health HTTP/1.1" 200 1 "-" "Ruby"
/usr/bin/tail: cannot open `rack9test/logs/production.log' for reading: No such file or directory
^C
Terminating...

Comment 7 Ram Ranganathan 2012-09-19 00:27:24 UTC
There is still a pending issue on why devenv, PROD + INT has the older versions of the rubygem passenger than the builds in brew even though they are tagged as candidate -- the versions on devenv/PROD/INT are off by atleast 2 versions.

Comment 8 Troy Dawson 2012-10-19 22:10:53 UTC
The updated packages that have this fix have been built:

rubygem-passenger-3.0.4-15.el6_3
ruby193-rubygem-passenger-3.0.12-19.el6_3

Comment 9 Troy Dawson 2013-02-05 17:52:05 UTC
Passenger versions have been updated to newer versions that this bug, and are consistent across dev, int, stg and prod.


Note You need to log in before you can comment on or make changes to this bug.