Bug 852215

Summary: Ruby application reports error in logs on pre-spawn server processes on INT
Product: OKD Reporter: Ram Ranganathan <ramr>
Component: ContainersAssignee: Troy Dawson <tdawson>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: low Docs Contact:
Priority: low    
Version: 2.xCC: jhou, mfisher, wsun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-02-05 17:52:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Ram Ranganathan 2012-08-27 21:33:32 UTC
Description of problem:
When you create a ruby 1.8 or 1.9 application on production there's a couple of errors in the logs that connection was refused -- see below for additional info.

The issue has to do with how Passenger starts up (pre-spawns) server processes.
What the pre-spawning does is send a HTTP request back to the app, so as to force apache/mod_passenger to start up a Rack process to serve the application.
There's a bug in the URI its connecting to which has 127.0.0.1 hardcoded in it.

Version-Release number of selected component (if applicable):
2.0.16

How reproducible:
Always on prod.

Steps to Reproduce:
1. rhc app create -a $app -t ruby-1.{8,9}
2. ssh into the app and look at the app logs.
3.
  
Actual results:
Prespawn errors.

Expected results:
No errors.

Additional info:

[Sun Aug 26 04:34:36 2012] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.12 configured -- resuming normal operations
/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `new'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `'

Comment 1 Ram Ranganathan 2012-08-27 21:44:50 UTC
As a workaround, if you still want the Rack server processes pre-started (what prespawn does), you can add a 
    $app/.openshift/action_hooks/post_start_$apptype 
hook that does something like 
     curl http://$app-$ns.rhcloud.com/ -o /dev/null.

Comment 2 Troy Dawson 2012-08-28 15:33:34 UTC
Patched packages have been built.  They are not yet in the libra candidate repo.

https://brewweb.devel.redhat.com/buildinfo?buildID=231060
https://brewweb.devel.redhat.com/buildinfo?buildID=231069

Comment 3 Troy Dawson 2012-08-29 22:30:38 UTC
The fixed packages are now in the libra candidate repo.  This can be tested on developement machines.

Comment 4 Ram Ranganathan 2012-09-05 06:29:44 UTC
Verified on devenv w/ a ruby 1.8 and ruby 1.9 apps on a new devenv launched today. Assigning on QA.

Comment 5 Jianwei Hou 2012-09-05 07:46:35 UTC
Verified on denenv_2114

Steps:
1. Create scalable/non-scalable ruby-1.8/ruby-1.9 applications
2. ssh into application, check log file under $appname/logs/

Result:
No 'connection refused' errors recorded in logs.

Comment 6 Wei Sun 2012-09-11 09:48:39 UTC
Reopen this bug,because this bug can be reproduced on INT(devenv_2143),but on devenv_2146,it can not be reproduced.


info:
[sunwei@dhcp-8-229 int]$ rhc-tail-files -a rack9test -d
Password: ******

Submitting form:
rhlogin: wsun+1
debug: true
Contacting https://int.openshift.redhat.com
Attempting to tail files: rack9test/logs/*
Use ctl + c to stop
ssh -t 159d24199cce431fa8508f28bd7ec45d.rhcloud.com 'tail rack9test/logs/*'
==> rack9test/logs/error_log-20120911-000000-EST <==
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `<main>'
/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `new'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `<main>'

==> rack9test/logs/access_log-20120911-000000-EST <==
203.114.244.88 - - [11/Sep/2012:05:30:22 -0400] "GET /health HTTP/1.1" 200 1 "-" "Ruby"
/usr/bin/tail: cannot open `rack9test/logs/production.log' for reading: No such file or directory
^C
Terminating...

Comment 7 Ram Ranganathan 2012-09-19 00:27:24 UTC
There is still a pending issue on why devenv, PROD + INT has the older versions of the rubygem passenger than the builds in brew even though they are tagged as candidate -- the versions on devenv/PROD/INT are off by atleast 2 versions.

Comment 8 Troy Dawson 2012-10-19 22:10:53 UTC
The updated packages that have this fix have been built:

rubygem-passenger-3.0.4-15.el6_3
ruby193-rubygem-passenger-3.0.12-19.el6_3

Comment 9 Troy Dawson 2013-02-05 17:52:05 UTC
Passenger versions have been updated to newer versions that this bug, and are consistent across dev, int, stg and prod.