Bug 852215 - Ruby application reports error in logs on pre-spawn server processes on INT
Ruby application reports error in logs on pre-spawn server processes on INT
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified All
low Severity low
: ---
: ---
Assigned To: Troy Dawson
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-27 17:33 EDT by Ram Ranganathan
Modified: 2015-05-14 18:58 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-02-05 12:52:05 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ram Ranganathan 2012-08-27 17:33:32 EDT
Description of problem:
When you create a ruby 1.8 or 1.9 application on production there's a couple of errors in the logs that connection was refused -- see below for additional info.

The issue has to do with how Passenger starts up (pre-spawns) server processes.
What the pre-spawning does is send a HTTP request back to the app, so as to force apache/mod_passenger to start up a Rack process to serve the application.
There's a bug in the URI its connecting to which has 127.0.0.1 hardcoded in it.

Version-Release number of selected component (if applicable):
2.0.16

How reproducible:
Always on prod.

Steps to Reproduce:
1. rhc app create -a $app -t ruby-1.{8,9}
2. ssh into the app and look at the app logs.
3.
  
Actual results:
Prespawn errors.

Expected results:
No errors.

Additional info:

[Sun Aug 26 04:34:36 2012] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.12 configured -- resuming normal operations
/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `new'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
    from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `'
Comment 1 Ram Ranganathan 2012-08-27 17:44:50 EDT
As a workaround, if you still want the Rack server processes pre-started (what prespawn does), you can add a 
    $app/.openshift/action_hooks/post_start_$apptype 
hook that does something like 
     curl http://$app-$ns.rhcloud.com/ -o /dev/null.
Comment 2 Troy Dawson 2012-08-28 11:33:34 EDT
Patched packages have been built.  They are not yet in the libra candidate repo.

https://brewweb.devel.redhat.com/buildinfo?buildID=231060
https://brewweb.devel.redhat.com/buildinfo?buildID=231069
Comment 3 Troy Dawson 2012-08-29 18:30:38 EDT
The fixed packages are now in the libra candidate repo.  This can be tested on developement machines.
Comment 4 Ram Ranganathan 2012-09-05 02:29:44 EDT
Verified on devenv w/ a ruby 1.8 and ruby 1.9 apps on a new devenv launched today. Assigning on QA.
Comment 5 Jianwei Hou 2012-09-05 03:46:35 EDT
Verified on denenv_2114

Steps:
1. Create scalable/non-scalable ruby-1.8/ruby-1.9 applications
2. ssh into application, check log file under $appname/logs/

Result:
No 'connection refused' errors recorded in logs.
Comment 6 Wei Sun 2012-09-11 05:48:39 EDT
Reopen this bug,because this bug can be reproduced on INT(devenv_2143),but on devenv_2146,it can not be reproduced.


info:
[sunwei@dhcp-8-229 int]$ rhc-tail-files -a rack9test -d
Password: ******

Submitting form:
rhlogin: wsun+1@redhat.com
debug: true
Contacting https://int.openshift.redhat.com
Attempting to tail files: rack9test/logs/*
Use ctl + c to stop
ssh -t 159d24199cce431fa8508f28bd7ec45d@rack9test-wsunbacktest.int.rhcloud.com 'tail rack9test/logs/*'
==> rack9test/logs/error_log-20120911-000000-EST <==
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `<main>'
/opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `new'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:105:in `connect'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:86:in `socket'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:90:in `head_request'
	from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.12/helper-scripts/prespawn:145:in `<main>'

==> rack9test/logs/access_log-20120911-000000-EST <==
203.114.244.88 - - [11/Sep/2012:05:30:22 -0400] "GET /health HTTP/1.1" 200 1 "-" "Ruby"
/usr/bin/tail: cannot open `rack9test/logs/production.log' for reading: No such file or directory
^C
Terminating...
Comment 7 Ram Ranganathan 2012-09-18 20:27:24 EDT
There is still a pending issue on why devenv, PROD + INT has the older versions of the rubygem passenger than the builds in brew even though they are tagged as candidate -- the versions on devenv/PROD/INT are off by atleast 2 versions.
Comment 8 Troy Dawson 2012-10-19 18:10:53 EDT
The updated packages that have this fix have been built:

rubygem-passenger-3.0.4-15.el6_3
ruby193-rubygem-passenger-3.0.12-19.el6_3
Comment 9 Troy Dawson 2013-02-05 12:52:05 EST
Passenger versions have been updated to newer versions that this bug, and are consistent across dev, int, stg and prod.

Note You need to log in before you can comment on or make changes to this bug.