Bug 1044980

Summary: Ruby app trys to load module and connection is refused
Product: OpenShift Container Platform Reporter: Nan Wei <nwei>
Component: ImageStreamsAssignee: Jason DeTiberus <jdetiber>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 2.0.0CC: adellape, bleanhar, gpei, jdetiber, vvitek, xtian, yanpzhan
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: rubygem-passenger-3.0.21-12.el6op Doc Type: Bug Fix
Doc Text:
A change to Passenger caused applications using Ruby 1.8 or 1.9 cartridges to report pre-starting errors, and applications would only start after a first request was made. The issue was fixed for Ruby 1.9 cartridges in an earlier release of OpenShift Enterprise; this bug fix updates Passenger to correct the issue for Ruby 1.8 cartridges as well. Pre-starting now functions properly for both Ruby cartridges, and the previous errors no longer occur.
Story Points: ---
Clone Of:
: 1212583 (view as bug list) Environment:
Last Closed: 2014-06-23 07:37:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1212583    

Description Nan Wei 2013-12-19 11:21:55 UTC
Description of problem:
Create ruby1.8 ruby 1.9 app, and check the error_log for the apps, it shows some error message as follow: 
"[Thu Dec 19 04:42:24 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c0,c1003
[Thu Dec 19 04:42:24 2013] [notice] Digest: generating secret for digest authentication ...
[Thu Dec 19 04:42:24 2013] [notice] Digest: done
[Thu Dec 19 04:42:24 2013] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations
/usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:145
/usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:145"
But it doesn't exist error message like this in the deven enironment.
OpenShift Enterprise 2.0.1 puddle: [2.0.z/2013-12-18.1]
How reproducible:
always
Steps to Reproduce:
1. Create a app 
#rhc app create $appname ruby-1.8/ruby-1.9.
2. cat haproyx/logs/scale_events.log
#cat /var/lib/openshift/$uuid/haproyx/logs/scale_events.log
Actual results:
[Thu Dec 19 04:42:24 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c0,c1003
[Thu Dec 19 04:42:24 2013] [notice] Digest: generating secret for digest authentication ...
[Thu Dec 19 04:42:24 2013] [notice] Digest: done
[Thu Dec 19 04:42:24 2013] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations
/usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:145
/usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request'
    from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:145
Expected results:
There is no error messages.

Comment 2 Nan Wei 2014-02-19 04:03:15 UTC
This bug isn't reproduced in ose puddle 2014-02-12.1

Comment 3 Nan Wei 2014-04-04 08:59:32 UTC
Openshift enterprise: 2.1/2014-04-03.2

1. Create ruby-1.8/1.9 app
2. Ssh into app and check ~/app-root/logs/ruby.log

[ru-nweidomain.ose21-manual.com.cn 533e6de8fdae85dd7e00001e]\> cat app-root/logs/ruby.log 
httpd: Could not reliably determine the server's fully qualified domain name, using node2.ose21-manual.com.cn for ServerName
[Fri Apr 04 01:31:47 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c3,c339
httpd: Could not reliably determine the server's fully qualified domain name, using node2.ose21-manual.com.cn for ServerName
[Fri Apr 04 01:31:47 2014] [notice] Digest: generating secret for digest authentication ...
[Fri Apr 04 01:31:47 2014] [notice] Digest: done
[Fri Apr 04 01:31:47 2014] [notice] Apache/2.2.22 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations
/usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
	from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new'
	from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect'
	from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket'
	from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request'
	from /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:145

So move it new.

Comment 6 Brenton Leanhardt 2014-04-08 13:47:26 UTC
I figured out what is going on.  Passenger is set to prepawn.  Our apache module is actually patched so that OpenShift can correctly spawn on an IP other than 127.0.0.1.  However, the passenger rubygem is not patched.  See /usr/share/rubygems/gems/passenger-3.0.21/helper-scripts/prespawn:

class TCPPrespawnLocation < PrespawnLocation
        def request_port
                @uri.port
        end

        def connect
                TCPSocket.new('127.0.0.1', request_port)
        end
end

It turns out it's trying to respawn by issuing a request to itself.  However 127.0.0.1 is hardcoded.  In my dev environment there actually was something listening on 127.0.0.1:8080 which is why I didn't see the error.

The good news is that this error is "harmless" (technically it breaks prespawning).  We'll need to patch the passenger rubygem to know how to prespawn on something other than 127.0.0.1.  I'll have to see how Online works around this (if they do).

Comment 8 Yanping Zhang 2014-06-13 06:29:57 UTC
Verified on ose 2.1.z/2014-06-12.2
Steps to verify:
1. Create ruby-1.8/1.9 app
2. Ssh into app and check ~/app-root/logs/ruby.log

Actual results:
2.Display no errors:
[rubytest-dom13.ose21z-manual.com.cn 539a96b9db26c8b7e9000062]\> cat app-root/logs/ruby.log 
httpd: Could not reliably determine the server's fully qualified domain name, using node2.ose21z-manual.com.cn for ServerName
[Thu Jun 12 23:14:27 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c3,c80
httpd: Could not reliably determine the server's fully qualified domain name, using node2.ose21z-manual.com.cn for ServerName
[Thu Jun 12 23:14:27 2014] [notice] Digest: generating secret for digest authentication ...
[Thu Jun 12 23:14:27 2014] [notice] Digest: done
[Thu Jun 12 23:14:27 2014] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations
- - - [12/Jun/2014:23:14:29 -0700] "HEAD / HTTP/1.1" 200 - "-" "-"
- - - [12/Jun/2014:23:14:30 -0700] "HEAD / HTTP/1.1" 200 - "-" "-"
- - - [12/Jun/2014:23:14:34 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (Red Hat) (internal dummy connection)"
- - - [12/Jun/2014:23:14:35 -0700] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.15 (Red Hat) (internal dummy connection)"

Comment 11 errata-xmlrpc 2014-06-23 07:37:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-0781.html