Description of problem: create a ruby-1.9 app, connect to the app to check the log in /var/lib/openshift/<uuid>/ruby/logs/, Errno::ECONNREFUSED shown Version-Release number of selected component (if applicable): STG(devenv-stage_406) rhc-1.11.4 How reproducible: Always Steps to Reproduce: 1. create a ruby-1.9 app rhc app create r19 ruby-1.9 2. ssh into app rhc app ssh r19 3. check the log cat /var/lib/openshift/51e4feab2587c820ec00037a/ruby/logs/error_log-20130716-000000-EST Actual results: [r19-test.stg.rhcloud.com 51e4feab2587c820ec00037a]\> cat /var/lib/openshift/51e4feab2587c820ec00037a/ruby/logs/error_log-20130716-000000-EST [Tue Jul 16 04:05:32 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c3,c524 [Tue Jul 16 04:05:32 2013] [notice] Digest: generating secret for digest authentication ... [Tue Jul 16 04:05:32 2013] [notice] Digest: done [Tue Jul 16 04:05:32 2013] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.17 configured -- resuming normal operations /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED) from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:105:in `new' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:105:in `connect' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:86:in `socket' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:90:in `head_request' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:145:in `<main>' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED) from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:105:in `new' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:105:in `connect' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:86:in `socket' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:90:in `head_request' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.17/helper-scripts/prespawn:145:in `<main>' Expected results: no error shown. Additional info: no error shown with ruby-1.8 app
I cannot reproduce this on latest devenv Results are as expected. [Wed Aug 21 05:38:46 2013] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c0,c502 [Wed Aug 21 05:38:46 2013] [notice] Digest: generating secret for digest authentication ... [Wed Aug 21 05:38:46 2013] [notice] Digest: done [Wed Aug 21 05:38:46 2013] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations
test OK on devenv_3680. but issue still can be reproduced on INT(devenv_3680).
Is there any way how to get direct access on INT devenv to test it? If not I am not able to continue in fixing this bug.
(In reply to Petr Brazdil from comment #3) > Is there any way how to get direct access on INT devenv to test it? > > If not I am not able to continue in fixing this bug. To get logs, you could contact ops people, to reproduce it, INT(devenv_3680) means int.openshift.redhat.com which matches devenv_3680.
Any progress of this bug? I could reproduce this issue on OSE with package openshift-origin-cartridge-ruby-1.16.0-1.git.292.cb19e4f.el6op.noarch :)
Hi guys, I think this bug is a lefover from: https://bugzilla.redhat.com/show_bug.cgi?id=1022956 Passenger is trying to 'pre-spawn' the application, using the HEAD request. However, the HEAD request is fired against '127.0.0.1' and not against the application domain name. The request include 'Host' header, that should tell Apache, where to pass the request and 'wake up' the Rails application properly. However, the request is made against port 8080: HEAD 127.0.0.1:8080/ Host: 127.1.244.129 # -> Gear IP ... In case of devenv, the request return 404, because the broker is running on port 8080? In case of INT or PROD, the port 8080 is not used and thus the 'Connection refused' is thrown into the application log. In order to fix this bug: 1. The request must be made against the Apache which runs on the Gear 2. The correct port must be specified 3. The Host header must be processed correctly.
The issue can't be reproduced on devenv_4200. [r19-111.dev.rhcloud.com 52ce69fbb604fc8d7900019b]\> cat ruby/logs/error_log-20140109-000000-EST [Thu Jan 09 04:21:19 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c0,c1005 [Thu Jan 09 04:21:19 2014] [notice] Digest: generating secret for digest authentication ... [Thu Jan 09 04:21:19 2014] [notice] Digest: done [Thu Jan 09 04:21:19 2014] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations Will test it on INT/STG env after the INT/STG upgrade.
Nick, Yan Du, I just tried in PROD and I can see the error in logs. This is due to PassengerSpawnIPAddress is ignored in Passenger configuration and Passenger is now doing the request to 127.0.0.1 instead of the application IP. In PROD 127.0.0.1 will return connection refused whereas in DEVENV 127.0.0.1 will respond correctly. I'm tempted to remove the PassengerSpawnIPAddress entirely and replace it with a curl command which will serve the same as prespawn. Any thoughts?
A PR that should fix this problem: https://github.com/openshift/origin-server/pull/4440
Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/cba1d7e85b9167f86c3f1ac584b47c80d2ea95de Bug 984867 - Hardcode port 80 for PassengerPreStart
Checked on latest INT (devenv_4255), ths issue still can be reproduced. $ cat ruby/logs/error_log-20140122-000000-EST [Wed Jan 22 06:03:15 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c2,c656 [Wed Jan 22 06:03:15 2014] [notice] Digest: generating secret for digest authentication ... [Wed Jan 22 06:03:15 2014] [notice] Digest: done [Wed Jan 22 06:03:15 2014] [notice] Apache/2.2.22 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED) from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:145:in `<main>' /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED) from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `new' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:105:in `connect' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:86:in `socket' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:90:in `head_request' from /opt/rh/ruby193/root/usr/share/gems/gems/passenger-3.0.21/helper-scripts/prespawn:145:in `<main>'
Meng Bo: The problem should not be reproducible in PROD as the ports are now correct. See the description of this PR: https://github.com/openshift/origin-server/pull/4440 Is it possible to VERIFY this in PROD once this fix land there?
will retest the bug after the code merge to STG or PRO. Thanks.
Test on PROD and STG(devenv_stage_666) Issue still can be reproduced.
Yan Du the fix is not yet pushed into PROD, to verify it, you can rhc ssh into gear and cat ruby/etc/conf.d/openshift.conf file. You should see something like: PassengerPreStart http://127.11.102.1:80/ instead of: PassengerPreStart http://127.11.102.1:8080/
Test on INT(devenv_4363), seem issue still can be reproduced. [ruby19-111.int.rhcloud.com 52fb201403ef645237000001]\> cat ruby/etc/conf.d/openshift.conf | grep PassengerPreStart PassengerPreStart http://127.5.128.129:8080/
Michal, I am working with Troy on an updated passenger rpm that should resolve this issue. So, need not use the APP_DNS is openshift.conf since the dns may not be available when the app is prespawned by passenger.
Mrunal: Thanks for update, does the update also fix the problem with the 80 port?
Michal, I am not sure what the issue is with port 80. The rpm fix I have is replacing 127.0.0.1 by request_host in prespawn and that I think should fix the issue. I may have missed something?
The new passenger rpm should be available in latest devenv now. Could this be retested?
It's fixed, verified on devenv_4369, please refer to the results: 1. create a ruby-1.9 app rhc app create crb19s ruby-1.9 -s --no-git 2. ssh into app rhc app ssh crb19s 3. check the log [crb19s-cdm.dev.rhcloud.com 52fc646500361b4cf70000e6]\> cat ~/ruby/logs/error_log-20140213-000000-EST [Thu Feb 13 01:21:50 2014] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:openshift_t:s0:c0,c1003 [Thu Feb 13 01:21:51 2014] [notice] Digest: generating secret for digest authentication ... [Thu Feb 13 01:21:51 2014] [notice] Digest: done [Thu Feb 13 01:21:51 2014] [notice] Apache/2.2.15 (Unix) Phusion_Passenger/3.0.21 configured -- resuming normal operations [crb19s-cdm.dev.rhcloud.com 52fc646500361b4cf70000e6]\>