Description of problem: Running the template automation is intermittently hitting a "proxy: error reading status line". ==> httpd/error_log <== [Tue Nov 09 00:31:59 2010] [error] [client 10.11.231.193] (70014)End of file found: proxy: error reading status line from remote server localhost, referer: http://10.16.120.156/deltacloud/templates/new [Tue Nov 09 00:31:59 2010] [debug] mod_proxy_http.c(1456): [client 10.11.231.193] proxy: NOT Closing connection to client although reading from backend server localhost failed., referer: http://10.16.120.156/deltacloud/templates/new [Tue Nov 09 00:31:59 2010] [error] [client 10.11.231.193] proxy: Error reading from remote server returned by /deltacloud/templates, referer: http://10.16.120.156/deltacloud/templates/new [Tue Nov 09 00:31:59 2010] [debug] proxy_util.c(2029): proxy: HTTP: has released connection for (localhost) Still investigating...
please update this bug after the next official build.. thanks
also found @ provider account creation 10.16.120.31 - - [09/Feb/2011:12:14:19 -0500] "GET /deltacloud/javascripts/application.js?1297178292 HTTP/1.1" 200 6350 10.16.120.31 - - [09/Feb/2011:12:14:19 -0500] "GET /deltacloud/javascripts/jquery.ui-1.8.1/jquery-ui-1.8.1.custom.min.js?1297178292 HTTP/1.1" 200 31425 10.16.120.31 - - [09/Feb/2011:12:14:19 -0500] "GET /deltacloud/javascripts/modernizr-1.5.js?1297178292 HTTP/1.1" 200 32889 10.16.120.31 - - [09/Feb/2011:12:14:21 -0500] "POST /deltacloud/admin/provider_accounts HTTP/1.1" 302 123 10.16.120.31 - - [09/Feb/2011:12:14:48 -0500] "GET /deltacloud/admin/provider_accounts/7 HTTP/1.1" 502 530 [Wed Feb 09 12:14:11 2011] [error] [client 10.16.120.31] File does not exist: /var/www/html/favicon.ico [Wed Feb 09 12:14:12 2011] [error] [client 10.16.120.31] File does not exist: /var/www/html/favicon.ico [Wed Feb 09 12:14:49 2011] [error] [client 10.16.120.31] (20014)Internal error: proxy: error reading status line from remote server localhost:3000, referer: http://10.16.120.139/deltacloud/admin/provider_accounts/new [Wed Feb 09 12:14:49 2011] [error] [client 10.16.120.31] proxy: Error reading from remote server returned by /deltacloud/admin/provider_accounts/7, referer: http://10.16.120.139/deltacloud/admin/provider_accounts/new
STEPS TO REPRODUCE:: http://aeolus-fedora.virt.bos.redhat.com/deltacloud/admin/providers 1. create a provider w/ "name" and http://1.1.1.1:3002/api as the url
This bug will not reproduce if you are using a development machine.. apache must be proxying for rails. Recreated w/ using the released rpms + the rpms from http://aeolus-fedora.virt.bos.redhat.com/rpms/latest/ deltacloud-aggregator-0.0.2-0.fc14.20110209215937git325d592.x86_64.rpm 09-Feb-2011 17:03 1.4M deltacloud-aggregator-daemons-0.0.2-0.fc14.20110209215937git325d592.x86_64.rpm 09-Feb-2011 17:03 15K deltacloud-aggregator-doc-0.0.2-0.fc14.20110209215937git325d592.x86_64.rpm
<matty_dubs> weshay_hm: IMHO, it's two bugs... One is that the page should have a reasonable timeout. The other is more of a necessary new feature: the ability to handle more than one concurrent connection
some bugs were not listed as "phase 2"
Created attachment 480175 [details] new gem with proxy error fix Matt Warner sent QE the attached deltacloud-client.gem which contained some fixes for the proxy error. I noticed that after applying it, I cannot add a provider account and see the following exception in rails.log. I tried on both fedora and rhel, same results. Processing Admin::ProviderAccountsController#create (for 10.11.231.102 at 2011-02-22 11:42:11) [POST] Parameters: {"commit"=>"Add", "action"=>"create", "quota"=>{"maximum_running_instances"=>"unlimited"}, "controller"=>"admin/provider_accounts", "__map"=>{"set_selected_provider"=>{"method"=>"GET", "url"=>"/deltacloud/admin/provider_accounts/set_selected_provider"}}, "provider_account"=>{"x509_cert_pub_file"=>#<File:/tmp/RackMultipart20110222-5146-18sq092-0>, "label"=>"ec2acct", "x509_cert_priv_file"=>#<File:/tmp/RackMultipart20110222-5146-1kpl35-0>, "provider_id"=>"2", "username"=>"AKIAJJBQZYJW2LN26OJQ", "account_number"=>"3897-8764-2310", "password"=>"[FILTERED]"}} DeltaCloud::API::BackendError (): /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:371:in `handle_backend_error' /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:279:in `method_missing' /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:331:in `request' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:218:in `call' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:218:in `process_result' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:169:in `transmit' /usr/lib/ruby/1.8/net/http.rb:543:in `start' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:166:in `transmit' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute' /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:63:in `post' /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:328:in `send' /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:328:in `request' /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:275:in `method_missing' app/models/provider_account.rb:204:in `generate_auth_key' app/models/provider_account_observer.rb:9:in `after_create' /usr/lib/ruby/1.8/observer.rb:185:in `notify_observers' /usr/lib/ruby/1.8/observer.rb:184:in `each' /usr/lib/ruby/1.8/observer.rb:184:in `notify_observers' app/controllers/admin/provider_accounts_controller.rb:80:in `create' compass (0.10.4) lib/compass/app_integration/rails/actionpack2/action_controller.rb:7:in `process' haml (3.0.18) rails/./lib/sass/plugin/rack.rb:41:in `call' haml (3.0.18) rails/./lib/sass/plugin/rack.rb:41:in `call' thin (1.2.5) lib/thin/connection.rb:76:in `pre_process' thin (1.2.5) lib/thin/connection.rb:74:in `catch' thin (1.2.5) lib/thin/connection.rb:74:in `pre_process' thin (1.2.5) lib/thin/connection.rb:57:in `process' thin (1.2.5) lib/thin/connection.rb:42:in `receive_data' eventmachine (0.12.10) lib/eventmachine.rb:256:in `run_machine' eventmachine (0.12.10) lib/eventmachine.rb:256:in `run' thin (1.2.5) lib/thin/backends/base.rb:57:in `start' thin (1.2.5) lib/thin/server.rb:156:in `start' thin (1.2.5) lib/thin/controllers/controller.rb:80:in `start' thin (1.2.5) lib/thin/runner.rb:177:in `send' thin (1.2.5) lib/thin/runner.rb:177:in `run_command' thin (1.2.5) lib/thin/runner.rb:143:in `run!' thin (1.2.5) bin/thin:6 /usr/bin/thin:19:in `load' /usr/bin/thin:19 Rendering template within layouts/application Rendering layouts/error
fyi.. you can circumvent the proxy by starting the conductor w/ thin start -c /usr/share/aeolus-conductor -l /var/log/aeolus-conductor/thin.log -P /var/run/aeolus-conductor/thin.pid -a 0.0.0.0 -p 3000 -e production --user aeolus --group aeolus -d --prefix=/conductor -A rails
I hit this problem now too, unfortunately not able to reproduce it. Creating provider with url http://1.1.1.1:3002/api gives proxy error, but because of timeout -> proxy timeout should be increased. Anyway google suggests to use these options: keepalive Off SetEnv force-proxy-request-1.0 1 etEnv proxy-nokeepalive 1 But I can't say if this solves problem or not - I'm not able to reproduce bug now w/ or w/o above options.
Triaged 3/17
This has been fixed for a bit in deltacloud-client, but apparently not in a released RPM or in the version installed if you do a gem install. 0.3.0 is now an RC, and will hopefully be released shortly. I have a small patch to Conductor to gracefully handle the error: https://fedorahosted.org/pipermail/aeolus-devel/2011-April/001065.html This is not necessary to avoid the Apache errors, however; that's done with an earlier deltacloud-client patch.
*** Bug 643989 has been marked as a duplicate of this bug. ***
We've got rubygems-deltacloud-client in the testing repos now. This shouldn't happen anymore. (There's also a separate BZ for the one place this has happened recently.)
unfortunately this is still an issue.. but not one for the beta :(
Getting Proxy very often while adding/deleting provider .
Just hit this on the Aeolus 0.3.0 release rpms, on RHEL 6.1. (physical hardware, not a vm) Was about to add an EC2 provider account. Attaching the screenshot, plus apache's debug level error log for the request this occurred on. (ie not a huge file) Hope that's useful.
Created attachment 517967 [details] error log (in debug mode) for httpd, when this occurred Entries prior to the request that generated the error have been removed.
Created attachment 517970 [details] Screenshot of the proxy error, showing exact url request
Could you please attach thin log? (/var/log/aeolus-conductor/thin.log)
Sure. There's nothing much of interest in it, however there *is* an interesting error (with stack dump thing) at the end of rails.log. Attaching both now.
Created attachment 517972 [details] thin logfile
Created attachment 517973 [details] Rails log file
Created attachment 517975 [details] Deltacloud log with exception
Spent a while trying to debug this, no luck, though most often advice for "Internal error: proxy: error reading status line from remote server localhost" error is to disable keepalive connections. And I think it's worth to try it.
calling this fixed.. have not seen a proxy error in months.. will open if there is an issue.
closing out old bugs
perm close