Bug 651562

please update this bug after the next official build.. thanks

also found @  provider account creation


10.16.120.31 - - [09/Feb/2011:12:14:19 -0500] "GET /deltacloud/javascripts/application.js?1297178292 HTTP/1.1" 200 6350
10.16.120.31 - - [09/Feb/2011:12:14:19 -0500] "GET /deltacloud/javascripts/jquery.ui-1.8.1/jquery-ui-1.8.1.custom.min.js?1297178292 HTTP/1.1" 200 31425
10.16.120.31 - - [09/Feb/2011:12:14:19 -0500] "GET /deltacloud/javascripts/modernizr-1.5.js?1297178292 HTTP/1.1" 200 32889
10.16.120.31 - - [09/Feb/2011:12:14:21 -0500] "POST /deltacloud/admin/provider_accounts HTTP/1.1" 302 123
10.16.120.31 - - [09/Feb/2011:12:14:48 -0500] "GET /deltacloud/admin/provider_accounts/7 HTTP/1.1" 502 530



[Wed Feb 09 12:14:11 2011] [error] [client 10.16.120.31] File does not exist: /var/www/html/favicon.ico
[Wed Feb 09 12:14:12 2011] [error] [client 10.16.120.31] File does not exist: /var/www/html/favicon.ico
[Wed Feb 09 12:14:49 2011] [error] [client 10.16.120.31] (20014)Internal error: proxy: error reading status line from remote server localhost:3000, referer: http://10.16.120.139/deltacloud/admin/provider_accounts/new
[Wed Feb 09 12:14:49 2011] [error] [client 10.16.120.31] proxy: Error reading from remote server returned by /deltacloud/admin/provider_accounts/7, referer: http://10.16.120.139/deltacloud/admin/provider_accounts/new

STEPS TO REPRODUCE::

http://aeolus-fedora.virt.bos.redhat.com/deltacloud/admin/providers
1. create a provider w/ "name" and http://1.1.1.1:3002/api as the url

This bug will not reproduce if you are using a development machine.. apache must be proxying for rails.  

Recreated w/ using the released rpms  + the rpms from 
http://aeolus-fedora.virt.bos.redhat.com/rpms/latest/

	deltacloud-aggregator-0.0.2-0.fc14.20110209215937git325d592.x86_64.rpm	09-Feb-2011 17:03	1.4M	 
	deltacloud-aggregator-daemons-0.0.2-0.fc14.20110209215937git325d592.x86_64.rpm	09-Feb-2011 17:03	 15K	 
	deltacloud-aggregator-doc-0.0.2-0.fc14.20110209215937git325d592.x86_64.rpm

<matty_dubs> weshay_hm: IMHO, it's two bugs... One is that the page should have a reasonable timeout. The other is more of a necessary new feature: the ability to handle more than one concurrent connection

some bugs were not listed as "phase 2"

Created attachment 480175 [details]
new gem with proxy error fix

Matt Warner sent QE the attached deltacloud-client.gem which contained some fixes for the proxy error.  I noticed that after applying it, I cannot add a provider account and see the following exception in rails.log.  I tried on both fedora and rhel, same results.


Processing Admin::ProviderAccountsController#create (for 10.11.231.102 at 2011-02-22 11:42:11) [POST]
  Parameters: {"commit"=>"Add", "action"=>"create", "quota"=>{"maximum_running_instances"=>"unlimited"}, "controller"=>"admin/provider_accounts", "__map"=>{"set_selected_provider"=>{"method"=>"GET", "url"=>"/deltacloud/admin/provider_accounts/set_selected_provider"}}, "provider_account"=>{"x509_cert_pub_file"=>#<File:/tmp/RackMultipart20110222-5146-18sq092-0>, "label"=>"ec2acct", "x509_cert_priv_file"=>#<File:/tmp/RackMultipart20110222-5146-1kpl35-0>, "provider_id"=>"2", "username"=>"AKIAJJBQZYJW2LN26OJQ", "account_number"=>"3897-8764-2310", "password"=>"[FILTERED]"}}

DeltaCloud::API::BackendError ():
  /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:371:in `handle_backend_error'
  /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:279:in `method_missing'
  /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:331:in `request'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:218:in `call'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:218:in `process_result'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:169:in `transmit'
  /usr/lib/ruby/1.8/net/http.rb:543:in `start'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:166:in `transmit'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute'
  /usr/lib/ruby/gems/1.8/gems/rest-client-1.6.1/lib/restclient/resource.rb:63:in `post'
  /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:328:in `send'
  /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:328:in `request'
  /usr/lib/ruby/gems/1.8/gems/deltacloud-client-0.2.0/lib/deltacloud.rb:275:in `method_missing'
  app/models/provider_account.rb:204:in `generate_auth_key'
  app/models/provider_account_observer.rb:9:in `after_create'
  /usr/lib/ruby/1.8/observer.rb:185:in `notify_observers'
  /usr/lib/ruby/1.8/observer.rb:184:in `each'
  /usr/lib/ruby/1.8/observer.rb:184:in `notify_observers'
  app/controllers/admin/provider_accounts_controller.rb:80:in `create'
  compass (0.10.4) lib/compass/app_integration/rails/actionpack2/action_controller.rb:7:in `process'
  haml (3.0.18) rails/./lib/sass/plugin/rack.rb:41:in `call'
  haml (3.0.18) rails/./lib/sass/plugin/rack.rb:41:in `call'
  thin (1.2.5) lib/thin/connection.rb:76:in `pre_process'
  thin (1.2.5) lib/thin/connection.rb:74:in `catch'
  thin (1.2.5) lib/thin/connection.rb:74:in `pre_process'
  thin (1.2.5) lib/thin/connection.rb:57:in `process'
  thin (1.2.5) lib/thin/connection.rb:42:in `receive_data'
  eventmachine (0.12.10) lib/eventmachine.rb:256:in `run_machine'
  eventmachine (0.12.10) lib/eventmachine.rb:256:in `run'
  thin (1.2.5) lib/thin/backends/base.rb:57:in `start'
  thin (1.2.5) lib/thin/server.rb:156:in `start'
  thin (1.2.5) lib/thin/controllers/controller.rb:80:in `start'
  thin (1.2.5) lib/thin/runner.rb:177:in `send'
  thin (1.2.5) lib/thin/runner.rb:177:in `run_command'
  thin (1.2.5) lib/thin/runner.rb:143:in `run!'
  thin (1.2.5) bin/thin:6
  /usr/bin/thin:19:in `load'
  /usr/bin/thin:19

Rendering template within layouts/application
Rendering layouts/error

fyi.. you can circumvent the proxy by starting the conductor w/ 

thin start -c /usr/share/aeolus-conductor -l /var/log/aeolus-conductor/thin.log -P /var/run/aeolus-conductor/thin.pid -a 0.0.0.0 -p 3000 -e production --user aeolus --group aeolus  -d --prefix=/conductor -A rails

I hit this problem now too, unfortunately not able to reproduce it.  Creating provider with url http://1.1.1.1:3002/api gives proxy error, but because of timeout -> proxy timeout should be increased.

Anyway google suggests to use these options:
keepalive Off
SetEnv force-proxy-request-1.0 1
etEnv proxy-nokeepalive 1

But I can't say if this solves problem or not - I'm not able to reproduce bug now w/ or w/o above options.

Triaged 3/17

This has been fixed for a bit in deltacloud-client, but apparently not in a released RPM or in the version installed if you do a gem install. 0.3.0 is now an RC, and will hopefully be released shortly.

I have a small patch to Conductor to gracefully handle the error: https://fedorahosted.org/pipermail/aeolus-devel/2011-April/001065.html

This is not necessary to avoid the Apache errors, however; that's done with an earlier deltacloud-client patch.

*** Bug 643989 has been marked as a duplicate of this bug. ***

We've got rubygems-deltacloud-client in the testing repos now. This shouldn't happen anymore. (There's also a separate BZ for the one place this has happened recently.)

unfortunately this is still an issue.. but not one for the beta :(

Getting Proxy very often while adding/deleting provider .

Just hit this on the Aeolus 0.3.0 release rpms, on RHEL 6.1.  (physical hardware, not a vm)

Was about to add an EC2 provider account.

Attaching the screenshot, plus apache's debug level error log for the request this occurred on. (ie not a huge file)

Hope that's useful.

Created attachment 517967 [details]
error log (in debug mode) for httpd, when this occurred

Entries prior to the request that generated the error have been removed.

Created attachment 517970 [details]
Screenshot of the proxy error, showing exact url request

Could you please attach thin log? (/var/log/aeolus-conductor/thin.log)

Sure.  There's nothing much of interest in it, however there *is* an interesting error (with stack dump thing) at the end of rails.log.

Attaching both now.

Created attachment 517972 [details]
thin logfile

Created attachment 517973 [details]
Rails log file

Created attachment 517975 [details]
Deltacloud log with exception

Spent a while trying to debug this, no luck, though most often advice for
"Internal error: proxy: error reading status line from remote server localhost"
error is to disable keepalive connections. And I think it's worth to try it.

calling this fixed.. have not seen a proxy error in months.. will open if there is an issue.

closing out old bugs

perm close