Bug 788392

Summary:

aeolus-conductor goes dead when trying to delete running deployment.

Product:

[Retired] CloudForms Cloud Engine

Reporter:

Aziza Karol <akarol>

Component:

aeolus-conductor

Assignee:

Angus Thomas <athomas>

Status:

CLOSED DUPLICATE

QA Contact:

wes hayutin <whayutin>

Severity:

high

Docs Contact:

Priority:

unspecified

Version:

1.0.0

CC:

akarol, deltacloud-maint, jprovazn, ssachdev

Target Milestone:

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2012-02-08 13:30:39 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Attachments:

Description	Flags
proxy err	none

Description Aziza Karol 2012-02-08 07:21:36 UTC

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
I launched ec2 deployment and was trying to delete the running deployment.

Proxy error displayed in UI. see attached screenshot.

aeolus-conductor goes dead.
# service aeolus-conductor status
thin dead but pid file exists


deltacloud-core.log displays:
thin server (localhost:3002) [deltacloud-mock][17945]: 127.0.0.1 - - [08/Feb/2012 02:17:57] "DELETE /api/keys/ec2_ec2-frontend_1328685246_key_69966836872220 HTTP/1.1" 204 - 0.3802
I, [2012-02-08T02:18:48.107936 #17945]  INFO -- : New Aws::Ec2 using per_thread-connection mode
W, [2012-02-08T02:18:48.111679 #17945]  WARN -- : Rightscale::HttpConnection : request failure count: 1, exception: #<Errno::EPIPE: Broken pipe>
I, [2012-02-08T02:18:48.111967 #17945]  INFO -- : Opening new HTTPS connection to ec2.us-east-1.amazonaws.com:443
I, [2012-02-08T02:18:49.335414 #17945]  INFO -- : New Aws::Ec2 using per_thread-connection mode
W, [2012-02-08T02:18:49.498678 #17945]  WARN -- : ##### Aws::Ec2 returned an error: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidKeyPair.NotFound</Code><Message>The key pair 'ec2_ec2-frontend_1328685246_key_69966836872220' does not exist</Message></Error></Errors><RequestID>4d7f438b-518c-492c-89ed-d769a38f6aab</RequestID></Response> #####
W, [2012-02-08T02:18:49.499053 #17945]  WARN -- : ##### Aws::Ec2 request: ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAJRCPNVJYQBHCOBEA&Action=DescribeKeyPairs&KeyName.1=ec2_ec2-frontend_1328685246_key_69966836872220&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2012-02-08T07%3A18%3A49.000Z&Version=2010-08-31&Signature=wH6rmlDogcWuJ%2F0o1ywDJe5qGEEEyS6%2BeOwNDMkb5DU%3D ####
thin server (localhost:3002) [deltacloud-mock][17945]: Aws::AwsError:InvalidKeyPair.NotFound: The key pair 'ec2_ec2-frontend_1328685246_key_69966836872220' does not exist
REQUEST=ec2.us-east-1.amazonaws.com:443/?AWSAccessKeyId=AKIAJRCPNVJYQBHCOBEA&Action=DescribeKeyPairs&KeyName.1=ec2_ec2-frontend_1328685246_key_69966836872220&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2012-02-08T07%3A18%3A49.000Z&Version=2010-08-31&Signature=wH6rmlDogcWuJ%2F0o1ywDJe5qGEEEyS6%2BeOwNDMkb5DU%3D 
REQUEST ID=4d7f438b-518c-492c-89ed-d769a38f6aab 
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ses/../awsbase/awsbase.rb:572:in `request_info_impl'
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ec2/ec2.rb:177:in `request_info'
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ses/../awsbase/awsbase.rb:586:in `request_cache_or_info'
/usr/lib/ruby/gems/1.8/gems/aws-2.5.5/lib/ec2/ec2.rb:998:in `describe_key_pairs'
/usr/share/deltacloud-core/lib/deltacloud/drivers/ec2/ec2_driver.rb:283:in `keys'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `call'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/exceptions.rb:151:in `safely'
/usr/share/deltacloud-core/lib/deltacloud/drivers/ec2/ec2_driver.rb:282:in `keys'
/usr/share/deltacloud-core/lib/deltacloud/base_driver/base_driver.rb:193:in `key'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `send'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:93:in `show'
/usr/lib/ruby/1.8/benchmark.rb:293:in `measure'
/usr/share/deltacloud-core/lib/deltacloud/helpers/application_helper.rb:92:in `show'
/usr/share/deltacloud-core/lib/deltacloud/server.rb:786
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `instance_eval'
/usr/share/deltacloud-core/lib/sinatra/rabbit.rb:125:in `GET /api/keys/:id'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in `compile!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in `route_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in `process_route'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `route!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in `dispatch!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `catch'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in `invoke'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in `call!'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_syslog.rb:48:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_date.rb:31:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_accept.rb:149:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_driver_select.rb:45:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_matrix_params.rb:106:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_runtime.rb:36:in `call'
/usr/share/deltacloud-core/lib/sinatra/rack_etag.rb:41:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-accept-0.4.3/lib/rack/accept/context.rb:22:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/head.rb:9:in `call'
/usr/lib/ruby/gems/1.8/gems/rack-1.3.0/lib/rack/methodoverride.rb:24:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in `call'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:84:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `catch'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:82:in `pre_process'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `call'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1060:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `initialize'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `new'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1057:in `spawn_threadpool'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:1049:in `defer'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:54:in `process'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/connection.rb:42:in `receive_data'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
/usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `send'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
/usr/lib/ruby/gems/1.8/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
/usr/share/deltacloud-core/bin/deltacloudd:235
/usr/bin/deltacloudd:5:in `load'
/usr/bin/deltacloudd:5
thin server (localhost:3002) [deltacloud-mock][17945]: 127.0.0.1 - - [08/Feb/2012 02:18:49] "GET /api/keys/ec2_ec2-frontend_1328685246_key_69966836872220 HTTP/1.1" 404 133 0.2122




Additional info:
rpm -qa |egrep 'aeolus|deltacloud-core'
aeolus-conductor-0.8.0-23.el6.noarch
rubygem-aeolus-cli-0.3.0-8.el6.noarch
aeolus-configure-2.5.0-12.el6.noarch
deltacloud-core-rhevm-0.5.0-5.el6.noarch
deltacloud-core-0.5.0-5.el6.noarch
rubygem-aeolus-image-0.3.0-7.el6.noarch
deltacloud-core-ec2-0.5.0-5.el6.noarch
aeolus-all-0.8.0-23.el6.noarch
deltacloud-core-vsphere-0.5.0-5.el6.noarch
aeolus-conductor-doc-0.8.0-23.el6.noarch
aeolus-conductor-daemons-0.8.0-23.el6.noarch

Comment 1 Aziza Karol 2012-02-08 07:25:40 UTC

Created attachment 560167 [details]
proxy err

Comment 2 Jan Provaznik 2012-02-08 12:42:54 UTC

From the description above it looks like a thin segfault. AFAIK Jim was working on a patch related to these segfaults.

Comment 3 wes hayutin 2012-02-08 13:30:39 UTC

Aziza.. nice catch! Glad to see other people are hitting this.. it is a dupe though.

*** This bug has been marked as a duplicate of bug 786432 ***