699831 – Templates, building templates in us-west fails, us-east works

Bug 699831 - Templates, building templates in us-west fails, us-east works

Summary: Templates, building templates in us-west fails, us-east works

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	CloudForms Cloud Engine
Classification:	Retired
Component:	aeolus-conductor
Sub Component:
Version:	0.3.1
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Assignee:	Mike Orazi
QA Contact:	wes hayutin
Docs Contact:
URL:	https://hp-xw8600-01.rhts.eng.bos.red...
Whiteboard:
Depends On:
Blocks:	ce-beta ce-ami
TreeView+	depends on / blocked

Reported:	2011-04-26 17:43 UTC by wes hayutin
Modified:	2012-01-26 12:24 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Clone Of:
Environment:
Last Closed:	2011-04-26 17:50:09 UTC
Embargoed:

Attachments	(Terms of Use)
imagefactory log (126.59 KB, text/plain) 2011-04-26 17:43 UTC, wes hayutin	no flags	Details
View All

Description wes hayutin 2011-04-26 17:43:15 UTC

Created attachment 495002 [details]
imagefactory log

Description of problem:

recreate:
1. add a us-east provider account
2. add a us-west provider account
3. create a template, add packages (Java, web-server, in my test)
4. build for ec2

notice the build or us-east is successfull where the build for us-west fails due to ssh access issues.

see attached log file

Also unable to ssh into jeos ami in us-west using key in /etc/oz

Comment 1 wes hayutin 2011-04-26 17:50:09 UTC

was default security group in us-west


2011-04-26 13:48:00,011 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Waiting up to 300 seconds for ssh to become available on ec2-50-18-14-26.us-west-1.compute.amazonaws.com
2011-04-26 13:48:00,012 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Waiting for EC2 ssh access: 0/300
2011-04-26 13:48:14,864 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Waiting for EC2 ssh access: 10/300
2011-04-26 13:48:26,527 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Waiting for EC2 ssh access: 20/300
2011-04-26 13:48:28,056 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Waiting 20 seconds for remaining boot tasks
2011-04-26 13:48:48,063 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Customizing guest: ec2-50-18-14-26.us-west-1.compute.amazonaws.com
2011-04-26 13:48:48,064 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Removing utility package and repo
2011-04-26 13:48:53,481 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(21666) Message: Removal complete
2011-04-26 13:48:53,481 DEBUG oz.Guest.FedoraRemoteGuest pid(21666) Message: Installing additional repository files
2011-04-26 13:48:53,481 DEBUG oz.Guest.FedoraRemoteGuest pid(21666) Message: Installing custom packages

Comment 2 Shveta 2011-05-02 09:56:30 UTC

Uploaded image as imagefactory-ec2-us-east-1-8571-4451-8821/28b17710-af37-49bc-bf41-554d53cd4791.manifest.xml

2011-05-02 05:46:32,350 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Executing register command: euca-register -U http://ec2.us-east-1.amazonaws.com/ -A "AKIAI2KPFDYVZKSRTJMQ" -S "jbHDz9e+2EghTuPd/bSmZ9xbnTWlpjwrZc87uEF5" imagefactory-ec2-us-east-1-8571-4451-8821/28b17710-af37-49bc-bf41-554d53cd4791.manifest.xml
2011-05-02 05:46:33,331 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Register output: IMAGE	ami-c2e31cab

2011-05-02 05:46:33,332 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Extracted AMI ID: ami-c2e31cab 
2011-05-02 05:46:33,334 WARNING imagefactory.ImageWarehouse.ImageWarehouse pid(10061) Message: Creating a bucket returned status 500.  If only iwhd would provide a sane way to know if a bucket exists so we wouldn't have to try and create one every time...
2011-05-02 05:46:33,337 DEBUG imagefactory.ImageWarehouse.ImageWarehouse pid(10061) Message: Setting metadata ({'provider': 'ec2-us-east-1', 'uuid': '28b17710-af37-49bc-bf41-554d53cd4791', 'icicle': 'none', 'target_identifier': 'ami-c2e31cab', 'object_type': 'provider_image', 'image': '947960cd-84e9-4a70-a4ac-deaaed6c806c'}) for http://localhost:9090/provider_images/28b17710-af37-49bc-bf41-554d53cd4791
2011-05-02 05:46:33,582 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Stopping EC2 instance
2011-05-02 05:46:33,583 DEBUG boto pid(10061) Message: using calc_signature_2
2011-05-02 05:46:33,583 DEBUG boto pid(10061) Message: query string: AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=TerminateInstances&InstanceId.1=i-903203ff&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-05-02T09%3A46%3A33&Version=2009-11-30
2011-05-02 05:46:33,583 DEBUG boto pid(10061) Message: string_to_sign: GET
ec2.us-east-1.amazonaws.com
/
AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=TerminateInstances&InstanceId.1=i-903203ff&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-05-02T09%3A46%3A33&Version=2009-11-30
2011-05-02 05:46:33,584 DEBUG boto pid(10061) Message: len(b64)=44
2011-05-02 05:46:33,584 DEBUG boto pid(10061) Message: base64 encoded digest: +JEt3u1UCNgWcqZgrx7BaIpn4ieHp8QUuv5ApzhPPd4=
2011-05-02 05:46:33,584 DEBUG boto pid(10061) Message: Canonical: GET


Mon, 02 May 2011 09:46:33 GMT
/
2011-05-02 05:46:33,584 DEBUG boto pid(10061) Message: Method: GET
2011-05-02 05:46:33,585 DEBUG boto pid(10061) Message: Path: /?AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=TerminateInstances&InstanceId.1=i-903203ff&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-05-02T09%3A46%3A33&Version=2009-11-30&Signature=%2BJEt3u1UCNgWcqZgrx7BaIpn4ieHp8QUuv5ApzhPPd4%3D
2011-05-02 05:46:33,585 DEBUG boto pid(10061) Message: Data: 
2011-05-02 05:46:33,585 DEBUG boto pid(10061) Message: Headers: {'Date': 'Mon, 02 May 2011 09:46:33 GMT', 'Content-Length': '0', 'Authorization': 'AWS AKIAI2KPFDYVZKSRTJMQ:4koyPX0DuWGAP3WU0ZSMfamJNYc=', 'User-Agent': 'Boto/1.9b (linux2)'}
2011-05-02 05:46:33,585 DEBUG boto pid(10061) Message: Host: None
2011-05-02 05:46:33,586 DEBUG boto pid(10061) Message: encountered BadStatusLine exception, reconnecting
2011-05-02 05:46:33,586 DEBUG boto pid(10061) Message: establishing HTTP connection
2011-05-02 05:46:34,693 DEBUG boto pid(10061) Message: <?xml version="1.0" encoding="UTF-8"?>
<TerminateInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2009-11-30/">
    <requestId>4880d9af-7fa8-4494-8818-718a141519a0</requestId>
    <instancesSet>
        <item>
            <instanceId>i-903203ff</instanceId>
            <currentState>
                <code>32</code>
                <name>shutting-down</name>
            </currentState>
            <previousState>
                <code>16</code>
                <name>running</name>
            </previousState>
        </item>
    </instancesSet>
</TerminateInstancesResponse>
2011-05-02 05:46:34,695 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: FedoraBuilder instance 44951376 pushed image with uuid 947960cd-84e9-4a70-a4ac-deaaed6c806c to provider_image UUID (28b17710-af37-49bc-bf41-554d53cd4791) and set metadata: {'target_identifier': 'ami-c2e31cab', 'icicle': 'none', 'image': '947960cd-84e9-4a70-a4ac-deaaed6c806c', 'provider': 'ec2-us-east-1'}
2011-05-02 05:46:34,695 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(10061) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed percent complete from 0 to 100
2011-05-02 05:46:34,696 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(10061) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from PUSHING to COMPLETED
2011-05-02 05:46:41,212 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Waiting for EC2 ssh access: 270/300
2011-05-02 05:46:56,249 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Waiting for EC2 ssh access: 280/300
2011-05-02 05:47:11,283 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Waiting for EC2 ssh access: 290/300
2011-05-02 05:47:26,318 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Stopping EC2 instance
2011-05-02 05:47:26,319 DEBUG boto pid(10061) Message: using calc_signature_2
2011-05-02 05:47:26,319 DEBUG boto pid(10061) Message: query string: AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=TerminateInstances&InstanceId.1=i-c4ae8680&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-05-02T09%3A47%3A26&Version=2009-11-30
2011-05-02 05:47:26,319 DEBUG boto pid(10061) Message: string_to_sign: GET
ec2.us-west-1.amazonaws.com
/
AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=TerminateInstances&InstanceId.1=i-c4ae8680&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-05-02T09%3A47%3A26&Version=2009-11-30
2011-05-02 05:47:26,320 DEBUG boto pid(10061) Message: len(b64)=44
2011-05-02 05:47:26,320 DEBUG boto pid(10061) Message: base64 encoded digest: onW/lcmJ4psF+I0R2fx+EbT0kch2HA9o3Cp/7F1J754=
2011-05-02 05:47:26,320 DEBUG boto pid(10061) Message: Canonical: GET


Mon, 02 May 2011 09:47:26 GMT
/
2011-05-02 05:47:26,321 DEBUG boto pid(10061) Message: Method: GET
2011-05-02 05:47:26,334 DEBUG boto pid(10061) Message: Path: /?AWSAccessKeyId=AKIAI2KPFDYVZKSRTJMQ&Action=TerminateInstances&InstanceId.1=i-c4ae8680&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2011-05-02T09%3A47%3A26&Version=2009-11-30&Signature=onW/lcmJ4psF%2BI0R2fx%2BEbT0kch2HA9o3Cp/7F1J754%3D
2011-05-02 05:47:26,334 DEBUG boto pid(10061) Message: Data: 
2011-05-02 05:47:26,334 DEBUG boto pid(10061) Message: Headers: {'Date': 'Mon, 02 May 2011 09:47:26 GMT', 'Content-Length': '0', 'Authorization': 'AWS AKIAI2KPFDYVZKSRTJMQ:t2zRZ05YSazVXVMUURNRzqCO6zE=', 'User-Agent': 'Boto/1.9b (linux2)'}
2011-05-02 05:47:26,334 DEBUG boto pid(10061) Message: Host: None
2011-05-02 05:47:26,335 DEBUG boto pid(10061) Message: encountered BadStatusLine exception, reconnecting
2011-05-02 05:47:26,336 DEBUG boto pid(10061) Message: establishing HTTP connection
2011-05-02 05:47:27,936 DEBUG boto pid(10061) Message: <?xml version="1.0" encoding="UTF-8"?>
<TerminateInstancesResponse xmlns="http://ec2.amazonaws.com/doc/2009-11-30/">
    <requestId>00e6e648-5264-4b36-8f5a-835348851d62</requestId>
    <instancesSet>
        <item>
            <instanceId>i-c4ae8680</instanceId>
            <currentState>
                <code>48</code>
                <name>terminated</name>
            </currentState>
            <previousState>
                <code>48</code>
                <name>terminated</name>
            </previousState>
        </item>
    </instancesSet>
</TerminateInstancesResponse>
2011-05-02 05:47:27,937 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Exception during push_image
2011-05-02 05:47:27,937 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message: Unexpected error: (<class 'imagefactory.ImageFactoryException.ImageFactoryException'>)
2011-05-02 05:47:27,938 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message:              value: (Unable to gain ssh access after 300 seconds - aborting)
2011-05-02 05:47:27,939 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(10061) Message:          traceback: ['  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 409, in push_image\n    self.push_image_snapshot(image_id, provider, credentials)\n', '  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 562, in push_image_snapshot\n    raise ImageFactoryException("Unable to gain ssh access after 300 seconds - aborting")\n']
2011-05-02 05:47:27,939 DEBUG imagefactory.qmfagent.BuildAdaptor.BuildAdaptor pid(10061) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from PUSHING to FAILED


==================================================

Created a provider account for ec2-us-west-1 .
Created a realm us-west mapped to ec2-us-west-1 and us-west-1a
build a template which succeeds in us-east and fails in us-west , with the above error in imagefactory.log

Comment 3 Shveta 2011-05-02 10:34:41 UTC

 On ami-6e807f07

rpm -qa|grep aeolus
aeolus-conductor-0.2.0-3.el6.x86_64
aeolus-configure-2.0.0-10.el6.noarch
aeolus-conductor-doc-0.2.0-3.el6.x86_64
aeolus-conductor-daemons-0.2.0-3.el6.x86_64

Comment 4 wes hayutin 2011-05-02 11:11:20 UTC

fix is here..

/etc/init.d/deltacloud-ec2-us-west-1

  $PROG -i $DRIVER -e $ENV -p $PORT --provider us-west-1 >> $LOGFILE 2>&1 &


morazi added --provider us-west-1

Comment 5 wes hayutin 2011-05-02 12:21:10 UTC

modify /usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py

Line 585...
This:
----------------------
self.guest.guest_execute_command(guestaddr,
"[ -f /etc/init.d/firstboot ] && /sbin/chkconfig firstboot off")

To:
----------------------
self.guest.guest_execute_command(guestaddr,
"[ -f /etc/init.d/firstboot ] && /sbin/chkconfig firstboot off
|| /bin/true")

then run clean up /configure

This should be in the next ami build

Comment 6 wes hayutin 2011-08-01 19:58:48 UTC

release pending...

Comment 7 wes hayutin 2011-08-01 19:59:14 UTC

release pending.. 2

Comment 8 wes hayutin 2011-08-01 19:59:40 UTC

release pending...

Comment 10 wes hayutin 2011-12-08 13:56:19 UTC

perm close

Comment 11 wes hayutin 2011-12-08 13:59:26 UTC

closing out old bugs

Note You need to log in before you can comment on or make changes to this bug.