Bug 746027 - Push to EC2 timing out, specifically ssh connection to instance
Summary: Push to EC2 timing out, specifically ssh connection to instance
Keywords:
Status: CLOSED DUPLICATE of bug 745731
Alias: None
Product: CloudForms Cloud Engine
Classification: Retired
Component: imagefactory
Version: 1.0.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
Assignee: jrd
QA Contact: wes hayutin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2011-10-13 17:19 UTC by Steve Reichard
Modified: 2013-02-27 04:26 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-10-13 17:51:15 UTC


Attachments (Terms of Use)

Description Steve Reichard 2011-10-13 17:19:15 UTC
Description of problem:

While attempting to push to EC2 a error occurred reporting that connection timed out.  



from imagefactory.log
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: nicmodel: virtio, clockoffset: utc
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: mousetype: ps2, disk_bus: virtio, disk_dev: vda
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: icicletmp: /var/lib/oz/icicletmp/rhel6_1_x86_64-web-bbb7c8fd-91a8-49be-8db9-033815d53ca8, listen_port: 45741
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: Original ISO path: /var/lib/oz/isos/RHEL-61x86_64-url.iso
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: Modified ISO cache: /var/lib/oz/isos/RHEL-61x86_64-url-oz.iso
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: Output ISO path: /var/lib/imagefactory/images/rhel6_1_x86_64-web-bbb7c8fd-91a8-49be-8db9-033815d53ca8-url-oz.iso
2011-10-13 10:26:09,138 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: ISO content path: /var/lib/oz/isocontent/rhel6_1_x86_64-web-bbb7c8fd-91a8-49be-8db9-033815d53ca8-url
2011-10-13 10:26:09,139 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Being asked to push for provider ec2-us-east-1
2011-10-13 10:26:09,139 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: distro: RHEL-6 - update: 1 - arch: x86_64
2011-10-13 10:26:09,140 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: acting as EC2 user: 7785-6245-6303
2011-10-13 10:26:09,140 DEBUG imgfac.BuildJob.BuildAdaptor pid(9011) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from NEW to PUSHING
2011-10-13 10:26:09,141 DEBUG imgfac.BuildJob.BuildAdaptor pid(9011) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed percent complete from 0 to 0
2011-10-13 10:26:09,141 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Starting ami ami-31d41658 with instance_type m1.large
2011-10-13 10:26:31,436 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Creating temporary security group (imagefactory-bbb7c8fd-91a8-49be-8db9-033815d53ca8)
2011-10-13 10:26:56,461 WARNING boto pid(9011) Message: Error converting code (pending) to int
2011-10-13 10:27:06,463 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Waiting for EC2 instance to start: 0/300
2011-10-13 10:27:17,389 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Waiting for EC2 instance to start: 10/300
2011-10-13 10:27:18,573 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Waiting for EC2 ssh access: 0/300
2011-10-13 10:28:15,622 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Waiting 20 seconds for remaining boot tasks
2011-10-13 10:28:35,642 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Stopping cron and killing any updatedb process that may be running
2011-10-13 10:28:37,446 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Done
2011-10-13 10:29:19,033 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Customizing guest: ec2-107-22-15-144.compute-1.amazonaws.com
2011-10-13 10:29:19,034 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: Installing additional repository files
2011-10-13 10:29:19,034 DEBUG oz.Guest.RHEL6RemoteGuest pid(9011) Message: Installing custom packages
2011-10-13 10:29:29,070 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Terminating EC2 instance and deleting temp security group
2011-10-13 10:29:29,427 WARNING boto pid(9011) Message: Error converting code (Client.UserInitiatedShutdown) to int
2011-10-13 10:29:29,428 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Instance status (shutting-down) - waiting for 'terminated'. [0 of 300 seconds elapsed]
2011-10-13 10:29:34,523 WARNING boto pid(9011) Message: Error converting code (Client.UserInitiatedShutdown) to int
2011-10-13 10:29:34,523 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Instance status (shutting-down) - waiting for 'terminated'. [5 of 300 seconds elapsed]
2011-10-13 10:29:39,613 WARNING boto pid(9011) Message: Error converting code (Client.UserInitiatedShutdown) to int
2011-10-13 10:29:39,614 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Instance status (shutting-down) - waiting for 'terminated'. [10 of 300 seconds elapsed]
2011-10-13 10:29:44,740 WARNING boto pid(9011) Message: Error converting code (Client.UserInitiatedShutdown) to int
2011-10-13 10:29:44,740 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Instance status (shutting-down) - waiting for 'terminated'. [15 of 300 seconds elapsed]
2011-10-13 10:29:49,829 WARNING boto pid(9011) Message: Error converting code (Client.UserInitiatedShutdown) to int
2011-10-13 10:29:49,830 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Instance status (shutting-down) - waiting for 'terminated'. [20 of 300 seconds elapsed]
2011-10-13 10:29:55,109 WARNING boto pid(9011) Message: Error converting code (Client.UserInitiatedShutdown) to int
2011-10-13 10:29:55,419 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Removed temporary security group (imagefactory-bbb7c8fd-91a8-49be-8db9-033815d53ca8)
2011-10-13 10:29:55,420 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Exception caught in ImageFactory
2011-10-13 10:29:55,425 DEBUG imgfac.builders.BaseBuilder.RHEL6_ec2_Builder pid(9011) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 432, in push_image
    credentials)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 605, in push_image_snapshot_ec2
    self.guest.do_customize(guestaddr)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 969, in do_customize
    tunnels=self.tunnels)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 469, in guest_execute_command
    command, timeout, tunnels)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 342, in ssh_execute_command
    return subprocess_check_output(cmd)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 310, in subprocess_check_output
    raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode)
SubprocessException: 'ssh -i /tmp/tmpdAWRob -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root.amazonaws.com yum -y install httpd php ' failed(255): ssh: connect to host ec2-107-22-15-144.compute-1.amazonaws.com port 22: Connection timed out


2011-10-13 10:29:55,425 DEBUG imgfac.BuildJob.BuildAdaptor pid(9011) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from PUSHING to FAILED


Version-Release number of selected component (if applicable):

[root@cf-cloudforms9 oz]# /pub/scripts/post_install_configuration_scripts/cf-versions 
Red Hat Enterprise Linux Server release 6.1 (Santiago)
Linux cf-cloudforms9.cloud.lab.eng.bos.redhat.com 2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Sep 29 10:24:25 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
postgresql-8.4.7-2.el6.x86_64
mongodb-1.8.0-6.el6.x86_64
euca2ools-1.3.1-4.el6_0.noarch
package gsoap is not installed
ruby-1.8.7.299-7.el6_1.1.x86_64
rubygems-1.8.10-1.el6.noarch
deltacloud-core-0.5.0-0.2.1317167702git.el6.noarch
rubygem-deltacloud-client-0.4.0-3.el6.noarch
package libdeltacloud is not installed
hail-0.8-0.2.gf9c5b967.el6_0.x86_64
puppet-2.6.6-1.el6_0.noarch
aeolus-configure-2.1.0-1.el6.noarch
package condor is not installed
iwhd-0.98.15.z3-2.el6.x86_64
imagefactory-0.7.0-1.el6.noarch
aeolus-conductor-daemons-0.4.0-1.el6.noarch
aeolus-conductor-0.4.0-1.el6.noarch
[root@cf-cloudforms9 oz]# 


How reproducible:

I was 2 for 2 in reproducing then heard others are seeing this.

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 2 wes hayutin 2011-10-13 17:51:15 UTC

*** This bug has been marked as a duplicate of bug 745731 ***

Comment 3 wes hayutin 2011-11-28 01:21:49 UTC
removing bugs from tracker as part of 0.6 release


Note You need to log in before you can comment on or make changes to this bug.