| Summary: | Push fails for ec2, specifically when ssh'ing into instance | ||
|---|---|---|---|
| Product: | [Retired] CloudForms Cloud Engine | Reporter: | Aziza Karol <akarol> |
| Component: | rubygem-aeolus-image | Assignee: | Ian McLeod <imcleod> |
| Status: | CLOSED ERRATA | QA Contact: | wes hayutin <whayutin> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 1.0.0 | CC: | akarol, cpelland, dajohnso, deltacloud-maint, dgao, jrd, ricardo.arguello, sreichar, ssachdev |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2012-05-15 19:44:00 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
*** Bug 746027 has been marked as a duplicate of this bug. *** higher priority bug here.. looking for status This seems to be a result of our 10 second default SSH timeout. This has worked well in the past but it seems we are getting more and more of these particular types of errors. (Suggesting, perhaps, that EC2 and us-east in particular is getting more congested.) I will increase the default timeout for most SSH commands to 30 seconds for 0.8.0. if 0.8.0 is great then a week from release.. we'll need to back port this fix into the current release in aeolus-testing. Increased timeout to 30 seconds for all SSH commands other than the SSH ping during instance startup. This is: https://github.com/aeolusproject/imagefactory/commit/4b47426d47e60b554966a1e34ef1ad90e07a226f This will be part of 0.8.0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2012-0592.html |
Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Build and push for ec2 2. aeolus-image push --provider ec2-us-east-1 --id 6f46c5a8-8d79-4be1-ab5d-a0918354b9d7 Imgaefactory log: 2011-10-13 04:38:36,788 DEBUG imgfac.qmfagent.ImageFactoryAgent.ImageFactoryAgent pid(14699) Message: Method called: name = push_image args = {'credentials': '*** REDACTED ***', 'image': '6f46c5a8-8d79-4be1-ab5d-a0918354b9d7', 'build': '', 'providers': ['ec2-us-east-1']} handle = <cqmf2.AgentEvent; proxy of <Swig Object of type 'qmf::AgentEvent *' at 0x27915d0> > addr = redhat.com:imagefactory:8e10c7fa-dc8e-48c3-b330-99b69d67f901:image_factory subtypes = {} userId = anonymous 2011-10-13 04:38:36,789 DEBUG imgfac.ImageWarehouse.ImageWarehouse pid(14699) Message: Getting metadata (['latest_unpushed']) from http://localhost:9090/images/6f46c5a8-8d79-4be1-ab5d-a0918354b9d7 2011-10-13 04:38:36,794 DEBUG imgfac.ImageWarehouse.ImageWarehouse pid(14699) Message: Querying (http://localhost:9090/target_images/_query) with expression ($build == "63ec67d7-340d-4af2-8f89-92966c231812" && $target == "ec2") 2011-10-13 04:38:36,798 DEBUG imgfac.ImageWarehouse.ImageWarehouse pid(14699) Message: Getting metadata (['template']) from http://localhost:9090/target_images/2fab333d-f4ea-4a5c-86b1-3ff3ba0b80e6 2011-10-13 04:38:36,800 DEBUG imgfac.ImageWarehouse.ImageWarehouse pid(14699) Message: Created Image Warehouse instance http://localhost:9090 - buckets(target_images, templates, icicles, provider_images) 2011-10-13 04:38:36,805 DEBUG imgfac.ImageWarehouse.ImageWarehouse pid(14699) Message: Created Image Warehouse instance http://localhost:9090 - buckets(target_images, templates, icicles, provider_images) 2011-10-13 04:38:36,808 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Found neither a call-time config nor a config file - doing nothing 2011-10-13 04:38:37,360 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: Original URL http://download.fedoraproject.org/pub/fedora/linux/releases/15/Fedora/x86_64/os/ resolved to http://download.fedora.devel.redhat.com/pub/fedora/linux/releases/15/Fedora/x86_64/os/ 2011-10-13 04:38:37,361 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: Name: Fedora 15-6ef841f4-75c7-46d4-92a6-6f1b39f3675b, UUID: 4a618701-9625-4c36-854d-3ad3b5b06a48 2011-10-13 04:38:37,361 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: MAC: 52:54:00:bd:e2:9d, distro: Fedora 2011-10-13 04:38:37,361 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: update: 15, arch: x86_64, diskimage: /var/lib/imagefactory/images/Fedora 15-6ef841f4-75c7-46d4-92a6-6f1b39f3675b.dsk 2011-10-13 04:38:37,362 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: nicmodel: virtio, clockoffset: utc 2011-10-13 04:38:37,362 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: mousetype: ps2, disk_bus: virtio, disk_dev: vda 2011-10-13 04:38:37,363 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: icicletmp: /var/lib/oz/icicletmp/Fedora 15-6ef841f4-75c7-46d4-92a6-6f1b39f3675b, listen_port: 40737 2011-10-13 04:38:37,363 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: Original ISO path: /var/lib/oz/isos/Fedora15x86_64-url.iso 2011-10-13 04:38:37,363 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: Modified ISO cache: /var/lib/oz/isos/Fedora15x86_64-url-oz.iso 2011-10-13 04:38:37,364 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: Output ISO path: /var/lib/imagefactory/images/Fedora 15-6ef841f4-75c7-46d4-92a6-6f1b39f3675b-url-oz.iso 2011-10-13 04:38:37,364 DEBUG oz.Guest.FedoraRemoteGuest pid(14699) Message: ISO content path: /var/lib/oz/isocontent/Fedora 15-6ef841f4-75c7-46d4-92a6-6f1b39f3675b-url 2011-10-13 04:38:37,365 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Being asked to push for provider ec2-us-east-1 2011-10-13 04:38:37,365 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: distro: Fedora - update: 15 - arch: x86_64 2011-10-13 04:38:37,387 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: acting as EC2 user: 9598-0080-3741 2011-10-13 04:38:37,388 DEBUG imgfac.BuildJob.BuildAdaptor pid(14699) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from NEW to PUSHING 2011-10-13 04:38:37,389 DEBUG imgfac.BuildJob.BuildAdaptor pid(14699) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed percent complete from 0 to 0 2011-10-13 04:38:37,391 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Starting ami ami-c31cd8aa with instance_type m1.large 2011-10-13 04:38:37,831 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Creating temporary security group (imagefactory-6ef841f4-75c7-46d4-92a6-6f1b39f3675b) 2011-10-13 04:38:50,729 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Waiting for EC2 instance to start: 0/300 2011-10-13 04:39:01,588 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Waiting for EC2 instance to start: 10/300 2011-10-13 04:39:12,400 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Waiting for EC2 instance to start: 20/300 2011-10-13 04:39:18,984 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Waiting for EC2 ssh access: 0/300 2011-10-13 04:39:36,326 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Waiting 20 seconds for remaining boot tasks 2011-10-13 04:39:56,342 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Stopping cron and killing any updatedb process that may be running 2011-10-13 04:39:58,205 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Done 2011-10-13 04:40:08,237 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Terminating EC2 instance and deleting temp security group 2011-10-13 04:40:08,669 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Instance status (shutting-down) - waiting for 'terminated'. [0 of 300 seconds elapsed] 2011-10-13 04:40:13,751 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Instance status (shutting-down) - waiting for 'terminated'. [5 of 300 seconds elapsed] 2011-10-13 04:40:18,971 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Removed temporary security group (imagefactory-6ef841f4-75c7-46d4-92a6-6f1b39f3675b) 2011-10-13 04:40:18,972 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Exception caught in ImageFactory 2011-10-13 04:40:18,973 DEBUG imgfac.builders.BaseBuilder.Fedora_ec2_Builder pid(14699) Message: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 432, in push_image credentials) File "/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 601, in push_image_snapshot_ec2 self.install_euca_tools(guestaddr) File "/usr/lib/python2.7/site-packages/imgfac/builders/Fedora_ec2_Builder.py", line 441, in install_euca_tools self.guest.guest_execute_command(guestaddr, "yum -y install euca2ools") File "/usr/lib/python2.7/site-packages/oz/RedHat.py", line 469, in guest_execute_command command, timeout, tunnels) File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 342, in ssh_execute_command return subprocess_check_output(cmd) File "/usr/lib/python2.7/site-packages/oz/ozutil.py", line 310, in subprocess_check_output raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode) SubprocessException: 'ssh -i /tmp/tmp0Izmyl -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root.amazonaws.com yum -y install euca2ools' failed(255): ssh: connect to host ec2-50-17-42-253.compute-1.amazonaws.com port 22: Connection timed out 2011-10-13 04:40:18,974 DEBUG imgfac.BuildJob.BuildAdaptor pid(14699) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from PUSHING to FAILED Additional info: [root@dell-pesc430-03 templates]# rpm -qa | egrep 'aeolus|imagefactory|iwhd' aeolus-conductor-0.5.0-0.20111012134715git383de2f.fc15.noarch iwhd-0.99.2.ee04.dirty-1.fc15.x86_64 imagefactory-jeosconf-ec2-rhel-0.1.0-1.fc15.noarch rubygem-imagefactory-console-0.5.0-4.20110824113238gitd9debef.fc15.noarch aeolus-conductor-doc-0.5.0-0.20111012134715git383de2f.fc15.noarch imagefactory-jeosconf-ec2-fedora-0.1.0-1.fc15.noarch aeolus-configure-2.2.0-1.20111007151942git26136f2.fc15.noarch rubygem-aeolus-image-0.1.0-3.20111003170706git8f23238.fc15.noarch aeolus-all-0.5.0-0.20111012134715git383de2f.fc15.noarch imagefactory-0.7.0-1.fc15.noarch aeolus-conductor-daemons-0.5.0-0.20111012134715git383de2f.fc15.noarch rubygem-aeolus-cli-0.1.0-3.20111003133323git9451323.fc15.noarch