Bug 749653

Summary: Oz: intermittent ssh timeout during customize_and_generate_icicle
Product: [Retired] CloudForms Cloud Engine Reporter: wes hayutin <whayutin>
Component: imagefactoryAssignee: Steve Loranz <sloranz>
Status: CLOSED ERRATA QA Contact: Martin Kočí <mkoci>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 1.0.0CC: akarol, cpelland, dajohnso, deltacloud-maint, dgao, hbrock, mkoci, nobody, ssachdev
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-15 20:17:17 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description wes hayutin 2011-10-27 19:41:25 UTC 
Description of problem:

oz.Guest.FedoraGuest pid(7601) Message: Waiting for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to finish installing, 3000/3600
2011-10-27 15:14:16,170 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to finish installing, 2990/3600
2011-10-27 15:14:26,782 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to finish installing, 2980/3600
2011-10-27 15:14:37,119 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to finish installing, 2970/3600
2011-10-27 15:14:46,418 INFO oz.Guest.FedoraGuest pid(7601) Message: Install of Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 succeeded
2011-10-27 15:14:46,418 INFO oz.Guest.FedoraGuest pid(7601) Message: Generate XML for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 with bootdev hd
2011-10-27 15:14:46,419 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Generated XML:
<?xml version="1.0"?>
<domain type="kvm">
  <name>Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75</name>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <uuid>e7ee4533-2bed-45af-bb49-8ff830f903e6</uuid>
  <clock offset="utc"/>
  <vcpu>1</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <console device="pty"/>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:a4:34:ad"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <console type="pty">
      <target port="0"/>
    </console>
    <serial type="tcp">
      <source mode="bind" host="127.0.0.1" service="54381"/>
      <protocol type="raw"/>
      <target port="1"/>
    </serial>
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/var/lib/imagefactory/images/base-image-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75.dsk"/>
    </disk>
  </devices>
</domain>

2011-10-27 15:14:46,419 DEBUG imgfac.builders.BaseBuilder.Fedora_vsphere_Builder pid(7601) Message: Base install complete - Doing customization and ICICLE generation
2011-10-27 15:14:46,420 INFO oz.Guest.FedoraGuest pid(7601) Message: Customizing image
2011-10-27 15:14:46,420 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Generated XML:
<?xml version="1.0"?>
<domain type="kvm">
  <name>Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75</name>
  <memory>1048576</memory>
  <currentMemory>1048576</currentMemory>
  <uuid>e7ee4533-2bed-45af-bb49-8ff830f903e6</uuid>
  <clock offset="utc"/>
  <vcpu>1</vcpu>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <os>
    <type>hvm</type>
    <boot dev="hd"/>
  </os>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <console device="pty"/>
    <graphics port="-1" type="vnc"/>
    <interface type="bridge">
      <source bridge="virbr0"/>
      <mac address="52:54:00:a4:34:ad"/>
      <model type="virtio"/>
    </interface>
    <input bus="ps2" type="mouse"/>
    <console type="pty">
      <target port="0"/>
    </console>
    
    <disk device="disk" type="file">
      <target dev="vda" bus="virtio"/>
      <source file="/var/lib/imagefactory/images/base-image-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75.dsk"/>
    </disk>
  <serial type="tcp"><source mode="bind" host="127.0.0.1" service="54381"/><protocol type="raw"/><target port="1"/></serial></devices>
</domain>

2011-10-27 15:14:46,421 INFO oz.Guest.FedoraGuest pid(7601) Message: Collection Setup
2011-10-27 15:14:46,423 INFO oz.Guest.FedoraGuest pid(7601) Message: Setting up guestfs handle for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75
2011-10-27 15:14:46,423 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Adding disk image /var/lib/imagefactory/images/base-image-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75.dsk
2011-10-27 15:14:46,423 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Launching guestfs
2011-10-27 15:14:49,846 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Inspecting guest OS
2011-10-27 15:14:50,767 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Getting mountpoints
2011-10-27 15:14:50,767 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Root device: /dev/VolGroup00/LogVol00
2011-10-27 15:14:50,858 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Step 1: Uploading ssh keys
2011-10-27 15:14:50,880 INFO oz.Guest.FedoraGuest pid(7601) Message: Generating new openssh key
2011-10-27 15:14:50,881 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Step 2: setup sshd
2011-10-27 15:14:51,015 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Step 3: Open up the firewall
2011-10-27 15:14:51,021 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Step 4: Guest announcement
2011-10-27 15:14:51,069 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Step 5: Set SELinux to permissive mode
2011-10-27 15:14:51,072 INFO oz.Guest.FedoraGuest pid(7601) Message: Cleaning up guestfs handle for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75
2011-10-27 15:14:51,072 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Syncing
2011-10-27 15:14:51,358 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Unmounting all
2011-10-27 15:14:51,405 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Killing guestfs subprocess
2011-10-27 15:14:51,994 INFO oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot
2011-10-27 15:14:51,994 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 300/300
2011-10-27 15:15:02,288 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 290/300
2011-10-27 15:15:12,755 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 280/300
2011-10-27 15:15:23,130 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 270/300
2011-10-27 15:15:33,525 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 260/300
2011-10-27 15:15:44,032 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 250/300
2011-10-27 15:15:54,461 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for guest Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to boot, 240/300
2011-10-27 15:16:04,801 DEBUG oz.Guest.FedoraGuest pid(7601) Message: IP address of guest is 192.168.122.133
2011-10-27 15:16:04,802 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Installing additional repository files
2011-10-27 15:16:07,180 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Installing custom packages
2011-10-27 15:17:19,273 INFO oz.Guest.FedoraGuest pid(7601) Message: Uploading custom files
2011-10-27 15:17:19,274 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Running custom commands
2011-10-27 15:17:19,274 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Syncing
2011-10-27 15:19:11,008 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Waiting for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75 to shutdown, 60/60
2011-10-27 15:21:43,479 INFO oz.Guest.FedoraGuest pid(7601) Message: Collection Teardown
2011-10-27 15:21:43,481 INFO oz.Guest.FedoraGuest pid(7601) Message: Setting up guestfs handle for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75
2011-10-27 15:21:43,481 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Adding disk image /var/lib/imagefactory/images/base-image-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75.dsk
2011-10-27 15:21:43,481 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Launching guestfs
2011-10-27 15:21:46,926 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Inspecting guest OS
2011-10-27 15:21:47,814 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Getting mountpoints
2011-10-27 15:21:47,814 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Root device: /dev/VolGroup00/LogVol00
2011-10-27 15:21:47,905 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Teardown step 1
2011-10-27 15:21:47,906 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Resetting authorized_keys
2011-10-27 15:21:47,942 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Teardown step 2
2011-10-27 15:21:47,942 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Resetting sshd_config
2011-10-27 15:21:47,972 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Resetting sshd service
2011-10-27 15:21:48,016 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Teardown step 3
2011-10-27 15:21:48,016 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Resetting iptables rules
2011-10-27 15:21:48,019 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Teardown step 4
2011-10-27 15:21:48,019 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Resetting announcement to host
2011-10-27 15:21:48,028 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Removing reportip
2011-10-27 15:21:48,028 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Resetting crond service
2011-10-27 15:21:48,036 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Teardown step 5
2011-10-27 15:21:48,044 INFO oz.Guest.FedoraGuest pid(7601) Message: Cleaning up guestfs handle for Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75
2011-10-27 15:21:48,044 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Syncing
2011-10-27 15:21:48,202 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Unmounting all
2011-10-27 15:21:48,253 DEBUG oz.Guest.FedoraGuest pid(7601) Message: Killing guestfs subprocess
2011-10-27 15:21:48,326 DEBUG imgfac.builders.BaseBuilder.Fedora_vsphere_Builder pid(7601) Message: Exception caught in ImageFactory
2011-10-27 15:21:48,326 DEBUG imgfac.builders.BaseBuilder.Fedora_vsphere_Builder pid(7601) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_vsphere_Builder.py", line 105, in build_upload
    self.output_descriptor = guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1032, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, True)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1008, in _internal_customize
    self.do_customize(guestaddr)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 978, in do_customize
    self.guest_execute_command(guestaddr, 'sync')
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 469, in guest_execute_command
    command, timeout, tunnels)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 342, in ssh_execute_command
    return subprocess_check_output(cmd)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 310, in subprocess_check_output
    raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode)
SubprocessException: 'ssh -i /etc/oz/id_rsa-icicle-gen -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root.122.133 sync' failed(255): Warning: Permanently added '192.168.122.133' (RSA) to the list of known hosts.
Connection to 192.168.122.133 timed out while waiting to read


2011-10-27 15:21:48,327 INFO oz.Guest.FedoraGuest pid(7601) Message: Cleaning up guest named Fedora 15 Audrey Client-68e06ff4-64a9-4a1b-a7ea-1595aacfbf75
2011-10-27 15:21:48,622 INFO oz.Guest.FedoraGuest pid(7601) Message: Cleaning up after install
2011-10-27 15:21:48,645 DEBUG imgfac.builders.BaseBuilder.Fedora_vsphere_Builder pid(7601) Message: Exception caught in ImageFactory
2011-10-27 15:21:48,646 DEBUG imgfac.builders.BaseBuilder.Fedora_vsphere_Builder pid(7601) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_vsphere_Builder.py", line 49, in build_image
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_vsphere_Builder.py", line 105, in build_upload
    self.output_descriptor = guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1032, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, True)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1008, in _internal_customize
    self.do_customize(guestaddr)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 978, in do_customize
    self.guest_execute_command(guestaddr, 'sync')
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 469, in guest_execute_command
    command, timeout, tunnels)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 342, in ssh_execute_command
    return subprocess_check_output(cmd)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 310, in subprocess_check_output
    raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode)
SubprocessException: 'ssh -i /etc/oz/id_rsa-icicle-gen -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root.122.133 sync' failed(255): Warning: Permanently added '192.168.122.133' (RSA) to the list of known hosts.
Connection to 192.168.122.133 timed out while waiting to read


[root@qeblade31 nodes]# rpm -qa | grep oz
oz-0.7.0-3.el6.noarch
[root@qeblade31 nodes]# rpm -qa | grep aeolus
aeolus-configure-2.2.0-1.20111024205543gitf9232cc.el6.noarch
rubygem-arel-2.0.10-0.aeolus.el6.noarch
rubygem-rack-mount-0.7.1-3.aeolus.el6.noarch
aeolus-conductor-0.5.0-0.20111025005411git5ff5abd.el6.noarch
rubygem-ZenTest-4.3.3-2.aeolus.el6.noarch
aeolus-conductor-doc-0.5.0-0.20111025005411git5ff5abd.el6.noarch
aeolus-all-0.5.0-0.20111025005411git5ff5abd.el6.noarch
rubygem-aeolus-cli-0.1.0-3.20111024205439gitc3aa990.el6.noarch
aeolus-conductor-daemons-0.5.0-0.20111025005411git5ff5abd.el6.noarch
rubygem-aeolus-image-0.1.0-4.20111024205454git6b2b696.el6.noarch
[root@qeblade31 nodes]# rpm -qa | grep factory
rubygem-imagefactory-console-0.5.0-4.20110824113238gitd9debef.el6.noarch
imagefactory-0.8.0-1.el6.noarch
imagefactory-jeosconf-ec2-fedora-0.8.0-1.el6.noarch
imagefactory-jeosconf-ec2-rhel-0.8.0-1.el6.noarch
[root@qeblade31 nodes]#
Comment 1 jrd 2011-10-28 12:53:46 UTC 
Stev, plz see if you can do anything with this in Ian's absence.
Comment 2 Hugh Brock 2011-10-28 13:33:43 UTC 
Is it possible this is just a timeout? Can someone test with a longer timeout
on the ssh connection?
Comment 3 Steve Loranz 2011-10-28 20:51:32 UTC 
I'm changing the title to more accurately reflect the problem.  I suggest that we audit the timeouts in ImageFactory and Oz and either bump them up significantly, or make them configurable.
Comment 4 Steve Loranz 2012-01-11 19:32:37 UTC 
I spoke with Wes and he agreed this could be moved to on_qa.  I'm not seeing this in any of my testing and we're not seeing more reports of this.
Comment 5 Martin Kočí 2012-01-13 13:24:14 UTC 
Neither me I'm not able to reproduce this issue so far.
Comment 6 Martin Kočí 2012-01-18 16:08:08 UTC 
I have reproduced the issue here http://hudson.rhq.lab.eng.bos.redhat.com:8080/hudson/view/DEV-CloudForms/job/ImageFactory-KocaTesting2/220/consoleFull.

Only the penultimate build failed (aeolus-image build --target vsphere --template templates/bug751209.tdl)

The test ran commands in 
["aeolus-image build --target rhevm --template templates/bug751209.tdl;",\
              "aeolus-image build --target rhevm --template templates/bug751209.tdl;",\
              "aeolus-image build --target rhevm --template templates/bug751209.tdl;",\
              "aeolus-image build --target rhevm --template templates/bug751209.tdl;",\
              "aeolus-image build --target rhevm --template templates/bug751209.tdl;",\
              "aeolus-image build --target vsphere --template templates/bug751209.tdl;",\
              "aeolus-image build --target vsphere --template templates/bug751209.tdl;"]
array where is 10 seconds delay between each start. 

2012-01-18 10:35:07,969 DEBUG imgfac.builders.BaseBuilder.RHEL6_vsphere_Builder pid(28572) Message: Exception caught in ImageFactory
2012-01-18 10:35:07,971 DEBUG imgfac.builders.BaseBuilder.RHEL6_vsphere_Builder pid(28572) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_vsphere_Builder.py", line 49, in build_image
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_vsphere_Builder.py", line 115, in build_upload
    self.output_descriptor = self.guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1177, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, True)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1157, in _internal_customize
    icicle = self.do_icicle(guestaddr)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 483, in do_icicle
    'rpm -qa')
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 476, in guest_execute_command
    command, timeout, tunnels)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 355, in ssh_execute_command
    return subprocess_check_output(cmd)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 323, in subprocess_check_output
    raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode)
SubprocessException: 'ssh -i /etc/oz/id_rsa-icicle-gen -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root.122.134 rpm -qa' failed(255): Warning: Permanently added '192.168.122.134' (RSA) to the list of known hosts.
Connection to 192.168.122.134 timed out while waiting to read


with packages:
#rpm -qa | grep 'aeolus\|imagefactory-\|oz-\|iwhd'
rubygem-imagefactory-console-0.5.0-4.20110824113238gitd9debef.el6.noarch
aeolus-configure-2.6.0-0.20120112133633git8701011.el6.noarch
aeolus-conductor-doc-0.9.0-0.20120111153517gite55da85.el6.noarch
imagefactory-jeosconf-ec2-rhel-1.0.0rc2_10_gcf8da4e-1.el6.noarch
rubygem-ZenTest-4.3.3-2.aeolus.el6.noarch
rubygem-aeolus-cli-0.4.0-0.20120111093540git5ec8dc7.el6.noarch
aeolus-conductor-0.9.0-0.20120111153517gite55da85.el6.noarch
iwhd-1.2-3.el6.x86_64
imagefactory-jeosconf-ec2-fedora-1.0.0rc2_10_gcf8da4e-1.el6.noarch
aeolus-conductor-daemons-0.9.0-0.20120111153517gite55da85.el6.noarch
oz-0.8.0-0.20111219203204git5775e9d.el6.noarch
aeolus-all-0.9.0-0.20120111153517gite55da85.el6.noarch
imagefactory-1.0.0rc2_10_gcf8da4e-1.el6.noarch
rubygem-arel-2.0.10-0.aeolus.el6.noarch
rubygem-aeolus-image-0.4.0-0.20120111093551git67fdf6f.el6.noarch
Comment 7 Steve Loranz 2012-01-25 18:32:51 UTC 
This commit raises the timeout value for this operation.

https://github.com/aeolusproject/oz/commit/ee20ffa8e600e352e2677b5764b12b5ec9352e6c
Comment 8 Martin Kočí 2012-01-30 17:10:17 UTC 
so far all tests passed. So I'd dare to move this bug to VERIFIED with packages
#rpm -qa | grep 'aeolus\|imagefactory-\|oz-\|iwhd'
iwhd-1.1-2.el6.x86_64
oz-0.9.0-0.20120127190146git043d582.el6.noarch
aeolus-conductor-doc-0.9.0-0.20120118181603git71cd8bc.el6.noarch
aeolus-all-0.9.0-0.20120118181603git71cd8bc.el6.noarch
rubygem-imagefactory-console-0.5.0-4.20110824113238gitd9debef.el6.noarch
aeolus-conductor-daemons-0.9.0-0.20120118181603git71cd8bc.el6.noarch
imagefactory-jeosconf-ec2-fedora-1.0.0rc3_5_g1041f75-1.el6.noarch
rubygem-arel-2.0.10-0.aeolus.el6.noarch
rubygem-rack-mount-0.7.1-3.aeolus.el6.noarch
rubygem-ZenTest-4.3.3-2.aeolus.el6.noarch
rubygem-aeolus-image-0.4.0-0.20120118121635git0d31a37.el6.noarch
rubygem-aeolus-cli-0.4.0-0.20120118121626git6fddd65.el6.noarch
aeolus-configure-2.6.0-0.20120118121620gita996371.el6.noarch
imagefactory-jeosconf-ec2-rhel-1.0.0rc3_5_g1041f75-1.el6.noarch
aeolus-conductor-0.9.0-0.20120118181603git71cd8bc.el6.noarch
imagefactory-1.0.0rc3_5_g1041f75-1.el6.noarch

I'll reopen this bug if issue occurs again or move back to ON_DEV.
Comment 9 errata-xmlrpc 2012-05-15 20:17:17 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2012-0588.html