Bug 801907 - After starting a "Build All" - status reporting incorrect if 1 build fails
Summary: After starting a "Build All" - status reporting incorrect if 1 build fails
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: CloudForms Cloud Engine
Classification: Retired
Component: aeolus-conductor
Version: 1.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
Assignee: Angus Thomas
QA Contact: wes hayutin
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-03-09 19:39 UTC by Steve Reichard
Modified: 2012-08-30 17:16 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-08-30 17:16:51 UTC
Embargoed:


Attachments (Terms of Use)
ss1 (217.00 KB, image/png)
2012-04-18 09:48 UTC, Rehana
no flags Details
TPL (861 bytes, text/plain)
2012-04-18 09:57 UTC, Rehana
no flags Details

Description Steve Reichard 2012-03-09 19:39:21 UTC
Description of problem:


In my setup I have 5 provider accounts configured.

I've noticed if I do a build all and one build fails, and others succeed I do not get status back from any of the builds.  I need to go to the log to check the status.


I should probably investigate why one build failed, but that would be another BZ -
2012-03-09 14:04:27,972 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(2b438956) Message: Exception caught in ImageFactory
2012-03-09 14:04:27,994 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(2b438956) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 66, in build_image
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 117, in build_upload
    self.output_descriptor = guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1178, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, True)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1162, in _internal_customize
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 117, in build_upload
    self.output_descriptor = guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1178, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, True)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1162, in _internal_customize
    self._collect_teardown(modified_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 274, in _collect_teardown
    g_handle = self._guestfs_handle_setup(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 886, in _guestfs_handle_setup
    g.launch()
  File "/usr/lib/python2.6/site-packages/guestfs.py", line 152, in launch
    return libguestfsmod.launch (self._o)
RuntimeError: link: /tmp/.guestfs-0/kernel /tmp/.guestfs-0/kernel.13749: File exists


Version-Release number of selected component (if applicable):


How reproducible:

I have seen this several times in the past, but have not found a reliable reproducer.

Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 wes hayutin 2012-03-12 20:09:39 UTC
The issue you hit w/ 
RuntimeError: link: /tmp/.guestfs-0/kernel /tmp/.guestfs-0/kernel.13749: File
exists

is tracked here:
https://bugzilla.redhat.com/show_bug.cgi?id=790528

Comment 2 Martin Kočí 2012-03-12 20:32:12 UTC
With what version of imagefactory ?

Comment 3 Steve Reichard 2012-03-14 11:19:16 UTC
[root@cf-cloudforms5 imagefactory]#
/pub/scripts/post_install_configuration_scripts/cf-versions 
Red Hat Enterprise Linux Server release 6.2 (Santiago)
Linux cf-cloudforms5.cloud.lab.eng.bos.redhat.com 2.6.32-220.4.2.el6.x86_64 #1
SMP Mon Feb 6 16:39:28 EST 2012 x86_64 x86_64 x86_64 GNU/Linux
postgresql-8.4.9-1.el6_1.1.x86_64
mongodb-1.8.2-3.el6.x86_64
euca2ools-1.3.1-4.el6_0.noarch
ruby-1.8.7.352-6.el6.x86_64
rubygems-1.8.16-1.el6.noarch
deltacloud-core-0.5.0-5.el6.noarch
rubygem-deltacloud-client-0.5.0-2.el6.noarch
package libdeltacloud is not installed
hail-0.8-0.2.gf9c5b967.el6_0.x86_64
puppet-2.6.11-1.el6_1.noarch
aeolus-configure-2.5.0-16.el6.noarch
iwhd-1.2-3.el6.x86_64
imagefactory-1.0.0rc8-1.el6.noarch
aeolus-conductor-daemons-0.8.0-39.el6.noarch
aeolus-conductor-0.8.0-39.el6.noarch
[root@cf-cloudforms5 imagefactory]#

Comment 4 wes hayutin 2012-04-17 21:47:15 UTC
is this still an issue??

Comment 5 Rehana 2012-04-18 09:47:28 UTC
I tried to reproduce the scenario by giving a wrong url for "Aeolus" repository.

1. Got the build failed for "vsphere" and "rhevm"
2. Observed that the status of ec2 got updated and push button got enabled for ec2
3. PFA screen shot (PFA:ss1.png)

error from image factory log;

2012-04-18 05:42:38,406 INFO oz.Guest.RHEL6Guest thread(57b8228a) Message: Cleaning up after install
2012-04-18 05:42:38,420 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(57b8228a) Message: Exception caught in ImageFactory
2012-04-18 05:42:38,421 DEBUG imgfac.builders.BaseBuilder.RHEL6_rhevm_Builder thread(57b8228a) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 66, in build_image
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imgfac/builders/Fedora_rhevm_Builder.py", line 117, in build_upload
    self.output_descriptor = guest.customize_and_generate_icicle(libvirt_xml)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1178, in customize_and_generate_icicle
    return self._internal_customize(libvirt_xml, True)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1154, in _internal_customize
    self.do_customize(guestaddr)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 1110, in do_customize
    tunnels=self.tunnels)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 476, in guest_execute_command
    command, timeout, tunnels)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 355, in ssh_execute_command
    return subprocess_check_output(cmd)
  File "/usr/lib/python2.6/site-packages/oz/ozutil.py", line 323, in subprocess_check_output
    raise SubprocessException("'%s' failed(%d): %s" % (cmd, retcode, stderr), retcode)
SubprocessException: 'ssh -i /etc/oz/id_rsa-icicle-gen -F /dev/null -o ServerAliveInterval=30 -o StrictHostKeyChecking=no -o ConnectTimeout=10 -o UserKnownHostsFile=/dev/null -o PasswordAuthentication=no root.122.58 yum -y install "rhev-agent" ' failed(1): Warning: Permanently added '192.168.122.58' (RSA) to the list of known hosts.
http://download.devel.redhat.com/rel-eng/CloudForms/CloudEngine/1.0/latest/el6-engine-and-agents/x86_64/repodata/repomd.xml: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404"
Trying other mirror.
Error: Nothing to do

info:

 rpm -qa | grep aeolus
aeolus-conductor-doc-0.8.9-1.el6.noarch
aeolus-configure-2.5.3-1.el6.noarch
aeolus-conductor-0.8.9-1.el6.noarch
rubygem-aeolus-image-0.3.0-12.el6.noarch
aeolus-all-0.8.9-1.el6.noarch
rubygem-aeolus-cli-0.3.1-1.el6.noarch
aeolus-conductor-daemons-0.8.9-1.el6.noarch

Comment 6 Rehana 2012-04-18 09:48:30 UTC
Created attachment 578294 [details]
ss1

Comment 7 Rehana 2012-04-18 09:57:37 UTC
Created attachment 578295 [details]
TPL

Attaching the template file i used to create the scenario.

Comment 8 James Laska 2012-04-18 11:56:37 UTC
The attached screenshot (comment#6) shows a 'build all' test where 2 providers failed (vsphere and rhevm), and 2 succeeded (ec2 and mock).  The two failed builds are indicated in the UI.

Assuming the recreate is valid, and the reproducer instructions valid, this appears to confirm that the reported problem is resolved.

Steve, can you examine the screenshot, and the recreate, to confirm?

Comment 9 Steve Reichard 2012-04-18 12:25:46 UTC
I have noticed the build and push failure messages in later beta6s.

I believe this is now resolved.

spr

Comment 10 Rehana 2012-04-18 12:28:49 UTC
thanks...
moving the status to verified

rpm -qa | grep aeolus
aeolus-conductor-doc-0.8.9-1.el6.noarch
aeolus-configure-2.5.3-1.el6.noarch
aeolus-conductor-0.8.9-1.el6.noarch
rubygem-aeolus-image-0.3.0-12.el6.noarch
aeolus-all-0.8.9-1.el6.noarch
rubygem-aeolus-cli-0.3.1-1.el6.noarch
aeolus-conductor-daemons-0.8.9-1.el6.noarch


Note You need to log in before you can comment on or make changes to this bug.