Bug 723897

Summary:	Documentation: unable to build rhevm,vmware at the same time w/ aeolus-image
Product:	[Retired] CloudForms Cloud Engine	Reporter:	wes hayutin <whayutin>
Component:	Documentation	Assignee:	Justin Clift <jclift>
Status:	CLOSED CURRENTRELEASE	QA Contact:	wes hayutin <whayutin>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	0.3.1	CC:	akarol, clalance, dajohnso, deltacloud-maint, imcleod, kwade, markmc, morazi, sreichar, ssachdev
Target Milestone:	beta
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:	722805	Environment:
Last Closed:	2011-12-08 13:48:09 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description wes hayutin 2011-07-21 13:41:16 UTC

+++ This bug was initially created as a clone of Bug #722805 +++

Description of problem:

I setup my conductor w/ the configurations required for both rhevm and vmware.
I tried four attempts.

1. aeolus-image build --target rhevm,vmware --template /root/RHEL_template01.xml (vmware failed)

2. aeolus-image build --target rhevm --template /root/RHEL_template01.xml (pass, build completed)

3. aeolus-image build --target vmware --template /root/RHEL_template01.xml (pass, build completed)

4. Just to make sure a combined build failed.. I ran #1 again 
aeolus-image build --target rhevm,vmware --template /root/RHEL_template01.xml (vmware failed)


#1 had an error
2011-07-17 21:21:36,074 DEBUG oz.Guest.RHEL6Guest pid(13273) Message: Checking if there is enough space on the filesystem
2011-07-17 21:21:36,077 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1200, in iso_generate_install_media
    self.cleanup_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1204, in cleanup_iso
    shutil.rmtree(self.iso_contents)
  File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.6/shutil.py", line 217, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "/usr/lib64/python2.6/shutil.py", line 215, in rmtree
    os.remove(fullname)
OSError: [Errno 2] No such file or directory: '/var/lib/oz/isocontent/RHEL_westestVmware01-url/images/efiboot.img'

The fourth attempt failed w/

2011-07-17 22:05:20,544 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1191, in iso_generate_install_media
    self.copy_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 939, in copy_iso
    os.makedirs(self.iso_contents)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/var/lib/oz/isocontent/RHEL_westestVmware01-url'


I'll attach the logs to all four builds

After speaking w/ Hugh on Friday, building w/ multiple targets should be supported, or that was my understanding.

--- Additional comment from whayutin on 2011-07-17 22:14:59 EDT ---

Created attachment 513547 [details]
vmware and rhevm #1

--- Additional comment from whayutin on 2011-07-17 22:15:22 EDT ---

Created attachment 513548 [details]
rhevm only

--- Additional comment from whayutin on 2011-07-17 22:15:47 EDT ---

Created attachment 513549 [details]
vmware only

--- Additional comment from whayutin on 2011-07-17 22:16:31 EDT ---

[root@dell-pem600-01 ~]# rpm -q imagefactory
imagefactory-0.2.3-1.el6.noarch
[root@dell-pem600-01 ~]# rpm -q iwhd
iwhd-0.97-1.el6.x86_64
[root@dell-pem600-01 ~]# rpm -q oz
oz-0.5.0-1.el6.noarch
[root@dell-pem600-01 ~]# rpm -qa |grep aeolus
aeolus-configure-2.0.1-0.el6.20110715084043git3c66412.noarch
aeolus-conductor-0.3.0-0.el6.20110715133516gitd6def1e.noarch
aeolus-conductor-daemons-0.3.0-0.el6.20110715133516gitd6def1e.noarch
rubygem-aeolus-image-0.0.1-1.el6.20110715133516gitd6def1e.noarch
aeolus-all-0.3.0-0.el6.20110715133516gitd6def1e.noarch
aeolus-conductor-doc-0.3.0-0.el6.20110715133516gitd6def1e.noarch

--- Additional comment from whayutin on 2011-07-17 22:20:03 EDT ---

Created attachment 513552 [details]
rhevm and vmware attempt #2

--- Additional comment from whayutin on 2011-07-17 22:23:10 EDT ---

I also attempted the build using..

[root@dell-pem600-01 ~]# aeolus-image build --target vmware,rhevm --image 0d84d791-d9dc-4acb-9537-b42dcc825550 --template /root/RHEL_template01.xml 

and one of the image builds failed w/
2011-07-17 22:21:47,375 DEBUG imagefactory.BuildJob.BuildAdaptor pid(13273) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from BUILDING to FAILED
2011-07-17 22:21:47,376 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Exception caught in ImageFactory
2011-07-17 22:21:47,376 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 125, in build_image
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1200, in iso_generate_install_media
    self.cleanup_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1204, in cleanup_iso
    shutil.rmtree(self.iso_contents)
  File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.6/shutil.py", line 212, in rmtree
    rmtree(fullname, ignore_errors, onerror)
  File "/usr/lib64/python2.6/shutil.py", line 217, in rmtree
    onerror(os.remove, fullname, sys.exc_info())
  File "/usr/lib64/python2.6/shutil.py", line 215, in rmtree
    os.remove(fullname)
OSError: [Errno 2] No such file or directory: '/var/lib/oz/isocontent/RHEL_westestVmware01-url/images/pxeboot/vmlinuz'

--- Additional comment from whayutin on 2011-07-17 22:27:36 EDT ---

*** Bug 721050 has been marked as a duplicate of this bug. ***

--- Additional comment from whayutin on 2011-07-17 22:39:20 EDT ---

my last attempt at this was to build w/ ec2, and then take that image id and build for rhevm, vmware


[root@dell-pem600-01 ~]# aeolus-image build --target ec2 --template /root/RHEL_template02.xml 

Target Image: a40e8a4d-8715-4c50-aafc-003d3d886cc9
Image: 74898c4e-b9a6-4676-ace1-a4d8c21c9591
Build: 17329a0f-a9a3-4d3c-a918-6b20adf69c12
Status: COMPLETED
Percent Complete: 100
[root@dell-pem600-01 ~]# aeolus-image build --target vmware,rhevm --image 74898c4e-b9a6-4676-ace1-a4d8c21c9591 --template /root/RHEL_template02.xml 

Target Image: 77fc9fc4-f8f5-45c9-919b-775897ab3360
Image: 74898c4e-b9a6-4676-ace1-a4d8c21c9591
Build: e441cd64-edde-467b-8301-61f93390f28c
Status: BUILDING
Percent Complete: 0

Target Image: 099e8363-1d86-4482-9723-6dbd47670fd0
Image: 74898c4e-b9a6-4676-ace1-a4d8c21c9591
Build: e441cd64-edde-467b-8301-61f93390f28c
Status: BUILDING
Percent Complete: 0
[root@dell-pem600-01 ~]# 


I get..
2011-07-17 22:37:10,447 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1191, in iso_generate_install_media
    self.copy_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 939, in copy_iso
    os.makedirs(self.iso_contents)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/var/lib/oz/isocontent/RHEL_westest02-url'

2011-07-17 22:37:10,449 DEBUG imagefactory.BuildJob.BuildAdaptor pid(13273) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from FAILED to FAILED
2011-07-17 22:37:10,450 DEBUG oz.Guest.RHEL6Guest pid(13273) Message: Extracting ISO contents
2011-07-17 22:37:10,480 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Exception caught in ImageFactory
2011-07-17 22:37:10,480 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1191, in iso_generate_install_media
    self.copy_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 958, in copy_iso
    current = os.getcwd()
OSError: [Errno 2] No such file or directory

--- Additional comment from whayutin on 2011-07-17 22:41:40 EDT ---

[root@dell-pem600-01 ~]# aeolus-image build --target ec2,vmware --template /root/RHEL_template02.xml 

Target Image: 09125a52-3294-4a9f-9e3a-7b2092c22d64
Image: bd5b0b49-8fe0-4657-a9d5-cae367f84e33
Build: 32bb34dc-e81e-4189-8a2f-6350a1395b10
Status: New
Percent Complete: 0

Target Image: 2e07315c-7a2e-4815-b787-1ad471d9c95f
Image: bd5b0b49-8fe0-4657-a9d5-cae367f84e33
Build: 32bb34dc-e81e-4189-8a2f-6350a1395b10
Status: BUILDING
Percent Complete: 0


vmware failed w/ the following.. ec2 worked

2011-07-17 22:40:39,758 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1191, in iso_generate_install_media
    self.copy_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 958, in copy_iso
    current = os.getcwd()
OSError: [Errno 2] No such file or directory

2011-07-17 22:40:39,759 DEBUG imagefactory.BuildJob.BuildAdaptor pid(13273) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from BUILDING to FAILED
2011-07-17 22:40:39,760 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Exception caught in ImageFactory
2011-07-17 22:40:39,760 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 125, in build_image
    self.build_upload(build_id)
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1191, in iso_generate_install_media
    self.copy_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 958, in copy_iso
    current = os.getcwd()
OSError: [Errno 2] No such file or directory

2011-07-17 22:40:39,760 DEBUG imagefactory.BuildJob.BuildAdaptor pid(13273) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from FAILED to FAILED
2011-07-17 22:40:39,994 DEBUG imagefactory.BuildJob.BuildAdaptor pid(13273) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed percent complete from 0 to 100
2011-07-17 22:40:39,995 DEBUG imagefactory.BuildJob.BuildAdaptor pid(13273) Message: Raising event with agent handler (<ImageFactoryAgent(Thread-1, initial)>), changed status from NEW to COMPLETED
2011-07-17 22:40:39,995 DEBUG imagefactory.builders.BaseBuilder.RHEL6Builder pid(13273) Message: Completed placeholder warehouse object for linux non-upload image...

--- Additional comment from ssachdev on 2011-07-18 08:00:16 EDT ---

aeolus-image build --target ec2,vmware --template template2.tpl
==================================================================


aeolus-image push --provider ec2-us-east-1,vmware --id ce997af4-28a0-4858-94dd-bd45e13bc920

Provider Image: 5158c62b-97dd-444d-8539-1eb6947fa70d
Image: ce997af4-28a0-4858-94dd-bd45e13bc920
Build: bcdcc303-9dbc-4f19-9bdd-28091a54212e
Status: New
Percent Complete: 0

Provider Image: 05be1f5f-b762-4462-9179-b8515cd22f35
Image: ce997af4-28a0-4858-94dd-bd45e13bc920
Build: bcdcc303-9dbc-4f19-9bdd-28091a54212e
Status: PUSHING
Percent Complete: 0
=================================================

Tried for ec2 and vmware together , Build and Push succeeds . 
Able to launch two deployments for ec2 and vmware respectively.

--- Additional comment from whayutin on 2011-07-18 09:12:18 EDT ---

ah yes.. building w/ ec2,vmware WORKS! w/ a fedora based template, but fails w/ a RHEL based template


[root@dell-pem600-01 ~]# aeolus-image build --target ec2,vmware --template /root/fedora.tpl 

Target Image: e72602bc-84d1-4a07-965f-7072ca6310e9
Image: a3817a75-e7a2-494b-a171-0ca7f9800808
Build: 51cffe0c-2bef-4af8-ab73-5586815b7861
Status: COMPLETED
Percent Complete: 100

Target Image: be51086c-f405-4d17-a7ca-c34b60974b67
Image: a3817a75-e7a2-494b-a171-0ca7f9800808
Build: 51cffe0c-2bef-4af8-ab73-5586815b7861
Status: BUILDING
Percent Complete: 0
[root@dell-pem600-01 ~]# cat /root/fedora.tpl 
<?xml version="1.0"?>
<template>
  <name>wes_vmware</name>
  <description>desc</description>
  <os>
    <name>Fedora</name>
    <arch>x86_64</arch>
    <version>14</version>
    <install type="url">
      <url>http://download.fedoraproject.org/pub/fedora/linux/releases/14/Fedora/x86_64/os/</url>
    </install>
  </os>
  <repositories>
    <repository name="custom">
      <url>http://repos.fedorapeople.org/repos/aeolus/demo/webapp/</url>
      <signed>false</signed>
    </repository>
  </repositories>
</template>

[root@dell-pem600-01 ~]#

--- Additional comment from whayutin on 2011-07-18 09:13:54 EDT ---

whoops.. spoke too soon.. my fedora build failed... w/ 

2011-07-18 09:12:35,226 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(13273) Message: Exception caught in ImageFactory
2011-07-18 09:12:35,227 DEBUG imagefactory.builders.BaseBuilder.FedoraBuilder pid(13273) Message: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/imagefactory/builders/FedoraBuilder.py", line 157, in build_upload
    self.guest.generate_install_media(force_download=False)
  File "/usr/lib/python2.6/site-packages/oz/RedHat.py", line 439, in generate_install_media
    return self.iso_generate_install_media(fetchurl, force_download)
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 1191, in iso_generate_install_media
    self.copy_iso()
  File "/usr/lib/python2.6/site-packages/oz/Guest.py", line 958, in copy_iso
    current = os.getcwd()
OSError: [Errno 2] No such file or directory

--- Additional comment from whayutin on 2011-07-18 09:46:46 EDT ---

so on Shveta's setup.. not mine :) 
You can build a fedora template w/ ec2,vmware

Notice the last two entries have the same UUID

[root@snowstorm ~]# aeolus-image list --images
UUID                                     NAME              TARGET     OS         OS VERSION     ARCH       DESCRIPTION     
ce997af4-28a0-4858-94dd-bd45e13bc920     shveta_vmware     ec2        Fedora     14             x86_64     desc            
ce997af4-28a0-4858-94dd-bd45e13bc920     shveta_vmware     vmware     Fedora     14             x86_64     desc            
1e2b0474-2b39-41b8-87c2-377dd069ea30     shveta_vmware     ec2        Fedora     14             x86_64     desc            
a9046f5b-0f8c-4d71-b381-ddaeff69f03a     shveta_vmware     ec2        Fedora     14             x86_64     desc            
a9046f5b-0f8c-4d71-b381-ddaeff69f03a     shveta_vmware     vmware     Fedora     14             x86_64     desc

--- Additional comment from markmc on 2011-07-18 09:51:45 EDT ---

it looks to me like oz is using the same isocontent directory for both builds and probably deleting the directory when the first build completes

i.e. its using the template name to attempt to make the dir unique:

  self.iso_contents = os.path.join(self.data_dir, "isocontent",
                                   self.tdl.name + "-" + self.tdl.installtype)

--- Additional comment from imcleod on 2011-07-18 13:18:19 EDT ---

We already have code in the Factory that works around this by giving each Oz build object a unique name (by concatenating the name found in the TDL with the UUID of the target_image being built).  

However, at some point Oz switched from using a separate "name" property within the Oz top level object to always referencing the name property of the embedded TDL.

I've updated the Factory to directly modify the name property of the TDL object that we pass to Oz, rather than the name property of the resulting build object.  The change is now in our master branch.

This should resolve the issue.

--- Additional comment from dajohnso on 2011-07-20 17:42:50 EDT ---

Trying to verify this, the latest imagefactory rpm dies without error while attempting to download the iso mentioned in the template.

[root@ibm-crichton-02 ~]# service imagefactory status
imagefactory dead but pid file exists

[root@ibm-crichton-02 ~]# rpm -qa | grep factory
rubygem-image_factory_console-0.4.0-1.el6.20110719153543gitd39221e.noarch
imagefactory-0.4.0-1.el6.noarch

Last log statements
=====================
2011-07-20 17:36:45,262 INFO oz.Guest.RHEL6Guest pid(6172) Message: Fetching the original install media from http://refarch.cloud.lab.eng.bos.redhat.com/pub/kits/os/rhel/6/6.1/Server/x86_64/rhel-server-6.1-x86_64-dvd.iso
2011-07-20 17:36:45,329 DEBUG oz.Guest.RHEL6Guest pid(6172) Message: 3kB of 3516418kB
<snip>
2011-07-20 17:36:50,250 DEBUG oz.Guest.RHEL6Guest pid(6172) Message: 81920kB of 3516418kB
2011-07-20 17:36:50,287 DEBUG oz.Guest.RHEL6Guest pid(6172) Message: 307201kB of 3516418kB


Feel free to poke around this beakerbox:
ibm-crichton-02.rhts.eng.bos.redhat.com

--- Additional comment from whayutin on 2011-07-20 17:59:10 EDT ---

This has been recreated on two different boxes


[root@hp-z200-07 ~]# rpm -qa | grep imagefactory
imagefactory-0.3.1_5_g00c77c3-1.el6.noarch
rubygem-imagefactory-console-0.4.0-1.el6.20110720133919git89a6b60.noarch


Recreate:
1. setup vmware and rhevm in aeolus
2. aeolus-image build --target vsphere,rhevm --template $template

The build ends of crashing imagefactory around the time the iso's are downloaded.

** interesting note would be to:
1. aeolus-image build --target rhevm --template $template (should work)
2. aeolus-image build --target vsphere --template $template (should work)
3. aeolus-image build --target rhevm, vsphere --template $template (fails)

imagefactory crashes..

--- Additional comment from imcleod on 2011-07-20 18:00:36 EDT ---

OK.  I may have to punt this over to Chris.  My guess is that Oz is trying to download the ISO two separate times.  I'm not sure how to stop this.

Chris,

Thoughts?

--- Additional comment from clalance on 2011-07-21 09:36:36 EDT ---

So my guess here is that imagefactory is creating two oz objects with the same distro/version/arch, and that the downloads are tromping all over each other.  It's a known problem in Oz, but not one that I have fixed yet.

I have an idea of how to fix it, but the code to do it is non-trivial.  I can work on it for the 0.6.0 release of Oz, but I would say that for the time being, we document that you can't do two builds at the same time.

--- Additional comment from whayutin on 2011-07-21 09:40:40 EDT ---

After speaking w/ Hugh, Mike, Angus.. removing from blocker.. cloning for doc

Comment 1 wes hayutin 2011-08-01 19:11:24 UTC

BZ 723897 - Unable to build RHEV-M and VMware at the same time with aeolus-image
Users should not attempt to build/push images to both rhevm and vmware at the same time.

Comment 2 wes hayutin 2011-08-01 19:48:56 UTC

removing from tracker

Comment 3 wes hayutin 2011-08-01 19:54:12 UTC

release pending...

Comment 4 wes hayutin 2011-08-01 19:56:06 UTC

release pending...

Comment 6 wes hayutin 2011-12-08 13:48:09 UTC

perm close