Bug 472722

From IRC:

<Lana> ok - i'm just wondering where that bit belongs then
<Lana> coz it sure looks important to me!
<mattf> Configuring MRG Grid to use EC2 Basic and EC2 Enhanced
<mattf> that section is all about Configuring an EC2 AMI to run with EC2 Enhanced
<mattf> should be under 10.2
<Lana> ok , so it needs to go first in the enhanced section then?
<mattf> or second, EC2 Enhanced involves configuring your local grid setup and creating an EC2 AMI with the "Configuring MRG Grid to use EC2 Basic and EC2 Enhanced" instructions
<mattf> there are 2 parts to EC2 Enhanced, local and remote (in EC2)

LKB

"Configuring MRG Grid to use EC2 Basic and EC2 Enhanced"
- This section is EC2 Enhanced only.  Really, it's specific to AMIs that will be used with EC2E.

10.1. EC2 Basic:
- Suggest adding Whenry's intro to this feature:
"MRG/EC2 Basic usage. With basic usage a user can submit an AMI as a job to EC2.
 This is useful when deploying a complete application stack into EC2. The AMI
contains the operating system and all the required packages. EC2 will boot the
image and the image can initialize the application stack on boot. 
Understanding basic usage is also important when using MRG EC2 Enhanced."
- "There is no need to transfer an executable because the AMI has already being loaded to EC2."
  - Suggest removing this sentence.  There's nothing to preclude the user from setting up their AMI to transfer files to the instance if they want.  It is just done outside of condor's control.
  - Reword next paragraph.  Numberous short sentences that don't mean much (ie "The AMI ID of the image is specified")
  - Should we mention how to use the EC2 tools?  That should be documented in AWS website, and putting it here means we have to keep up to date.  Suggest just referencing them instead of providing step by step instructions.
- Example at the bottom is an example script to place ON the AMI in rc.local (ie the work the AMI will perform).  This example reads data from the user-data field, creates a file called output.txt, and transfers that file out of the AMI before shutting down.
- I still think we need a section on AMI creation where we highlight things like the user data, that all files to be perserved must be transferred off the AMI by the user, and that the job must do a 'shutdown -h -P now' at the end or condor won't know the job has completed.

10.2 EC2 Enhanced
  - Think we should mention the key difference with EC2 Enhanced: It runs vanilla universe jobs in EC2.
  - Still think we need a separate section for AMI creation.

OK. I think I might have it now (crosses fingers). I'll brew it up for you to review shortly.

LKB

(In reply to comment #2)
> "Configuring MRG Grid to use EC2 Basic and EC2 Enhanced"
> - This section is EC2 Enhanced only.  Really, it's specific to AMIs that will
> be used with EC2E.
> 

I caught that one yesterday.

> 10.1. EC2 Basic:
> - Suggest adding Whenry's intro to this feature:
> "MRG/EC2 Basic usage. With basic usage a user can submit an AMI as a job to
> EC2.
>  This is useful when deploying a complete application stack into EC2. The AMI
> contains the operating system and all the required packages. EC2 will boot the
> image and the image can initialize the application stack on boot. 
> Understanding basic usage is also important when using MRG EC2 Enhanced."

<para>
	With &EC2B; an AMI can be submitted as a job to EC2. This is useful when deploying a complete application stack into EC2. The AMI contains the operating system and all the required packages. EC2 will boot the image and the image can initialize the application stack on boot. &EC2B; knowledge is also important when using &EC2E;.
</para>

> - "There is no need to transfer an executable because the AMI has already being
> loaded to EC2."
>   - Suggest removing this sentence.  There's nothing to preclude the user from
> setting up their AMI to transfer files to the instance if they want.  It is
> just done outside of condor's control.

It's gone.

>   - Reword next paragraph.  Numberous short sentences that don't mean much (ie
> "The AMI ID of the image is specified")

Wow, what an ugly paragraph. Here's a re-write:

<para>
	The AMI ID of the image needs to be specified in the job submission file. User data can also be passed to the remote job if it is required. Applications that require user data can access it using a Representational State Transfer (REST) based interface. Information on how to access image instance data, including user data, is available from <ulink url="http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-instancedata.html">Amazon Web Services Developer Guide</ulink>.
</para>

>   - Should we mention how to use the EC2 tools?  That should be documented in
> AWS website, and putting it here means we have to keep up to date.  Suggest
> just referencing them instead of providing step by step instructions.

More info? What exactly do you mean by 'EC2 tools' (Are you referring to Elasticfox? If so, I think it's  valuable in the format it is in, as people wouldn't necessarily guess that something like it exists. I wouldn't bother going into further depth though, as it isn't our product)?

> - Example at the bottom is an example script to place ON the AMI in rc.local
> (ie the work the AMI will perform).  This example reads data from the user-data
> field, creates a file called output.txt, and transfers that file out of the AMI
> before shutting down.

<para>
	This example contains a script for running an EC2 job in an AMI. Edit the <filename>/etc/rc.local</filename> file in the AMI and place this code at the end.
</para>
<para>
	This example reads data from the <parameter>user-data</parameter> field, creates a a file called <filename>output.txt</filename> and transfers that file out of the AMI before shutting down.
</para>

> - I still think we need a section on AMI creation where we highlight things
> like the user data, that all files to be perserved must be transferred off the
> AMI by the user, and that the job must do a 'shutdown -h -P now' at the end or
> condor won't know the job has completed.
> 

Agreed.

> 10.2 EC2 Enhanced
>   - Think we should mention the key difference with EC2 Enhanced: It runs
> vanilla universe jobs in EC2.

<section id="sect-Grid_User_Guide-Cloud_Computing-EC2E">
	<title>&EC2E;</title>
	<para>
		&EC2E; is used to run vanilla universe jobs in EC2.
	</para>
	<para>It uses security provided by the AWS X509 certificates as ...

Would you like to add more?


>   - Still think we need a separate section for AMI creation.

Agreed. Again :)

LKB

> >   - Should we mention how to use the EC2 tools?  That should be documented in
> > AWS website, and putting it here means we have to keep up to date.  Suggest
> > just referencing them instead of providing step by step instructions.
> 
> More info? What exactly do you mean by 'EC2 tools' (Are you referring to
> Elasticfox? If so, I think it's  valuable in the format it is in, as people
> wouldn't necessarily guess that something like it exists. I wouldn't bother
> going into further depth though, as it isn't our product)?

I'm talking about step 3 after "Advanced options".  That's info that the AWS docs provide for how to setup the Amazon EC2 tools.  Well, everything except the './ec2-add-group' and './ec2-authorize' stuff.

> > - Example at the bottom is an example script to place ON the AMI in rc.local
> > (ie the work the AMI will perform).  This example reads data from the user-data
> > field, creates a file called output.txt, and transfers that file out of the AMI
> > before shutting down.
> 
> <para>
>  This example contains a script for running an EC2 job in an AMI. Edit the
> <filename>/etc/rc.local</filename> file in the AMI and place this code at the
> end.
> </para>
> <para>
>  This example reads data from the <parameter>user-data</parameter> field,
> creates a a file called <filename>output.txt</filename> and transfers that file
> out of the AMI before shutting down.
> </para>

This example isn't "running an EC2 job in an AMI", this script IS the work the AMI is will perform (and there for the work the EC2 job will perform).  Suggest something like:

"Below is a simple example of a job that an AMI will execute.  Edit the ..."

> > - I still think we need a section on AMI creation where we highlight things
> > like the user data, that all files to be perserved must be transferred off the
> > AMI by the user, and that the job must do a 'shutdown -h -P now' at the end or
> > condor won't know the job has completed.
> > 
> 
> Agreed.

To start this section, I'd move the example at the end of 10.1 to a section named "Creating an AMI for use with EC2" or something like that.  I'd also add at the start of the section that the AMI is responsible for:
1. Retrieving "user-data"
2. Transferring input (if not staged)
3. Running a job
4. Transfer output
5. Shutdown instance (via  shutdown -h -P now)

> > 10.2 EC2 Enhanced
> >   - Think we should mention the key difference with EC2 Enhanced: It runs
> > vanilla universe jobs in EC2.
> 
> <section id="sect-Grid_User_Guide-Cloud_Computing-EC2E">
>  <title>&EC2E;</title>
>  <para>
>   &EC2E; is used to run vanilla universe jobs in EC2.
>  </para>
>  <para>It uses security provided by the AWS X509 certificates as ...
> 
> Would you like to add more?

The EC2 Enhanced feature is an extension of the EC2 feature, and allows running vanilla universe jobs in Amazon's EC2 service.  Whereas EC2 uses an AMI to execute a specific job, EC2 Enhanced uses generic AMIs to execute vanilla universe jobs.  From a user perspective, jobs executed with EC2 Enhanced act like any other vanilla universe job, only the execution node is in EC2 as opposed to a local condor pool.

In order to use EC2 Enhanced, you will need an AWS account with access to:
- EC2
- SQS
- S3

> >   - Still think we need a separate section for AMI creation.
> 
> Agreed. Again :)
> 
> LKB

For the EC2 Enhanced AMI creation section move the "Configuring MRG Grid to use MRG/EC2 Enhanced" there and change to "Configuring an AMI for use with MRG Grid EC2 Enhanced" or some such.

Move "Download and install the MRG/EC2 Enhanced RPMs" step 3 to the beginning of "Configuring MRG Grid to use MRG/EC2 Enhanced".


Also suggest massing into the intro in 10.1 mattf's description of the EC2 feature:

At a high level, Condor will manage the starting, monitoring and
cleaning up of EC2 resources. Your application will be installed in an
AMI stored in S3, and, once started, be responsible for the life-cycle
of your job and the termination of the AMI instance.

AMI instances running in EC2 do not have persistent storage directly
available to them between instantiations. It is advisable to program
your AMI to transfer your job's output out of the running instance
before you shut it down.

An AMI is fixed at creation time, meaning it cannot be customized before
an instance of it is started in EC2. However, an AMI instance can be
parameterized using EC2's "user-data" meta data functionality. This
allows for an AMI instance to receive instantiation specific input and
customize its operation.

(Below is best viewed with a fixed-width font)

At a low level,

    You:
      1) Create an AMI, upload it to S3, register it with EC2
 T    2) Create a job description and condor_submit it
 I
 M  Condor:                    AMI instance:
 E    3) Setup EC2 resources
\|/   4) Start AMI in EC2
 `                               5) Retrieve "user-data"
        (monitor                 6) Transfer input (if not staged)
         instance                7) Run job
         status)                 8) Transfer output
                                 9) Shutdown instance
     10) Clean up EC2 resources
     11) Remove job from Schedd

(In reply to comment #5)
> > >   - Should we mention how to use the EC2 tools?  That should be documented in
> > > AWS website, and putting it here means we have to keep up to date.  Suggest
> > > just referencing them instead of providing step by step instructions.
> > 
> > More info? What exactly do you mean by 'EC2 tools' (Are you referring to
> > Elasticfox? If so, I think it's  valuable in the format it is in, as people
> > wouldn't necessarily guess that something like it exists. I wouldn't bother
> > going into further depth though, as it isn't our product)?
> 
> I'm talking about step 3 after "Advanced options".  That's info that the AWS
> docs provide for how to setup the Amazon EC2 tools.  Well, everything except
> the './ec2-add-group' and './ec2-authorize' stuff.
> 

Hmmm ... I've been reading through it, and I think it's pretty crucial information. Also, we reference the AWS docs *a lot* and I think if we're not careful, all value in this document will be lost. We might as well put "See the AWS docs" at the top of the page and not bother with instructions at all. My vote is to keep them in, but of course I'm always up a for well-reasoned argument :)

> > > - Example at the bottom is an example script to place ON the AMI in rc.local
> > > (ie the work the AMI will perform).  This example reads data from the user-data
> > > field, creates a file called output.txt, and transfers that file out of the AMI
> > > before shutting down.
> > 
> > <para>
> >  This example contains a script for running an EC2 job in an AMI. Edit the
> > <filename>/etc/rc.local</filename> file in the AMI and place this code at the
> > end.
> > </para>
> > <para>
> >  This example reads data from the <parameter>user-data</parameter> field,
> > creates a a file called <filename>output.txt</filename> and transfers that file
> > out of the AMI before shutting down.
> > </para>
> 
> This example isn't "running an EC2 job in an AMI", this script IS the work the
> AMI is will perform (and there for the work the EC2 job will perform).  Suggest
> something like:
> 
> "Below is a simple example of a job that an AMI will execute.  Edit the ..."

<para>
	This example contains a script for a job to be executed by an AMI. Edit the <filename>/etc/rc.local</filename> file in the AMI and place this code at the end.
</para>

> 
> > > - I still think we need a section on AMI creation where we highlight things
> > > like the user data, that all files to be perserved must be transferred off the
> > > AMI by the user, and that the job must do a 'shutdown -h -P now' at the end or
> > > condor won't know the job has completed.
> > > 
> > 
> > Agreed.
> 
> To start this section, I'd move the example at the end of 10.1 to a section
> named "Creating an AMI for use with EC2" or something like that.  I'd also add
> at the start of the section that the AMI is responsible for:
> 1. Retrieving "user-data"
> 2. Transferring input (if not staged)
> 3. Running a job
> 4. Transfer output
> 5. Shutdown instance (via  shutdown -h -P now)

This will have to occur for 1.1.1.

> 
> > > 10.2 EC2 Enhanced
> > >   - Think we should mention the key difference with EC2 Enhanced: It runs
> > > vanilla universe jobs in EC2.
> > 
> > <section id="sect-Grid_User_Guide-Cloud_Computing-EC2E">
> >  <title>&EC2E;</title>
> >  <para>
> >   &EC2E; is used to run vanilla universe jobs in EC2.
> >  </para>
> >  <para>It uses security provided by the AWS X509 certificates as ...
> > 
> > Would you like to add more?
> 
> The EC2 Enhanced feature is an extension of the EC2 feature, and allows running
> vanilla universe jobs in Amazon's EC2 service.  Whereas EC2 uses an AMI to
> execute a specific job, EC2 Enhanced uses generic AMIs to execute vanilla
> universe jobs.  From a user perspective, jobs executed with EC2 Enhanced act
> like any other vanilla universe job, only the execution node is in EC2 as
> opposed to a local condor pool.

<para>
	The &EC2E; feature is an extension of &EC2B; that allows vanilla universe jobs to be run in Amazon's EC2 service.  &EC2E; uses generic AMIs to execute vanilla universe jobs. Jobs executed with &EC2E; act like any other vanilla universe job, except the execution node is in EC2 instead of a local condor pool.
</para>

> 
> In order to use EC2 Enhanced, you will need an AWS account with access to:
> - EC2
> - SQS
> - S3
> 
<para>
	To use &EC2E;, you will need an Amazon Web Services (AWS) account with access to the following features: 
	<itemizedlist>
		<listitem>
			<para>
				EC2
			</para>
		</listitem>
		<listitem>
			<para>
				SQS
			</para>
		</listitem>
		<listitem>
			<para>
				S3
			</para>
		</listitem>
	</itemizedlist>
</para>


> > >   - Still think we need a separate section for AMI creation.
> > 
> > Agreed. Again :)
> > 
> > LKB
> 
> For the EC2 Enhanced AMI creation section move the "Configuring MRG Grid to use
> MRG/EC2 Enhanced" there and change to "Configuring an AMI for use with MRG Grid
> EC2 Enhanced" or some such.
> 
> Move "Download and install the MRG/EC2 Enhanced RPMs" step 3 to the beginning
> of "Configuring MRG Grid to use MRG/EC2 Enhanced".
> 

Done.

> 
> Also suggest massing into the intro in 10.1 mattf's description of the EC2
> feature:
> 
> At a high level, Condor will manage the starting, monitoring and
> cleaning up of EC2 resources. Your application will be installed in an
> AMI stored in S3, and, once started, be responsible for the life-cycle
> of your job and the termination of the AMI instance.
> 
> AMI instances running in EC2 do not have persistent storage directly
> available to them between instantiations. It is advisable to program
> your AMI to transfer your job's output out of the running instance
> before you shut it down.
> 
> An AMI is fixed at creation time, meaning it cannot be customized before
> an instance of it is started in EC2. However, an AMI instance can be
> parameterized using EC2's "user-data" meta data functionality. This
> allows for an AMI instance to receive instantiation specific input and
> customize its operation.

<para>
	The starting, monitoring and cleaning up of EC2 resources occurs at the local level. The application is installed in an AMI stored in S3, and, once started, is responsible for the life-cycle of the job and the termination of the AMI instance.
</para>
<para>
	AMI instances running in EC2 do not have persistent storage directly available. It is advisable to program the AMI to transfer the output from a job out of the running instance before it is shut down.
</para>
<para>
	An AMI is fixed at creation time. This means that it cannot be customized before an instance of it is started in EC2. However, an AMI instance can be customized using EC2's <parameter>user-data</parameter> functionality. This allows for an AMI instance to receive instantiation specific input and customize its operation.
</para>

> 
> (Below is best viewed with a fixed-width font)
> 
> At a low level,
> 
>     You:
>       1) Create an AMI, upload it to S3, register it with EC2
>  T    2) Create a job description and condor_submit it
>  I
>  M  Condor:                    AMI instance:
>  E    3) Setup EC2 resources
> \|/   4) Start AMI in EC2
>  `                               5) Retrieve "user-data"
>         (monitor                 6) Transfer input (if not staged)
>          instance                7) Run job
>          status)                 8) Transfer output
>                                  9) Shutdown instance
>      10) Clean up EC2 resources
>      11) Remove job from Schedd

I can't work out a neat way to do this in text, so I'm going to hand it to the graphics guys to pretty it up. Once I have a graphic I'll get it into the docs.

LKB