Bug 641130

Have not seen any comments or proposals for fixing this bug from Deltacloud folks and it is over a month.  Should the severity be raised?  EC2 clouds with large numbers of images cannot be managed using Deltacloud due to this bug.

Michal, can you take a look and see what we can do to address this problem in the API?

Sure, I'll figure out what is going on there and let you guys know tomorrow.

OK, so results:

When using deltacloud API (amazon-ec2) gems listing of all *amazon* images returns 201 objects. (Timing: ~10s)

When using 'aws' library (new right_aws) and listing for all *amazon* the amount of images is same as above. (Timing: ~4s)

Now, when I use "describe_images()" in 'aws' library, without 'owner' parameter (so it basically it returns all images) I got 6654 objects on output. (Timing: ~15s)

So my conclusion:

This is not a bug, it's about features. Open question is, if we return all images or only images owned by current user or all Amazon images.

My opinion is to follow 'aws' library documentation and use listing in this way:

<snip>
ec2.describe_images_by_executable_by('522821470517')
ec2.describe_images_by_executable_by('self')
ec2.describe_images_by_executable_by('all')
</snip>

So the 'self', 'all' and ID parameter can by specified by 'owner_id' param in Deltacloud API and default would be 'self'.

So to be precise, there are some timings:

'aws' gem:

describe_images_by_executable_by('all') => 6654 images
=======================================================
real	0m43.183s
user	0m16.227s
sys	0m0.377s


ec2.describe_images_by_owner('amazon') => 201 images
=======================================================
real	0m2.008s
user	0m0.528s
sys	0m0.092s


'amazon-ec2' gem:

ec2.describe_images => 201 images
=======================================================
real	0m4.750s
user	0m0.627s
sys	0m0.093s

(In reply to comment #4)
> OK, so results:
> 
> When using deltacloud API (amazon-ec2) gems listing of all *amazon* images
> returns 201 objects. (Timing: ~10s)
> 
> When using 'aws' library (new right_aws) and listing for all *amazon* the
> amount of images is same as above. (Timing: ~4s)
> 
> Now, when I use "describe_images()" in 'aws' library, without 'owner' parameter
> (so it basically it returns all images) I got 6654 objects on output. (Timing:
> ~15s)
> 
> So my conclusion:
> 
> This is not a bug, it's about features. Open question is, if we return all
> images or only images owned by current user or all Amazon images.
> 
> My opinion is to follow 'aws' library documentation and use listing in this
> way:
> 
> <snip>
> ec2.describe_images_by_executable_by('522821470517')
> ec2.describe_images_by_executable_by('self')
> ec2.describe_images_by_executable_by('all')
> </snip>
> 
> So the 'self', 'all' and ID parameter can by specified by 'owner_id' param in
> Deltacloud API and default would be 'self'.

Questions:

1, What is the distinction between all images vs all Amazon images? (is this ami vs aki/ari??)
2. What is the valid set of images an end-user would want to and "could" create instances from (self only? Amazon??)
3. What happens when the number of images from question 2 is large?  Should there be a way to indicate the list is truncated to the caller and should there be a way to say: "I want images starting with image_id=xxxx"?  Then at least the caller could elect to make multiple calls in such an instance after realizing the list is incomplete.

Moving to Cloud Engine/Core/!.0 at the request of cpelland

Dave

Michal, what's the status here?  Did we ever fix this?

Chris Lalancette

*** Bug 641405 has been marked as a duplicate of this bug. ***

closing out old bugs that dev has not addressed :(

perm close

The Deltacloud API documentation says the following:

%%%%%%%%%%%%%%
GET /api/images
Return a list of all images available in the back-end cloud. By default this call will return all images that are available to the given user account. 
Optionally a client may restrict the list of images returned by specifying the owner_id or architecture parameters in the request (architecture is one of x86_64 for 64-bit processors or i386 for 32-bit processors). 
%%%%%%%%%%%%%%

So by default, seems as if the call returns all images an account *can access* - which is different from all images the user associated with the account owns. In my cleaned out account, I do not own any images yet I have access to a number that I can create instances from:
[rlandy@localhost server]$ curl -X GET --user "XXX:XX" -F "owner_id=421842149539" "http://localhost:3003/api/images?format=xml"
<?xml version='1.0' encoding='utf-8' ?>
<images>
</images>
[rlandy@localhost server]
[rlandy@localhost server]$ curl -X GET --user "XX:XX" -F "owner_id=amazon" "http://localhost:3003/api/images?format=xml" | more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 14  109k   14 16003  100   149   2886     26  0:00:38  0:00<?xml version='1.0' encoding='utf-8' ?>
:05<images>
curl -X GET --user "XX:XX"  "http://localhost:3003/api/images?format=xml" |more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 14  109k   14 16002    0     0   4103  <?xml version='1.0' encoding='utf-8' ?>
 
 AWS differenciates:  Amazon images as 238, All images 8069. API does not seem to pick up the difference in adding -F "owner_id=amazon" (although it does recognize amazon as a valid owner_id.
 
 Either way: this comment does not verify:
 "
> So the 'self', 'all' and ID parameter can by specified by 'owner_id' param in
> Deltacloud API and default would be 'self'.
"
'self' in this case would be zero.

You can limit the list by user_id and architecture but the last question, on how to handle the output if even the limited list is long, is open.

Yeah, the tricky part here is that, really the full list of 8000+ images seems like the valid output from /api/images -- i.e. those are the images that the user can validly use to launch. We don't want to restrict users to only images owned by either themselves or by amazon.

Is there some possibility in the API to restrict the content returned if the list is larger than a certain size -- i.e. if there are more than 1000 images, returned a more limited set of summary information? I'm not sure if this would solve the problem though. Or return an error saying that the list is too long, please provider more search filters?

I would suggest to fix our documentation:

%%%%%%%%%%%%%%
GET /api/images
Return a list of images available in the back-end cloud.
Optionally a client may restrict the list of images returned by specifying the
owner_id or architecture parameters in the request (architecture is one of
x86_64 for 64-bit processors or i386 for 32-bit processors). 
%%%%%%%%%%%%%%

What do you think?

moving version to 1.0.0 .  version = found in version

Given the age of this issue, can we get a next step or possibly CLOSE_WONT_FIX or CLOSED_CURRENT_RELEASE as appropriate.