It sounds like some nodes are misconfigured.

As a wild guess, I'd say one of the Glance nodes is configured to use the file store, whereas the rest of the nodes are using Swift. A snapshot/image was created from the node using the file store and then the failure happens when you try to boot an instance using that image from one of the other nodes.

This is, of course, a wild guess. Is it possible to have the config files and some more info about the environment?

This looks similar to https://bugs.launchpad.net/glance/+bug/1581111

Could you make sure "file" is part of the "stores" option of the "glance_store" section in all of your glance-api.conf configuration files?

The URI issue is also seen in the glance log :
https://gist.github.com/jtaleric/c34d96cbc141df275ba4cbe1723ea2f9

Here is the image in question :
https://gist.github.com/jtaleric/2379f1e4512bc163929698a17d805344

Here is reviewing a different image in glance :
https://gist.github.com/jtaleric/9541722e02066c46762142bf5bea3a7a

Could this because the Snapshot (file) vs Swift backend?

Exporting the image via glance image-download ( failed 2 out of the 3 times -- which would be expected since only a single controller had the snapshot). However, re-adding the image to glace so it was backed by swift seems to have fixed the issue.

Question is, why didn't this happen auto-magically?

Re-opening as we need to understand what we expect Snapshot to do in this case. Please refer to my comment. #4

(In reply to Joe Talerico from comment #6)
> Re-opening as we need to understand what we expect Snapshot to do in this
> case. Please refer to my comment. #4


I'm sorry, I'm having a bit of a hard time parsing this. What do you mean exactly? What happened with the snapshot? Could you share the configuration files? How did you take the snapshot?

Jeremy can you share how you took the snapshot?

Flavio, what exactly do you not understand?

Let me breaking this down...
1) Import a image into Glance.
2) Launch a guest, set some things up in the guest, take snapshot.
3) Create a bunch of guests from the above snapshot -- this will result in many failures. 

To fix this: 
1) Export snapshot from glance (this will fail 2 out of the 3 times you attempt, depending on where HAProxy sends the request)
2) Re-import image into Glance

The biggest difference between the snapshot image vs the re-imported image is the location of the image. The snapshot is file:// and the re-imported image is the swift:// location.

I clicked "Create Snapshot" in Horizon.

(In reply to Joe Talerico from comment #8)
> Jeremy can you share how you took the snapshot?
> 
> Flavio, what exactly do you not understand?
> 
> Let me breaking this down...
> 1) Import a image into Glance.
> 2) Launch a guest, set some things up in the guest, take snapshot.
> 3) Create a bunch of guests from the above snapshot -- this will result in
> many failures. 
> 
> To fix this: 
> 1) Export snapshot from glance (this will fail 2 out of the 3 times you
> attempt, depending on where HAProxy sends the request)
> 2) Re-import image into Glance
> 
> The biggest difference between the snapshot image vs the re-imported image
> is the location of the image. The snapshot is file:// and the re-imported
> image is the swift:// location.


As I've mentioned already in the email thread and earlier in this BZ, the above sounds like there's a misconfigured glance-api node. Could you please upload the config files for these glance-api nodes and/or verify that they are all configured the same way.

It really sounds like this snapshot was created on a glance-api node that was using the `file` store. On the upload the node saved the image locally and all other glance-api nodes fail to download it because they're not configured to use such store. Now it might be working because it's pointing to swift where, apparently, all glance-api nodes have access to.

We are going to close this one out given it happened some time ago.  If there are still similar or new issues getting something like this configured or tested in OSP11, please open a new bug. Thanks.