Bug 1609959

It's normal to see curl requests for a Glance URL that does not have a "/file" suffix. That is how the client (cinder) requests the image's metadata.

Updating the title because the issue is in the SolidFire driver.

I traced the problem to an issue with the Cinder SolidFire driver's "template
caching" feature. The non-admin user is the owner of the Glance image, and so
the driver attempts to create an internal cache of the image, which it then
clones to create the new volume requested by the user. But, when the cache
entry is created, two errors occur that result in cache entry consisting of a
blank volume. When the user makes a subsequent request for another volume from
that image, the blank cache entry is cloned, resulting in another blank
volume. Here is the overall sequence that results in a blank entry in the
template cache:

1. SolidFire driver determines the image has not been cached, so it downloads
   the image from Glance to a temporary location, creates a new (blank) volume
   on the SolidFire backend, and tries to copy the image data to the SolidFire
   volume. This volume represents the "template cache" entry.

2. The first error occurs when the driver attempts to create an iSCSI
   connection to the "template cache" volume that it just created. The error I
   see in the sosreports is:

iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)

3. The driver attempts to clean up by dropping any iSCSI connections to the
   "template cache" volume, deleting the volume, and purging the "template
   cache" entry.

4. The second error occurs when it tries to drop any iSCSI connections:

UnboundLocalError: local variable 'attach_info' referenced before assignment

5. The second error is not handled properly, and a side effect is the blank
   "template cache" volume is never deleted or purged from the cache.

All of this results in a blank "template cache" volume, so subsequent requests
to create more volumes from the same image result in more blank volumes.

When an admin attempts to create a volume from the same image, because the
image is not owned by the admin, the SolidFire driver does not attempt to use
the template caching feature. When this happens, Cinder downloads a fresh copy
of the Glance image, and does not rely on the SolidFire driver to do any
cloning. This explains why you can observe the new volume's status cycling
from 'creating' to 'downloading' to 'available'.

So, one suggestion is to troubleshoot the iSCSI login failure, which will
likely require examining logs on the SolidFire backend.

A better approach would be to disable the driver's template caching feature,
by setting 'sf_allow_template_caching = False' in the driver's section in
cinder.conf. The whole feature was deprecated by the vendor in Queens because
it essentially duplicates Cinder's own "image volume cache" feature. If image
caching is a performance requirement, then see [1] to enable the feature.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/storage_guide/ch-cinder#section-cinder-image-cache

In light of the fact that the vendor deprecated their own image caching feature in Queens [1] (and that change is already in OSP-13), then the cleanest way to resolve issues like this is to backport the vendor's patch to this OSP release.

[1] https://review.openstack.org/511623

Driver related issue, no access to SolidFire to verify/automate. 
Setting OtherQA and setting close loop flags.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0079