Bug 1552583

Summary: [Glance] Interoperable image import should be disabled by default on 3 node HA setup
Product: Red Hat OpenStack Reporter: Christian Schwede (cschwede) <cschwede>
Component: openstack-tripleo-heat-templatesAssignee: Pranali Deore <pdeore>
Status: CLOSED ERRATA QA Contact: Mike Abrams <mabrams>
Severity: urgent Docs Contact: Kim Nylander <knylande>
Priority: urgent    
Version: 13.0 (Queens)CC: abishop, eglynn, fpercoco, jamsmith, jschluet, mabrams, mburns, pdeore, pgrist, scohen, srevivo, tshefi
Target Milestone: rcKeywords: Triaged
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: openstack-tripleo-heat-templates-8.0.2-21.el7ost puppet-glance-12.5.0-2.el7ost Doc Type: Bug Fix
Doc Text:
The glance-direct method requires a shared staging area when used in a HA configuration. Image uploads using the 'glance-direct' method may fail in an HA environment if a common staging area is not present. Incoming requests to the controller nodes are distributed across the available controller nodes. One controller handles the first step and another controller handles the second request with both controllers writing the image to different staging areas. The second controller will not have access to the same staging area used by the controller handling the first step. Glance supports multiple image import methods, including the 'glance-direct' method. This method uses a three-step approach: creating an image record, uploading the image to a staging area, and then transferring the image from the staging area to the storage backend so the image becomes available. In an HA setup (i.e., with 3 controller nodes), the glance-direct method requires a common staging area using a shared file system across the controller nodes. The list of enabled Glance import methods can now be configured. The default configuration does not enable the 'glance-direct' method (web-download is enabled by default). To avoid the issue and reliably import images to Glance in an HA environment, do not enable the 'glance-direct' method.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-06-27 13:35:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1552584    

Description Christian Schwede (cschwede) 2018-03-07 12:03:30 UTC
Glance added the support for interoperable image imports. Image uploads are done using a two-step approach: first uploading it to the Glance API using a local staging area, and then triggering an upload from the staging area to the storage backend.

In the case of an HA setup (ie with 3 controller nodes) this requires a shared consistent filesystem across the controller nodes. Otherwise it might happen that the image is uploaded to one node, and triggering the import is executed on a different node - where the image does not exist.

Therefore we need to disable this by default, thus operators need to enable it explicitly if they have a shared filesystem across the controller nodes (for example using NFS).

We need to do:
1. Set enabled_import_methods = web-download by default in glance-api.conf
2. Make this option changeable in t-h-t
3. Document this

Comment 6 Alan Bishop 2018-05-11 11:45:37 UTC
Moving to POST because upstream reviews have been approved on master (although one has not merged due to CI gate issues). Patches have been proposed to stable/queens.

Comment 8 Paul Grist 2018-05-11 20:43:48 UTC
Adding some more detail on the description. We definitely need to get the proper default in because there is another implication of the out of the box settings.

The shared filesystem will only come into picture if customer wants to use a new feature (image import with glance-direct method). It doesn't have any impact with traditional way of creating the image.

In the HA setup, there is default support to use the root disk for staging. So if a customer does use the image import features, there is a clear risk that the root disk will fill and cause issues.  The intended approach for OSP-13 is to disable it and customers can opt-in and will see the docs for adding the shared file system for HA configs.

Comment 9 Alan Bishop 2018-05-14 12:14:24 UTC
All (3 of 3) patches have been proposed downstream.

Comment 10 Alan Bishop 2018-05-14 20:59:05 UTC
One of the upstream patches was incorrect, and an update has been posted.

Comment 26 errata-xmlrpc 2018-06-27 13:35:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.