Bug 1041737

Summary: [RFE][glance]: Add multifilesystem store to support NFS servers as backend
Product: Red Hat OpenStack Reporter: RHOS Integration <rhos-integ>
Component: openstack-glanceAssignee: Flavio Percoco <fpercoco>
Status: CLOSED ERRATA QA Contact: Dafna Ron <dron>
Severity: medium Docs Contact:
Priority: medium    
Version: unspecifiedCC: adahms, eglynn, fpercoco, gfidente, markmc, nlevinki, pbandark, scohen, slong, sputhenp, tshefi, yeylon
Target Milestone: Upstream M3Keywords: FutureFeature, Triaged
Target Release: 5.0 (RHEL 7)   
Hardware: Unspecified   
OS: Unspecified   
URL: https://blueprints.launchpad.net/glance/+spec/glance-multifilesystem-store
Whiteboard: upstream_milestone_icehouse-3 upstream_status_implemented upstream_definition_approved
Fixed In Version: openstack-glance-2014.1-2.el7ost Doc Type: Enhancement
Doc Text:
Previously, it was not possible to mount all disks to a single directory when configuring multiple NFS servers as a backend using the filesystem store. This was due to the filesystem store only allowing administrators to configure a single directory using the filesystem_store_datadir parameter in the glance-api.conf file. While it is possible to use MHDDFS (a FUSE plug-in: https://romanrm.net/mhddfs), which mounts multiple NFS servers to a single directory, MHDDFS does not allow you to evenly store the data on all the disks. Another major drawback is that it is very difficult to know the number and type of images stored on a disk when one of the disks is broken because the Glance registry stores the location specified in the filesystem_store_datadir parameter. This enhancement fixes the above issues by adding multi-filesystem support to the current filesystem store.
Story Points: ---
Clone Of:
: 1081957 (view as bug list) Environment:
Last Closed: 2014-07-08 15:31:48 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1081957    

Description RHOS Integration 2013-12-12 19:06:22 UTC
Cloned from launchpad blueprint https://blueprints.launchpad.net/glance/+spec/glance-multifilesystem-store.

Description:

Description:

Currently if you want to configure multiple nfs server as a backend using filesystem store, you cannot mount all disks to a single directory. filesystem store allows administrator to configure only single directory with filesystem_store_datadir parameter in the glance-api.conf.
It is possible to use mhddfs (fuse plugin :https://romanrm.net/mhddfs) which mounts multiple nfs servers to a single directory but it doesn't allow you to evenly store the data on all the disks. Another major drawback is when one of the disk is broken it is very hard to know how many and what images are stored on that disk as glance registry stores location specified in the filesystem_store_datadir parameter.

With multifilesystem, it is possible to over come above issues.

We are proposing to do following things.
1. Add a new store multifilesystem.
2. Add new parameters in the glance-api.conf

Example

3 nfs servers, exporting 1TB of disks
10.2.3.10
10.2.3.11
10.2.3.12

On Glance server,
Administrator will mount complete export tree as below

10.2.3.10:/export/nfs-partition-1  /var/glance/store-1
10.2.3.11:/export/nfs-partition-1  /var/glance/store-2
10.2.3.12:/export/nfs-partition-1  /var/glance/store-3

Edit glance-api.conf

Add new parameter multifilesystem_store_datadir to configure multiple directories.
#multifilesystem_store_datadir = <directory>:<priority number>
multifilesystem_store_datadir = /var/glance/store-1:100,/var/glance/store-2:200,/var/glance/store-3:300

Priority number: Each directory will be given a priority number.
Priority number with less value will be given top priority. If all directories is given same priority number, then the disk with more available space will be selected for storing image.

Specification URL (additional information):

None

Comment 9 Tzach Shefi 2014-05-13 14:44:57 UTC
launchpad source (and RFE here) includes three errors:

1. This is wrong doesn't work:
multifilesystem_store_datadir = /var/glance/store-1:100,/var/glance/store-2:200,/var/glance/store-3:300

Using this comma separated format -> only one dir is created at this path  "/var/glance/store-1:100,/var/glance/store-2:200,/var/glance/store-3:300" all images are created under this folder, parsing doesn't detect comma. 

The correct format is:  
multifilesystem_store_datadirs= /var/glance/store-1:100    
multifilesystem_store_datadirs= /var/glance/store-2:200
...

2. Note _datadirs instead of  _datadir as appears on launchpad, missing an s. 

3.  filesystem_store_datadir parameter's default value "/var/lib/glance/images" must be cleared when using multifilesystem. This isn't emphasized or clear on conf file description.

Comment 13 errata-xmlrpc 2014-07-08 15:31:48 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHEA-2014-0851.html