Bug 849756

http://docs.openstack.org/api/openstack-object-storage/1.0/content/large-object-creation.html

Objects that are larger than 5GB must be segmented, prior to upload. You then upload the segments like you would any other object and create a manifest object telling OpenStack Object Storage how to find the segments of the large object. The segments remain individually addressable, but retrieving the manifest object streams all the segments concatenated. There is no limit to the number of segments that can be a part of a single large object.

In order to ensure the download works correctly, you must upload all the object segments to the same container, ensure each object name has a common prefix where their names sort in the order they should be concatenated. You also create and upload a manifest file. The manifest file is simply a zero-byte file with the extra X-Object-Manifest: <container>/<prefix> header, where <container> is the container the object segments are in and <prefix> is the common prefix for all the segments.

It is best to upload all the segments first and then create or update the manifest. With this method, the full object will not be available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and then update the manifest to point to this new location. During the upload of the new segments, the original manifest will still be available to download the first set of segments.

What is being asked for here?

I have confirmed — empirically — that it's possible to upload files larger than 5GB with the current RHS/glusterfs release (3.3.0, swift 1.4.8).

Are we looking for some under-the-covers mechanism in RHS to segment large files and create the manifest object on a large file write? And the reverse on a read?

/bin/swift has the option to specify a segment size, e.g., storing a large file in the 'images' container in segments:

% swift -A https://f17node1:443/auth/v1.0 -U vol0:kaleb -K testing upload --segment-size 1000000000 images $sixgigfile

splits the file into $segment-sized files and stores them on the swift/gluster volume in /images_segments/$sixgigfile/.../* plus a manifest in /images/$sixgigfile

Versus, e.g. as a single unsegmented file:

% swift -A https://f17node1:443/auth/v1.0 -U vol0:kaleb -K testing upload images $sixgigfile

which stores the file in /images/$sixgigfile

To download, regardless of whether the file is segmented or not:

% swift -A https://f17node1:443/auth/v1.0 -U vol0:kaleb -K testing download images $sixgigfile

(In reply to comment #3)
> 
> I have confirmed — empirically — that it's possible to upload files larger
> than 5GB with the current RHS/glusterfs release (3.3.0, swift 1.4.8).

N.B. that /usr/lib/python2.7/site-packages/swift/common/constraints.py, e.g., has:

   MAX_FILE_SIZE = 5 * 1024 * 1024 * 1024 + 2

Which we appear to over-ride in our /usr/lib/python2.7/site-packages/swift/plugins/constraints.py add-on to:

   MAX_FILE_SIZE = 0xffffffffffffffff

There does not seem to be a bug here.  If there is please reopen.