Bug 849756 - UFO swift large object support, large object creation
UFO swift large object support, large object creation
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: gluster-swift (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Luis Pabón
pushpesh sharma
: FutureFeature
Depends On:
  Show dependency treegraph
Reported: 2012-08-20 15:08 EDT by Kaleb KEITHLEY
Modified: 2016-11-08 17:24 EST (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-07-29 16:54:48 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Kaleb KEITHLEY 2012-08-20 15:08:10 EDT
Description of problem:

Blocker BZ for RHS 2.1

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
Actual results:

Expected results:

Additional info:
Comment 2 Kaleb KEITHLEY 2012-09-05 13:49:00 EDT

Objects that are larger than 5GB must be segmented, prior to upload. You then upload the segments like you would any other object and create a manifest object telling OpenStack Object Storage how to find the segments of the large object. The segments remain individually addressable, but retrieving the manifest object streams all the segments concatenated. There is no limit to the number of segments that can be a part of a single large object.

In order to ensure the download works correctly, you must upload all the object segments to the same container, ensure each object name has a common prefix where their names sort in the order they should be concatenated. You also create and upload a manifest file. The manifest file is simply a zero-byte file with the extra X-Object-Manifest: <container>/<prefix> header, where <container> is the container the object segments are in and <prefix> is the common prefix for all the segments.

It is best to upload all the segments first and then create or update the manifest. With this method, the full object will not be available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and then update the manifest to point to this new location. During the upload of the new segments, the original manifest will still be available to download the first set of segments.
Comment 3 Kaleb KEITHLEY 2012-09-14 08:44:13 EDT
What is being asked for here?

I have confirmed — empirically — that it's possible to upload files larger than 5GB with the current RHS/glusterfs release (3.3.0, swift 1.4.8).

Are we looking for some under-the-covers mechanism in RHS to segment large files and create the manifest object on a large file write? And the reverse on a read?
Comment 4 Kaleb KEITHLEY 2012-09-14 10:12:42 EDT
/bin/swift has the option to specify a segment size, e.g., storing a large file in the 'images' container in segments:

% swift -A https://f17node1:443/auth/v1.0 -U vol0:kaleb -K testing upload --segment-size 1000000000 images $sixgigfile

splits the file into $segment-sized files and stores them on the swift/gluster volume in /images_segments/$sixgigfile/.../* plus a manifest in /images/$sixgigfile

Versus, e.g. as a single unsegmented file:

% swift -A https://f17node1:443/auth/v1.0 -U vol0:kaleb -K testing upload images $sixgigfile

which stores the file in /images/$sixgigfile

To download, regardless of whether the file is segmented or not:

% swift -A https://f17node1:443/auth/v1.0 -U vol0:kaleb -K testing download images $sixgigfile
Comment 5 Kaleb KEITHLEY 2012-09-24 11:38:45 EDT
(In reply to comment #3)
> I have confirmed — empirically — that it's possible to upload files larger
> than 5GB with the current RHS/glusterfs release (3.3.0, swift 1.4.8).

N.B. that /usr/lib/python2.7/site-packages/swift/common/constraints.py, e.g., has:

   MAX_FILE_SIZE = 5 * 1024 * 1024 * 1024 + 2

Which we appear to over-ride in our /usr/lib/python2.7/site-packages/swift/plugins/constraints.py add-on to:

   MAX_FILE_SIZE = 0xffffffffffffffff
Comment 6 Luis Pabón 2013-07-29 16:54:48 EDT
There does not seem to be a bug here.  If there is please reopen.

Note You need to log in before you can comment on or make changes to this bug.