Previously, 'Content-Length' of the object which was stored as metadata in the extended attribute was not validated with actual size of object during processing of a GET request. As a consequence, when an object was modified from file interface and later accessed over Swift interface, the Swift client used to either receive incomplete/inconsistent data or the request used to fail entirely. As a fix, now a check is made if the 'Content-Length' stored as metadata is same as the actual size of the object. If not same, the stored metadata is invalidated and the stored Content-Length is updated. Wit this fix, the entire object data is returned to the client and the request completes successfully.
Description of problem:
Inconsistent or incomplete data is returned to a swift client when objects are modified from file interface and a GET is performed on such an object.
Version-Release number of selected component (if applicable):
RHS 3.1
swiftonfile-1.13.1-2
How reproducible:
Always, consistently reproducible
Steps to Reproduce:
1. Create a file from the file interface or from the object interface
2. initially file downloads correctly with curl or swift client
3. modify the file on the file side by appending to the end of the file, for example "echo test >> obj.txt". This leaves the swift xattrs unchanged, but they are now showing the wrong size and etag.
4. download the file - the appended data is not downloaded. This is because content-length in the http request is set to the original size, so only the original N bytes of the file are downloaded. The etag for those bytes is correct still, so no message is logged by swift client.
5. modify the file by re-writing it to be shorter than the original object, without modifying the xattrs. For example, "echo t > obj.txt".
6. download hangs because content-length header is still set to the original size, and get request hangs waiting for content-length bytes to be delivered.
7. modify the file again by appending to it so that it's now longer than the original size, for example "date >> obj.txt", repeated until file length is larger than original object length stored in metadata.
8. download using curl successfully downloads the first N bytes of the object data, but etag is not correct (but not checked on the server side). The extra bytes are not downloaded.
9. download using swift client. The first N bytes of the object data are successfully downloaded, but the swift client logs a message about etag mismatch.
10. modify the file by so that xattrs are wiped out (for example, vi & save the file).
download returns the correct object data because new xattrs are calculated based on file contents in this case.
Actual results:
Inconsistent (or incomplete) data returned when objects are modified from file interface.
Expected results:
* Entire content of object must be returned as is
Upstream bug:
https://bugs.launchpad.net/swiftonfile/+bug/1416720
Upstream fix under review:
https://review.openstack.org/#/c/151897
Verified with glusterfs-3.7.1-13.el7rhgs and swiftonfile-1.13.1-4.el7rhgs.noarch with the following steps :
1. Create a object using cURL (PUT)
2. Appended the file from fuse mount
3. Got the modified object using cURL ( GET )
4. Created a object again using cURL (PUT)
5. Truncated the content of the file from the fuse mount
6. Got the modified object using cURL ( GET )
The object created using cURL and the same when modified from the fuse mount - had a consistent view
Doc text with some minor corrections:
"Previously, 'Content-Length' of the object which was stored as metadata in the extended attribute was not validated with actual size of object during processing of a GET request. As a consequence, when an object was modified from file interface and later accessed over Swift interface, the Swift client used to either receive incomplete/inconsistent data or the request used to fail entirely. As a fix, now a check is made if the 'Content-Length' stored as metadata is same as the actual size of the object. If not same, the stored metadata is invalidated and the stored Content-Length is updated. Wit this fix, the entire object data is returned to the client and request completes successfully."
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHSA-2015-1845.html
Description of problem: Inconsistent or incomplete data is returned to a swift client when objects are modified from file interface and a GET is performed on such an object. Version-Release number of selected component (if applicable): RHS 3.1 swiftonfile-1.13.1-2 How reproducible: Always, consistently reproducible Steps to Reproduce: 1. Create a file from the file interface or from the object interface 2. initially file downloads correctly with curl or swift client 3. modify the file on the file side by appending to the end of the file, for example "echo test >> obj.txt". This leaves the swift xattrs unchanged, but they are now showing the wrong size and etag. 4. download the file - the appended data is not downloaded. This is because content-length in the http request is set to the original size, so only the original N bytes of the file are downloaded. The etag for those bytes is correct still, so no message is logged by swift client. 5. modify the file by re-writing it to be shorter than the original object, without modifying the xattrs. For example, "echo t > obj.txt". 6. download hangs because content-length header is still set to the original size, and get request hangs waiting for content-length bytes to be delivered. 7. modify the file again by appending to it so that it's now longer than the original size, for example "date >> obj.txt", repeated until file length is larger than original object length stored in metadata. 8. download using curl successfully downloads the first N bytes of the object data, but etag is not correct (but not checked on the server side). The extra bytes are not downloaded. 9. download using swift client. The first N bytes of the object data are successfully downloaded, but the swift client logs a message about etag mismatch. 10. modify the file by so that xattrs are wiped out (for example, vi & save the file). download returns the correct object data because new xattrs are calculated based on file contents in this case. Actual results: Inconsistent (or incomplete) data returned when objects are modified from file interface. Expected results: * Entire content of object must be returned as is Upstream bug: https://bugs.launchpad.net/swiftonfile/+bug/1416720 Upstream fix under review: https://review.openstack.org/#/c/151897