DescriptionShravan Kumar Tiwari
2021-02-05 11:56:32 UTC
Description of problem:
When trying to upload a multi-part upload to radosgw. it fails with error "416(InvalidRange)"
Version-Release number of selected component (if applicable):
ceph 4.1.2
How reproducible:
When uploading a multi-part upload to radosgw, the completion of that is done by a HTTP POST which contains an XML structure that defines which parts Radosgw needs to put together to complete the multipart upload.
On a 39GB upload with 5MB chunk size, that yields about 7877 parts. That is under the default rgw_multipart_part_upload_limit of 10000. However, we were getting 416 errors (-ERANGE) even under these circumstances.
What it turns out to be is that the size of the POSTed XML exceeds a hardcoded limit, which effectively results in the multipart upload not being able to complete well before the 10k part upload limit was hit.
The hardcoded limit is 1MB: https://github.com/ceph/ceph/blob/956ceb853a58f6b6847b31fac34f2f0228a70579/src/rgw/rgw_rest.cc#L1635
#define COMPLETE_MULTIPART_MAX_LEN (1024 * 1024) /* api defines max 10,000 parts, this should be enough */
The XML that contains the structure for those 7877 parts is 1062389 bytes, and thus it throws -ERANGE.
Steps to Reproduce:
1.
2.
3.
Actual results:
upload is failing with error "416(InvalidRange)"
Expected results:
Upload should be successful and parameter setting for rgw_max_put_param_size should be made available to user.
Additional info:
- Upstream bug report: https://tracker.ceph.com/issues/38002
- Pull request with fix: https://github.com/ceph/ceph/pull/26070/files (needs trivial adaptation for Luminous)
After patching radosgw with above code and then raising rgw_max_put_param_size to 4194304, the multipart upload completes successfully.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix Update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2021:2445