Description of problem:

When trying to upload a multi-part upload to radosgw. it fails with error "416(InvalidRange)"


Version-Release number of selected component (if applicable):
ceph 4.1.2


How reproducible:

When uploading a multi-part upload to radosgw, the completion of that is done by a HTTP POST which contains an XML structure that defines which parts Radosgw needs to put together to complete the multipart upload.

On a 39GB upload with 5MB chunk size, that yields about 7877 parts. That is under the default rgw_multipart_part_upload_limit of 10000. However, we were getting 416 errors (-ERANGE) even under these circumstances.

What it turns out to be is that the size of the POSTed XML exceeds a hardcoded limit, which effectively results in the multipart upload not being able to complete well before the 10k part upload limit was hit.

The hardcoded limit is 1MB: https://github.com/ceph/ceph/blob/956ceb853a58f6b6847b31fac34f2f0228a70579/src/rgw/rgw_rest.cc#L1635

#define COMPLETE_MULTIPART_MAX_LEN (1024 * 1024) /* api defines max 10,000 parts, this should be enough */

The XML that contains the structure for those 7877 parts is 1062389 bytes, and thus it throws -ERANGE.


Steps to Reproduce:
1.
2.
3.

Actual results:
upload is failing with error "416(InvalidRange)"

Expected results:
Upload should be successful and parameter setting for rgw_max_put_param_size should be made available to user.

Additional info:

- Upstream bug report: https://tracker.ceph.com/issues/38002
- Pull request with fix: https://github.com/ceph/ceph/pull/26070/files (needs trivial adaptation for Luminous)

After patching radosgw with above code and then raising rgw_max_put_param_size to 4194304, the multipart upload completes successfully.