Bug 1496847

Summary: Failed CompleteMultipartUpload request does not release lock
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Casey Bodley <cbodley>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.0CC: anharris, cbodley, ceph-eng-bugs, gmeno, kbader, kdreyer, mbenjamin, owasserm, sweil, tserlin, vakulkar
Target Milestone: rc   
Target Release: 2.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: RHEL: ceph-10.2.7-44.el7cp Ubuntu: ceph_10.2.7-44redhat1 Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of:
: 1497853 (view as bug list) Environment:
Last Closed: 2017-10-17 18:12:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1497853    
Bug Blocks:    

Description Casey Bodley 2017-09-28 15:42:23 UTC
Description of problem:

A rados lock is used to prevent racing CompleteMultipartUpload requests on the same object. But if we fail the request (due to invalid arguments, missing parts, etc), the lock is not released.

This causes later CompleteMultipartUpload requests on that object to fail with '500 Internal Error' and log:

> RGWCompleteMultipart::execute() failed to acquire lock

Version-Release number of selected component (if applicable):

ceph-12.2.0-2.el7cp

How reproducible:

After a failed CompleteMultipartUpload request, all later CompleteMultipartUpload requests on that object will fail with '500 Internal Error' for a duration of rgw_mp_lock_max_time (default: 10 minutes)

Steps to Reproduce:
1. Create an object
2. Issue a CompleteMultipartUpload on that object with an invalid upload id
3. Reissue the same invalid CompleteMultipartUpload request

Actual results:

The first request fails with 404 NoSuchUpload, and the second request fails with this 500 InternalError:

botocore.exceptions.ClientError: An error occurred (InternalError) when calling the CompleteMultipartUpload operation (reached max retries: 4): This multipart completion is already in progress

Expected results:

Both requests should fail with 404 NoSuchUpload.

Additional info:

Comment 8 Christina Meno 2017-10-02 20:54:20 UTC
When can we expect the patch downstream?

Comment 21 errata-xmlrpc 2017-10-17 18:12:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2903