Bug 2253015 - deadlock in DeleteObjects when multisite is enabled
Summary: deadlock in DeleteObjects when multisite is enabled
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW-Multisite
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 7.0z1
Assignee: Casey Bodley
QA Contact: Hemanth Sai
URL:
Whiteboard:
: 2256668 (view as bug list)
Depends On:
Blocks: 2260311
TreeView+ depends on / blocked
 
Reported: 2023-12-05 15:46 UTC by Casey Bodley
Modified: 2024-03-22 22:54 UTC (History)
6 users (show)

Fixed In Version: ceph-18.2.0-147
Doc Type: Bug Fix
Doc Text:
Previously, during object deletion, the Ceph Object Gateway S3 DeleteObjects run together with a multi-site deployment, causing the Ceph Object Gateway to deadlock and stop accepting new requests. This was caused by the DeleteObjects requests processing several object deletions at a time. With this fix, the replication logs are serialized and the deadlock is prevented.
Clone Of:
Environment:
Last Closed: 2024-03-07 11:40:22 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 63373 0 None None None 2023-12-05 15:46:01 UTC
Ceph Project Bug Tracker 63528 0 None None None 2023-12-05 15:46:01 UTC
Red Hat Issue Tracker RHCEPH-8004 0 None None None 2023-12-05 15:48:14 UTC
Red Hat Product Errata RHBA-2024:1214 0 None None None 2024-03-07 11:40:26 UTC

Description Casey Bodley 2023-12-05 15:46:01 UTC
Description of problem:

the DeleteObjects operation has fairly new support for concurrency, with the configurable rgw_multi_obj_del_max_aio which defaults to 16 objects at a time

in a multisite configuration where deletes trigger datalog entries, these concurrent datalog writes can lead to deadlocks under RGWDataChangesLog::add_entry()

this was reported upstream in https://tracker.ceph.com/issues/63373 where the reproducer was a "warp s3 benchmark with around 10 clients"

it was also reported that the deadlocks go away when rgw_multi_obj_del_max_aio=1

Comment 1 RHEL Program Management 2023-12-05 15:46:13 UTC
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 9 errata-xmlrpc 2024-03-07 11:40:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:1214

Comment 10 J. Eric Ivancich 2024-03-22 22:54:58 UTC
*** Bug 2256668 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.