Bug 2253015

Summary:	deadlock in DeleteObjects when multisite is enabled
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Casey Bodley <cbodley>
Component:	RGW-Multisite	Assignee:	Casey Bodley <cbodley>
Status:	CLOSED ERRATA	QA Contact:	Hemanth Sai <hmaheswa>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	ceph-eng-bugs, cephqe-warriors, hmaheswa, kdreyer, rmandyam, rpollack
Target Milestone:	---
Target Release:	7.0z1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-18.2.0-147	Doc Type:	Bug Fix
Doc Text:	Previously, during object deletion, the Ceph Object Gateway S3 DeleteObjects run together with a multi-site deployment, causing the Ceph Object Gateway to deadlock and stop accepting new requests. This was caused by the DeleteObjects requests processing several object deletions at a time. With this fix, the replication logs are serialized and the deadlock is prevented.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2024-03-07 11:40:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2260311

Description Casey Bodley 2023-12-05 15:46:01 UTC

Description of problem:

the DeleteObjects operation has fairly new support for concurrency, with the configurable rgw_multi_obj_del_max_aio which defaults to 16 objects at a time

in a multisite configuration where deletes trigger datalog entries, these concurrent datalog writes can lead to deadlocks under RGWDataChangesLog::add_entry()

this was reported upstream in https://tracker.ceph.com/issues/63373 where the reproducer was a "warp s3 benchmark with around 10 clients"

it was also reported that the deadlocks go away when rgw_multi_obj_del_max_aio=1

Comment 1 RHEL Program Management 2023-12-05 15:46:13 UTC

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 9 errata-xmlrpc 2024-03-07 11:40:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 7.0 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:1214

Comment 10 J. Eric Ivancich 2024-03-22 22:54:58 UTC

*** Bug 2256668 has been marked as a duplicate of this bug. ***