Bug 2342928 - Unable to Delete Objects when RADOS Pool quota hits Max
Summary: Unable to Delete Objects when RADOS Pool quota hits Max
Keywords:
Status: VERIFIED
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 8.1
Assignee: Matt Benjamin (redhat)
QA Contact: Vidushi Mishra
Rivka Pollack
URL:
Whiteboard:
Depends On:
Blocks: 2351689 2345560
TreeView+ depends on / blocked
 
Reported: 2025-01-30 06:57 UTC by daniel parkes
Modified: 2025-05-27 15:25 UTC (History)
11 users (show)

Fixed In Version: ceph-19.2.1-116.el9cp
Doc Type: Bug Fix
Doc Text:
.Ceph Object Gateway can now delete objects when RADOS is at maximum pool capacity Previously, when a RADOS pool was near its maximum quota, the Ceph Object Gateway was not able to delete objects. With this fix, Ceph Object Gateway can delete objects even when RADOS has reached its maximum pool threshold.
Clone Of:
: 2345560 (view as bug list)
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 69723 0 None None None 2025-01-30 21:08:35 UTC
Github ceph ceph pull 61650 0 None open rgw/rados: enable object deletion at rados pool quota 2025-02-04 15:28:31 UTC
Red Hat Issue Tracker RHCEPH-10540 0 None None None 2025-01-30 06:58:01 UTC

Description daniel parkes 2025-01-30 06:57:38 UTC
Description of problem:

Issue Hit by the IBM Ceph as a service Object Storage team. They use RADOS pool quotas to enforce a quota on the RGW data pool at the cluster level.

"When an S3 data pool has quota configured, and enough data is uploaded to the store to hit the quota max, we cannot delete objects to drop below the quota max and enable S3 I/O to continue."

This current behavior freezes all customer IOs until they open a support ticket, and an SRE works on the ticket to expand the pool quota for the end user, which is not a good experience.

I did some quick testing, and it seems this behavior happens at the Rados level, not in the RGW client:

ceph df detail | grep testpool
testpool 27 32 62 MiB 62 MiB 0 B 13 186 MiB 186 MiB 0 B 0.10 62 GiB N/A 48 MiB N/A 0 B 0 B
And I can't delete any object at the rados level until I increase the quota size; the rm just hangs.

[root@ceph-node-04 ~]# rados -p testpool rm largeobj2
^C

Once I increase the quota, I can delete:

[root@ceph-node-04 ~]# ceph osd pool set-quota testpool max_bytes 80485760
set-quota max_bytes = 80485760 for pool testpool
[root@ceph-node-04 ~]# rados -p testpool rm largeobj
[root@ceph-node-04 ~]#

Thanks


Note You need to log in before you can comment on or make changes to this bug.