Description of problem:

Issue Hit by the IBM Ceph as a service Object Storage team. They use RADOS pool quotas to enforce a quota on the RGW data pool at the cluster level.

"When an S3 data pool has quota configured, and enough data is uploaded to the store to hit the quota max, we cannot delete objects to drop below the quota max and enable S3 I/O to continue."

This current behavior freezes all customer IOs until they open a support ticket, and an SRE works on the ticket to expand the pool quota for the end user, which is not a good experience.

I did some quick testing, and it seems this behavior happens at the Rados level, not in the RGW client:

ceph df detail | grep testpool
testpool 27 32 62 MiB 62 MiB 0 B 13 186 MiB 186 MiB 0 B 0.10 62 GiB N/A 48 MiB N/A 0 B 0 B
And I can't delete any object at the rados level until I increase the quota size; the rm just hangs.

[root@ceph-node-04 ~]# rados -p testpool rm largeobj2
^C

Once I increase the quota, I can delete:

[root@ceph-node-04 ~]# ceph osd pool set-quota testpool max_bytes 80485760
set-quota max_bytes = 80485760 for pool testpool
[root@ceph-node-04 ~]# rados -p testpool rm largeobj
[root@ceph-node-04 ~]#

Thanks