Bug 1313251

Summary:	RFE: rados tool should fail with ENPSPC when maximum objects quota exceeded
Product:	Red Hat Ceph Storage	Reporter:	Lubos Trilety <ltrilety>
Component:	RADOS	Assignee:	Josh Durgin <jdurgin>
Status:	CLOSED DEFERRED	QA Contact:	ceph-qe-bugs <ceph-qe-bugs>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	1.3.2	CC:	anharris, ceph-eng-bugs, ceph-qe-bugs, dzafman, kchai, kdreyer, nlevine, uboppana, vakulkar
Target Milestone:	rc	Keywords:	FutureFeature
Target Release:	3.*
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-10.2.0-1.el7cp	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-01-30 14:58:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Lubos Trilety 2016-03-01 09:09:03 UTC

Description of problem:
It is possible to exceed maximum object quota.

Version-Release number of selected component (if applicable):
ceph-0.94.5-9.el7cp.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Create some pool (e.g. test), set maximum objects quota (e.g. 20 objects), populate the pool so the objects quota is reached

2. create some small file
# echo 'test' > test.txt

3. try to put the file to the pool
# rados -p test put testobject test.txt
error putting test0/testobject: (28) No space left on device

4. ask for pool usage
# rados -p test df
pool name KB objects ...
test 1 20 ...
...

5. try to put the file to the pool
# rados -p test put testobject test.txt

6. check pool usage and get the created object from the pool
# rados -p test df
pool name KB objects ...
test 1 21 ...
...
# rados -p test get testobject test_it.txt
# cat test_it.txt
test

Actual results:
Quota is exceeded. Moreover the error message is not correct, there is enough space on the device, but the objects quota is reached.

Expected results:
The maximum objects quota cannot be exceeded and more proper error message is printed.

Additional info:

Comment 2 Samuel Just 2016-03-01 17:23:03 UTC

Yep, easy to reproduce.  I've created an upstream ticket.  We'll presumably get to it during the jewel stabilization phase.

Comment 3 Samuel Just 2016-03-01 17:24:30 UTC

Also, I think ENOSPC is a reasonable return value here.

Comment 4 Samuel Just 2016-03-01 17:29:48 UTC

Just looked at master, it seems to be fixed.  I don't think it's a good idea to backport it -- it required client side changes as well.  I'm marking this for 2.0.

Comment 6 Mike McCune 2016-03-28 23:35:19 UTC

This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions

Comment 7 Ken Dreyer (Red Hat) 2016-05-16 21:30:33 UTC

Based on Sam's comment #4 above and the upstream tracker ticket, this fix will be in RHCS 2 only (not RHCS 1.3).

Comment 10 Hemanth Kumar 2016-05-31 10:05:01 UTC

The error message is still not clear once the max object quota is reached.
The output should exit with a valid error message stating that the max object quota is exceeded. But, it gets stuck and the output is also not clear.

[root@magna020 ~]# rados -p pool3 put testobject11 test.txt --cluster master
2016-05-30 12:56:07.413123 7f2e7c87fa40  0 client.40291.objecter  FULL, paused modify 0x7f2e7cc61f50 tid 0
^C

Comment 11 Samuel Just 2016-06-02 18:30:17 UTC

The "getting stuck" there is deliberate.  It's waiting for the cluster to become not full.  Do we really want applications to get ENOSPC in that case?

Comment 12 Neil Levine 2016-06-02 18:50:48 UTC

This is an RFE not a bug for 2.0. Please modify subject or close this BZ and open a new one.

Comment 13 Hemanth Kumar 2016-06-03 10:05:11 UTC

(In reply to Samuel Just from comment #11)
> The "getting stuck" there is deliberate.  It's waiting for the cluster to
> become not full.  Do we really want applications to get ENOSPC in that case?

Sam,

I think Yes, If you are using any automated scripts it doesn't return if it gets stuck. SO it's better to provide the ENOSPC in such cases.

Comment 14 Samuel Just 2016-06-06 20:20:09 UTC

Yes, but it would also return ENOSPC to librbd, which would be really bad.  I think this is an RFE to improve the rados tool to not block in that case since we definitely don't want to return ENPSPC to most clients.

Comment 18 Drew Harris 2019-01-30 14:58:31 UTC

I have closed this issue because it has been inactive for some time now. If you feel this still deserves attention feel free to reopen it.