Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1505504

Summary:	space may be unreclaimable after inter-bucket copy (space leak)
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Matt Benjamin (redhat) <mbenjamin>
Component:	RGW	Assignee:	Matt Benjamin (redhat) <mbenjamin>
Status:	CLOSED ERRATA	QA Contact:	Ramakrishnan Periyasamy <rperiyas>
Severity:	high	Docs Contact:
Priority:	high
Version:	3.0	CC:	cbodley, ceph-eng-bugs, flucifre, hnallurv, kbader, kdreyer, mbenjamin, mkogan, owasserm, sweil, uboppana
Target Milestone:	rc	Keywords:	Regression
Target Release:	3.0
Hardware:	All
OS:	All
Whiteboard:
Fixed In Version:	RHEL: ceph-12.2.1-27.el7cp Ubuntu: ceph_12.2.1-30redhat1xenial	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-12-05 23:49:04 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Matt Benjamin (redhat) 2017-10-23 18:04:52 UTC

Description of problem:
Due to an object refcounting error, objects can become unreclaimable after being used as the source object in an inter-bucket copy, leaking space

How reproducible:
100%

Steps to Reproduce:
you can find the problem list this:
1. upload obj to bucket1
s3cmd put obj s3://bucket1
2. cp obj1 from bucket1 to bucket2
s3cmd cp s3://bucket1/obj s3://bucket2
3. del obj from bucket1 and bucket2
s3cmd del s3://bucket1/obj
s3cmd del s3://bucket2/obj
4. you can see the pool 'default.rgw.buckets.data', the data of obj can not deleted
ceph df | grep default.rgw.buckets.data

Comment 9 Federico Lucifredi 2017-10-25 14:55:18 UTC

Great job guys! Many thanks.

Comment 11 Ramakrishnan Periyasamy 2017-10-27 13:08:36 UTC

Moving the bug to Verified state. Thanks for the clarification Matt.

Verified the bug using s3_boto.

Steps followed to verify:
1. Created 2 buckets:

[ubuntu@magna100 ~]$ python s3test_get_bucket.py
aws_bucket    2017-10-27T09:34:34.723Z
bucket_1    2017-10-27T06:01:27.975Z
Buckets:  [<Bucket: aws_bucket>, <Bucket: bucket_1>]

2. Created one object in bucket_1:

[ubuntu@magna100 ~]$ python s3test_list_obj.py
<Key: aws_bucket,redhat1607329>
number of objects:  1
Objects listed
[ubuntu@magna090 ~]$ sudo rados ls -p default.rgw.buckets.data
78c96e27-2bfb-484a-a122-8a26cf1104db.4148.5_redhat1607329

3. Copied the object from one bucket to another (bucket_1 --> aws_bucket)

[ubuntu@magna100 ~]$ python s3_copy.py
objects copied to another bucket
[ubuntu@magna090 ~]$ sudo rados ls -p default.rgw.buckets.data
78c96e27-2bfb-484a-a122-8a26cf1104db.4148.1_redhat1607329
78c96e27-2bfb-484a-a122-8a26cf1104db.4148.5_redhat1607329

s3_boto function used for copy object from one to another bucket:
destbucket=conn.get_bucket('bucket_1')
destbucket.copy_key('redhat1607329', 'aws_bucket', 'redhat1607329')

4. Deleted the objects from both buckets:
[ubuntu@magna100 ~]$ python s3test_delete_obj.py
number of objects:  1
All objects deleted
[ubuntu@magna100 ~]$ python s3test_delete_obj.py
number of objects:  1
All objects deleted
[ubuntu@magna090 ~]$ sudo rados ls -p default.rgw.buckets.data
[ubuntu@magna090 ~]$ sudo rados df
POOL_NAME                 USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD     WR_OPS WR    
.rgw.root                 1113       4      0     12                  0       0        0    348   232k      4   4096
default.rgw.buckets.data     0       0      0      0                  0       0        0    339   417k    841 13776k
default.rgw.buckets.index    0       2      0      6                  0       0        0   1067  1103k    702      0
default.rgw.control          0       8      0     24                  0       0        0      0      0      0      0
default.rgw.log              0     207      0    621                  0       0        0  14323 14116k   9506      0
default.rgw.meta          1610       9      0     27                  0       0        0    201   180k    130  22528

total_objects    230
total_used       971M
total_avail      8288G
total_space      8289G

Comment 12 Ken Dreyer (Red Hat) 2017-11-01 18:33:35 UTC

*** Bug 1468031 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2017-12-05 23:49:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3387