Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 1211431

Summary: rgw deleting S3 objects leaves __shadow_ objects behind
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Andrew Hatfield <ahatfiel>
Component: RGWAssignee: Yehuda Sadeh <yehuda>
Status: CLOSED ERRATA QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: medium Docs Contact:
Priority: high    
Version: 1.3.0CC: cbodley, ceph-eng-bugs, ceph-qe-bugs, dblack, flucifre, icolle, kbader, kdreyer, mbenjamin, owasserm, sweil, tganguly, tmuthami, yehuda
Target Milestone: rc   
Target Release: 1.3.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-07-16 22:20:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Test Script none

Description Andrew Hatfield 2015-04-14 01:40:00 UTC 
Description of problem: rgw deleting S3 objects leaves __shadow_ objects behind

Internal BZ for upstream #10295 (Ceph Tracker)
http://tracker.ceph.com/issues/10295


We have an hosting provider prospect that is keen to get this resolved and move forward with Subscription purchases


Version-Release number of selected component (if applicable):
Using 0.80.7, observed also on .87, .89

How reproducible:


Steps to Reproduce:
1. Create 100 objects 10MB ea in rgw.
2. Observe that .rgw.buckets occupies 1GB data (as expected).
3. Delete all 100 objects.
4. rgw is now empty

    radosgw-admin bucket list --bucket=test
    []

5. Space occupied is unexpectedly high.

rados df:    .rgw.buckets           11     959M         0         7447G         303

There are 303 objects with names like default.4209.1__shadow_.cC7M_LMTe8oNgWoGZOiNCiJco9cEEOl_1

ceph df: the object store is basically empty but 3GB RAW is used
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
22345G 22342G 3126M 0.01
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
data 0 0 0 7447G 0
metadata 1 0 0 7447G 0
rbd 2 0 0 7447G 0
.rgw.root 3 840 0 7447G 3
.rgw.control 4 0 0 7447G 8
.rgw 5 331 0 7447G 2
.rgw.gc 6 0 0 7447G 521
.users.uid 7 1341 0 7447G 5
.users.email 8 43 0 7447G 4
.users 9 43 0 7447G 4
.rgw.buckets.index 10 0 0 7447G 1
.rgw.buckets 11 959M 0 7447G 303
12 0 0 7447G 0

Actual results:


Expected results:


Additional info:

rgw deleting S3 objects leaves __shadow_ objects behind
Comment 9 Federico Lucifredi 2015-06-18 22:38:10 UTC 
Added to errata priority backlog, https://trello.com/c/Oe0e5vVk/11-1-3-0-async-rhel-rgw-civet-garbage-collection
Comment 10 Yehuda Sadeh 2015-06-18 22:41:57 UTC 
Does not fix previously leaked objects. For previously leaked objects there's a need for a separate tool that I worked a while back, but it's not done yet.
Comment 11 Yehuda Sadeh 2015-06-22 23:48:22 UTC 
missing qa_ack
Comment 13 Tanay Ganguly 2015-07-10 12:30:17 UTC 
I am able to reproduce the same issue with the latest Async release. Even after deleting all the Objects i can see the shadow copies are not getting deleted.


[root@Cluster1 ~]# rpm -qa |grep ceph
ceph-osd-0.94.1-15.el7cp.x86_64
ceph-common-0.94.1-15.el7cp.x86_64
ceph-0.94.1-15.el7cp.x86_64
ceph-mon-0.94.1-15.el7cp.x86_64
ceph-deploy-1.5.25-1.el7cp.noarch

[root@RGW ~]# rpm -qa |grep ceph
ceph-radosgw-0.94.1-15.el7cp.x86_64
ceph-common-0.94.1-15.el7cp.x86_64

Please find the attached script for the operations which i performed.

I can still see lots the 
"default.4194.22__shadow_.Zba-hrAA_StUHa3y4SFkURAlxjmXAs9_3"

being present after i delete all the objects created.

Even the
root@Cluster1 ~]# ceph df
GLOBAL:
    SIZE      AVAIL     RAW USED     %RAW USED 
    3460G     3457G        2455M          0.07 

AND

[root@Cluster1 ~]# rados df
pool name                 KB      objects       clones     degraded      unfound           rd        rd KB           wr        wr KB
.rgw                       1            2            0            0           0          278          210          133           28
.rgw.buckets          972800          300            0            0           0         3312          609        28321     11264003
.rgw.buckets.index            0            3            0            0           0         4543         4598         3283            0
.rgw.control               0            8            0            0           0            0            0            0            0
.rgw.gc                    0           32            0            0           0         2909         4383         2334            0
.rgw.root                  1            3            0            0           0           45           30            3            3
.users                     1            2            0            0           0            2            1            2            2
.users.swift               1            1            0            0           0            2            1            1            1
.users.uid                 1            2            0            0           0          135          131           75            3
rbd                        0            0            0            0           0            0            0            0            0
  total used         2514728          353
  total avail     3625606264
  total space     3628120992

I used the puddle to upgrade my Cluster from 1.3.0 to 1.3.0_Async Release.


baseurl = http://puddle.ceph.redhat.com/puddles/RHCeph/1.3-RHEL-7/2015-07-06.2/Server-RH7-CEPH-TOOLS-1.3/x86_64/os
Comment 14 Tanay Ganguly 2015-07-10 13:23:37 UTC 
Created attachment 1050680 [details]
Test Script
Comment 15 Yehuda Sadeh 2015-07-10 19:49:24 UTC 
Not sure I understand how the test script relates to this bug. So you create a few objects, sleep 1200 seconds (20 minutes), then delete them?

The idea is to create the objects, delete them, then wait past the garbage collector's period (minimum of 2 hours, should wait more).
Comment 16 Tanay Ganguly 2015-07-11 04:31:37 UTC 
Thanks Yehuda,
I got it...To clean up the shadows copies it takes time i didn't know. (GC to take 2 hours or more)
Just checked again in my Cluster and saw all the shadow copies are deleted.

Hence its working fine as per the Bug, marking it as verified.

PS: The sleep was just like that ( I wanted to check my Cluster state and object copies )
Comment 19 Tanay Ganguly 2015-07-14 04:58:09 UTC 
[root@RGW ~]# rpm -qa |grep ceph
ceph-radosgw-0.94.1-15.el7cp.x86_64
ceph-common-0.94.1-15.el7cp.x86_64
Comment 22 errata-xmlrpc 2015-07-16 22:20:06 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2015:1240