Bug 1512333

Summary: [RGW] Swift object expiry incorrectly trims entries, leaving behind some of the objects to be not deleted
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Benjamin Schmaus <bschmaus>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED DUPLICATE QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: high Docs Contact:
Priority: unspecified    
Version: 2.3CC: cbodley, ceph-eng-bugs, kbader, kdreyer, mbenjamin, owasserm, sweil, tserlin
Target Milestone: rc   
Target Release: 2.5   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-01-03 15:59:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Benjamin Schmaus 2017-11-12 21:59:38 UTC 
Description of problem:
In cls_timeindex_list() though `to_index` has expired for a timespan, the marker is set for a subsequent index during the time boundary check. This marker is further returned to RGWObjectExpirer::process_single_shard(), where this out_marker is trimmed from the respective shard, resulting in a lost removal hint and a leaked object.

Version-Release number of selected component (if applicable):
2.3

How reproducible:
100%

Steps to Reproduce:
To reproduce this, I've used the following simple script and by having an aggressive object expiry gc interval setting rgw_objexp_gc_interval = 30

!/bin/sh
object="myobjects" 
container=newcontainer
objects=50
j=1
while [ $j -lt $objects ]
do
final_object=$(printf "%s$j" "$object")
touch $final_object
swift -A http://localhost:80/auth -U test:tester -K testing upload $container $final_object -H X-Delete-After:60
j=`expr $j + 1`
done 

For one such run, am left with the following leaked objects after the run

swift -A http://localhost:80/auth -U test:tester -K testing list newcontainer
myobjects20
myobjects24
myobjects33


Actual results:
Objects not expired

Expected results:
Objects should be expired

Additional info:

Upstream PR: https://github.com/ceph/ceph/pull/18821