Bug 1887661

Summary: [GSS][RGW] ERROR: remove_expired_obj (rchs3.3z6)
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Michele Valsecchi <mvalsecc>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: CLOSED ERRATA QA Contact: Vidushi Mishra <vimishra>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.3CC: cbodley, ceph-eng-bugs, gsitlani, kbader, mbenjamin, pdhange, sweil, tchandra, tserlin, vimishra
Target Milestone: ---   
Target Release: 3.3z7   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-12.2.12-139.el7cp Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-06 18:32:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Michele Valsecchi 2020-10-13 06:15:50 UTC 
Description of problem:

"ERROR: remove_expired_obj" error while deleting objects from a given bucket, after LC expiry for objects on containerized ceph environment.


Version-Release number of selected component (if applicable):

- rchs3.3z6

~~~
 $ grep ceph sos_commands/docker/docker_images 
172.X.Y.Z:8787/rhceph/rhceph-3-rhel7                         3-46                  d2e08b1fe5f3        7 weeks ago         598 MB
172.X.Y.Z:8787/rhceph/rhceph-3-rhel7                         3-42                  5111b510183b        7 months ago        593 MB
172.X.Y.Z:8787/rhceph/rhceph-3-rhel7                         3-40                  9ac1fe572938        9 months ago        593 MB
172.X.Y.Z:8787/rhceph/rhceph-3-rhel7                         3-33                  305370d4d927        11 months ago       593 MB
172.X.Y.Z:8787/rhceph/rhceph-3-rhel7                         3-28                  2797fe544972        14 months ago       597 MB
172.X.Y.Z:8787/rhceph/rhceph-3-rhel7                         3-18                  6075d2db99bf        21 months ago       593 MB

 $ grep ceph sos_commands/docker/docker_ps
debbd1f58b1e        172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46                             "/entrypoint.sh"         4 days ago          Up 4 days                                 ceph-mon-stg-hvn-ctrl01
ac9ee38b7727        172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46                             "/entrypoint.sh"         4 days ago          Up 4 days                                 ceph-mds-stg-hvn-ctrl01
0becac14d71c        172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46                             "/entrypoint.sh"         4 days ago          Up 4 days                                 ceph-mgr-stg-hvn-ctrl01
a609831c2aea        172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46                             "/entrypoint.sh"         4 days ago          Up 4 days                                 ceph-rgw-stg-hvn-ctrl01
~~~

- amount of errors produced per day
~~~
messages-20200924.gz:0
messages-20200925.gz:0
messages-20200926.gz:0
messages-20200927.gz:10259
messages-20200928.gz:0
messages-20200929.gz:29361
messages-20200930.gz:33282
~~~

How reproducible:
Rarely. But it's constantly happening in this env, on all nodes.

Steps to Reproduce:
1. Create containerized ceph cluster
2. Create buckets
3. Setup LC for buckets
4. Observe object deletion failure on LC expiry

Actual results:
Objects not getting deleted on LC expiry

Expected results:
Objects should be deleted on LC expiry


Additional info:

This issue is a regression of private BZ[1]


[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834781
Comment 28 Matt Benjamin (redhat) 2021-03-02 21:01:27 UTC 
*** Bug 1914993 has been marked as a duplicate of this bug. ***
Comment 51 errata-xmlrpc 2021-05-06 18:32:04 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 3.3 Security and Bug Fix Update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1518