Description of problem: "ERROR: remove_expired_obj" error while deleting objects from a given bucket, after LC expiry for objects on containerized ceph environment. Version-Release number of selected component (if applicable): - rchs3.3z6 ~~~ $ grep ceph sos_commands/docker/docker_images 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7 3-46 d2e08b1fe5f3 7 weeks ago 598 MB 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7 3-42 5111b510183b 7 months ago 593 MB 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7 3-40 9ac1fe572938 9 months ago 593 MB 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7 3-33 305370d4d927 11 months ago 593 MB 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7 3-28 2797fe544972 14 months ago 597 MB 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7 3-18 6075d2db99bf 21 months ago 593 MB $ grep ceph sos_commands/docker/docker_ps debbd1f58b1e 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46 "/entrypoint.sh" 4 days ago Up 4 days ceph-mon-stg-hvn-ctrl01 ac9ee38b7727 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46 "/entrypoint.sh" 4 days ago Up 4 days ceph-mds-stg-hvn-ctrl01 0becac14d71c 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46 "/entrypoint.sh" 4 days ago Up 4 days ceph-mgr-stg-hvn-ctrl01 a609831c2aea 172.X.Y.Z:8787/rhceph/rhceph-3-rhel7:3-46 "/entrypoint.sh" 4 days ago Up 4 days ceph-rgw-stg-hvn-ctrl01 ~~~ - amount of errors produced per day ~~~ messages-20200924.gz:0 messages-20200925.gz:0 messages-20200926.gz:0 messages-20200927.gz:10259 messages-20200928.gz:0 messages-20200929.gz:29361 messages-20200930.gz:33282 ~~~ How reproducible: Rarely. But it's constantly happening in this env, on all nodes. Steps to Reproduce: 1. Create containerized ceph cluster 2. Create buckets 3. Setup LC for buckets 4. Observe object deletion failure on LC expiry Actual results: Objects not getting deleted on LC expiry Expected results: Objects should be deleted on LC expiry Additional info: This issue is a regression of private BZ[1] [1] https://bugzilla.redhat.com/show_bug.cgi?id=1834781
*** Bug 1914993 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 3.3 Security and Bug Fix Update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:1518