Version-Release number of selected component (if applicable): tested on: ceph version 18.2.1-229.el9cp ceph version 16.2.10-266.el9cp See comments on outputs for each RHCS version Summary With versionsing configuration Status is configured as suspended, the LC process deletes expired objects but indexes persist. How reproducible: From Bash 1. Create a bucket 2. Add a file to the bucket 3. put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket 4. put-bucket-lifecycle-configuration 5. put-bucket-lifecycle-configuration with a past date from cephadm 6. radosgw-admin lc list 7. radosgw-admin lc process --bucket from Bash 8. ls bucket 9. cp object the object is listed and downloaded 10. put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Enabled from cephadm 11. radosgw-admin lc list 12. radosgw-admin lc process --bucket from Bash 13. ls bucket 14. cp object the object is gone and not listed. But is somehow on the bucket 15. rm bucket Steps to Reproduce: ### From Bash, configure aws client, create a bucket, add an object and put versioning and lifecycle $aws configure --profile=ceph ... $aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3api create-bucket --bucket testbucket $ $aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3api list-buckets { "Buckets": [ { "Name": "testbucket",: "CreationDate": "2024-10-11T14:58:37.624000+00:00" } ], "Owner": { "DisplayName": "User", "ID": "user" } } (END) $ $aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket $ $dd if=/dev/random of=file01.bin bs=1M count=1 $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 --acl=public-read-write s3 cp file01.bin s3://testbucket/file01.bin upload: ./file01.bin to s3://testbucket/file01.bin $ ### Configure versioning Status Suspended $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api get-bucket-versioning --bucket testbucket { "Status": "Suspended", "MFADelete": "Disabled" } $ $cat lc-exp.json { "Rules": [ { "Expiration": { "Date": "2024-10-09T00:00:00+00:00" }, "ID": "Delete after 09-10", "Prefix": "", "Status": "Enabled" } ] } $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lc-exp.json $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api get-bucket-lifecycle-configuration --bucket testbucket { "Rules": [ { "Expiration": { "Date": "2024-10-09T00:00:00+00:00" }, "ID": "Delete after 09-10", "Prefix": "", "Status": "Enabled" } ] } $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket 2024-10-11 17:02:52 1048576 file01.bin $ ### From cephadm process LC [ceph: root@mgmt-0 /]# radosgw-admin lc list [ { "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] [ceph: root@mgmt-0 /]# [ceph: root@mgmt-0 /]# radosgw-admin lc process --bucket testbucket ### On RHCS 5 outputs: 2024-10-15T17:34:16.121+0000 7fbf1affd700 0 lifecycle: ERROR: bucket shard callback failed. obj=file01.bin. ret=(125) Operation canceled ### But nothing on RHCS 7 [ceph: root@mgmt-0 /]# [ceph: root@mgmt-0 /]# radosgw-admin lc list [ { "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" } ] ### Check status COMPLETE but started timestamp hasn't changed ### From Bash verify object existence, it shouldn't exist $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket 2024-10-11 17:02:52 1048576 file01.bin $ ### Let's try with versioning Status Enabled $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Enabled --bucket testbucket $ ### From cephadm process LC [ceph: root@mgmt-0 /]# radosgw-admin lc list [ { "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" } ] [ceph: root@mgmt-0 /]# radosgw-admin lc process --bucket testbucket ### On RHCS 7 outputs 2024-10-11T15:18:20.962+0000 7f1a7d7fa640 0 lifecycle: ERROR: remove_expired_obj :testbucket[2d477110-e0f5-4b54-896d-6da177205552.267800.2]):file01.bin (2) No such file or directory wp_thrd: 0, 0 2024-10-11T15:18:20.962+0000 7f1a7d7fa640 0 lifecycle: ERROR: remove_expired_obj :testbucket[2d477110-e0f5-4b54-896d-6da177205552.267800.2]):file01.bin (2) No such file or directory wp_thrd: 0, 0 ### But nothing on RHCS 5 [ceph: root@mgmt-0 /]# ### Again from Bash, object is listed but doesn't exist anymore $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket ### On RHCS7 the object is listed 2024-10-11 17:02:52 1048576 file01.bin ### But not on RHCS5 $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 --acl=public-read-write s3 cp s3://testbucket/file01.bin download01.bin fatal error: An error occurred (404) when calling the HeadObject operation: Key "file01.bin" does not exist $ ### Ceph log errors while trying to access the object 2024-10-11T15:50:01.020+0000 7f4095d17640 -1 req 3761530966336190639 0.002000019s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=deletebucket:2d477110-e0f5-4b54-896d-6da177205552.267800.1,z=0a875f53-f401-4c41-97e4-adea129ffef6,az=0},d={b=deletebucket:2d477110-e0f5-4b54-896d-6da177205552.267800.1,z=2d477110-e0f5-4b54-896d-6da177205552,az=0}} returned ret=-2 2024-10-11T15:50:01.475+0000 7f40a953e640 -1 req 14709733441495016299 0.002000019s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yourbucket:2d477110-e0f5-4b54-896d-6da177205552.564426.2,z=0a875f53-f401-4c41-97e4-adea129ffef6,az=0},d={b=yourbucket:2d477110-e0f5-4b54-896d-6da177205552.564426.2,z=2d477110-e0f5-4b54-896d-6da177205552,az=0}} returned ret=-2 ### Manually delete object $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 rm s3://testbucket/file01.bin delete: s3://testbucket/file01.bin $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket $ ### RHCS 5 or RHCS 7, Delete bucket from cephadm, fails besides it's empty [ceph: root@mgmt-0 /]# radosgw-admin bucket list [ "testbucket" ] [ceph: root@mgmt-0 /]# [ceph: root@mgmt-0 /]# radosgw-admin bucket rm --bucket testbucket 2024-10-11T15:33:59.642+0000 7f8251b66800 -1 ERROR: could not remove non-empty bucket testbucket ### Try again $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 rm s3://testbucket/ --recursive --include "*" $ [ceph: root@mgmt-0 /]# radosgw-admin bucket rm --bucket testbucket 2024-10-11T15:37:18.611+0000 7f46583ce800 -1 ERROR: could not remove non-empty bucket testbucket
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Hello, the issue is whenever bucket-versioning Status is Suspended: $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket If LC is processed with Status=Enabled, it goes ok, but once LC is processed with Status=Suspended something is lost.
Hello, is there any update about this issue? I see activity on the past for RHCS 7,is this also been worked on RHCS 5?
Hello, 5 would be nice, but at least RHCS 6 and 7
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:9775
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days