Version-Release number of selected component (if applicable): tested on: ceph version 18.2.1-229.el9cp ceph version 16.2.10-266.el9cp See comments on outputs for each RHCS version Summary With versionsing configuration Status is configured as suspended, the LC process deletes expired objects but indexes persist. How reproducible: From Bash 1. Create a bucket 2. Add a file to the bucket 3. put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket 4. put-bucket-lifecycle-configuration 5. put-bucket-lifecycle-configuration with a past date from cephadm 6. radosgw-admin lc list 7. radosgw-admin lc process --bucket from Bash 8. ls bucket 9. cp object the object is listed and downloaded 10. put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Enabled from cephadm 11. radosgw-admin lc list 12. radosgw-admin lc process --bucket from Bash 13. ls bucket 14. cp object the object is gone and not listed. But is somehow on the bucket 15. rm bucket Steps to Reproduce: ### From Bash, configure aws client, create a bucket, add an object and put versioning and lifecycle $aws configure --profile=ceph ... $aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3api create-bucket --bucket testbucket $ $aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3api list-buckets { "Buckets": [ { "Name": "testbucket",: "CreationDate": "2024-10-11T14:58:37.624000+00:00" } ], "Owner": { "DisplayName": "User", "ID": "user" } } (END) $ $aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket $ $dd if=/dev/random of=file01.bin bs=1M count=1 $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 --acl=public-read-write s3 cp file01.bin s3://testbucket/file01.bin upload: ./file01.bin to s3://testbucket/file01.bin $ ### Configure versioning Status Suspended $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api get-bucket-versioning --bucket testbucket { "Status": "Suspended", "MFADelete": "Disabled" } $ $cat lc-exp.json { "Rules": [ { "Expiration": { "Date": "2024-10-09T00:00:00+00:00" }, "ID": "Delete after 09-10", "Prefix": "", "Status": "Enabled" } ] } $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lc-exp.json $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api get-bucket-lifecycle-configuration --bucket testbucket { "Rules": [ { "Expiration": { "Date": "2024-10-09T00:00:00+00:00" }, "ID": "Delete after 09-10", "Prefix": "", "Status": "Enabled" } ] } $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket 2024-10-11 17:02:52 1048576 file01.bin $ ### From cephadm process LC [ceph: root@mgmt-0 /]# radosgw-admin lc list [ { "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] [ceph: root@mgmt-0 /]# [ceph: root@mgmt-0 /]# radosgw-admin lc process --bucket testbucket ### On RHCS 5 outputs: 2024-10-15T17:34:16.121+0000 7fbf1affd700 0 lifecycle: ERROR: bucket shard callback failed. obj=file01.bin. ret=(125) Operation canceled ### But nothing on RHCS 7 [ceph: root@mgmt-0 /]# [ceph: root@mgmt-0 /]# radosgw-admin lc list [ { "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" } ] ### Check status COMPLETE but started timestamp hasn't changed ### From Bash verify object existence, it shouldn't exist $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket 2024-10-11 17:02:52 1048576 file01.bin $ ### Let's try with versioning Status Enabled $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Enabled --bucket testbucket $ ### From cephadm process LC [ceph: root@mgmt-0 /]# radosgw-admin lc list [ { "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" } ] [ceph: root@mgmt-0 /]# radosgw-admin lc process --bucket testbucket ### On RHCS 7 outputs 2024-10-11T15:18:20.962+0000 7f1a7d7fa640 0 lifecycle: ERROR: remove_expired_obj :testbucket[2d477110-e0f5-4b54-896d-6da177205552.267800.2]):file01.bin (2) No such file or directory wp_thrd: 0, 0 2024-10-11T15:18:20.962+0000 7f1a7d7fa640 0 lifecycle: ERROR: remove_expired_obj :testbucket[2d477110-e0f5-4b54-896d-6da177205552.267800.2]):file01.bin (2) No such file or directory wp_thrd: 0, 0 ### But nothing on RHCS 5 [ceph: root@mgmt-0 /]# ### Again from Bash, object is listed but doesn't exist anymore $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket ### On RHCS7 the object is listed 2024-10-11 17:02:52 1048576 file01.bin ### But not on RHCS5 $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 --acl=public-read-write s3 cp s3://testbucket/file01.bin download01.bin fatal error: An error occurred (404) when calling the HeadObject operation: Key "file01.bin" does not exist $ ### Ceph log errors while trying to access the object 2024-10-11T15:50:01.020+0000 7f4095d17640 -1 req 3761530966336190639 0.002000019s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=deletebucket:2d477110-e0f5-4b54-896d-6da177205552.267800.1,z=0a875f53-f401-4c41-97e4-adea129ffef6,az=0},d={b=deletebucket:2d477110-e0f5-4b54-896d-6da177205552.267800.1,z=2d477110-e0f5-4b54-896d-6da177205552,az=0}} returned ret=-2 2024-10-11T15:50:01.475+0000 7f40a953e640 -1 req 14709733441495016299 0.002000019s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yourbucket:2d477110-e0f5-4b54-896d-6da177205552.564426.2,z=0a875f53-f401-4c41-97e4-adea129ffef6,az=0},d={b=yourbucket:2d477110-e0f5-4b54-896d-6da177205552.564426.2,z=2d477110-e0f5-4b54-896d-6da177205552,az=0}} returned ret=-2 ### Manually delete object $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 rm s3://testbucket/file01.bin delete: s3://testbucket/file01.bin $ $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket $ ### RHCS 5 or RHCS 7, Delete bucket from cephadm, fails besides it's empty [ceph: root@mgmt-0 /]# radosgw-admin bucket list [ "testbucket" ] [ceph: root@mgmt-0 /]# [ceph: root@mgmt-0 /]# radosgw-admin bucket rm --bucket testbucket 2024-10-11T15:33:59.642+0000 7f8251b66800 -1 ERROR: could not remove non-empty bucket testbucket ### Try again $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 rm s3://testbucket/ --recursive --include "*" $ [ceph: root@mgmt-0 /]# radosgw-admin bucket rm --bucket testbucket 2024-10-11T15:37:18.611+0000 7f46583ce800 -1 ERROR: could not remove non-empty bucket testbucket
Please specify the severity of this bug. Severity is defined here: https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Hello, the issue is whenever bucket-versioning Status is Suspended: $aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket If LC is processed with Status=Enabled, it goes ok, but once LC is processed with Status=Suspended something is lost.
Hello, is there any update about this issue? I see activity on the past for RHCS 7,is this also been worked on RHCS 5?
Hello, 5 would be nice, but at least RHCS 6 and 7