Bug 2319199

Summary: [GSS] radosgw-admin lc process --bucket <bucket> doesn't remove objects if versioning is suspended
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Rodrigo Capa <rocapa>
Component: RGWAssignee: Soumya Koduri <skoduri>
Status: VERIFIED --- QA Contact: Chaithra <ckulal>
Severity: medium Docs Contact: Rivka Pollack <rpollack>
Priority: high    
Version: 7.1CC: bkunal, ceph-eng-bugs, cephqe-warriors, ckulal, jcaratza, kelwhite, mbenjamin, rocapa, rpollack, skoduri, smanjara, vasawant, vimishra
Target Milestone: ---Flags: rocapa: needinfo? (mbenjamin)
skoduri: needinfo? (mbenjamin)
vasawant: needinfo? (mbenjamin)
Target Release: 8.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-19.2.1-2 Doc Type: Bug Fix
Doc Text:
.Objects are now removed as per the lifecycle rules set when bucket versioning is suspended Previously, due to an error in the lifecycle code, the lifecycle process did not remove the objects if the bucket versioning was in the suspended state. As a result, the objects were still seen in the bucket listing. With this fix, the lifecycle code is fixed and now the lifecycle process removes objects as per the rules set and the objects are no longer listed in the bucket listing.
 Story Points: ---
Clone Of:
: 2336609 2336868 (view as bug list) Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2351689, 2336609, 2336868    

Description Rodrigo Capa 2024-10-16 14:17:53 UTC 
Version-Release number of selected component (if applicable):
tested on:
ceph version 18.2.1-229.el9cp
ceph version 16.2.10-266.el9cp

See comments on outputs for each RHCS version


Summary

With versionsing configuration Status is configured as suspended, the LC process deletes expired objects but indexes persist.


How reproducible:

From Bash
1. Create a bucket
2. Add a file to the bucket
3. put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket
4. put-bucket-lifecycle-configuration
5. put-bucket-lifecycle-configuration with a past date

from cephadm
6. radosgw-admin lc list
7. radosgw-admin lc process --bucket

from Bash
8. ls bucket
9. cp object
the object is listed and downloaded

10. put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Enabled

from cephadm
11. radosgw-admin lc list
12. radosgw-admin lc process --bucket

from Bash
13. ls bucket
14. cp object
the object is gone and not listed. But is somehow on the bucket
15. rm bucket




Steps to Reproduce:

### From Bash, configure aws client, create a bucket, add an object and put versioning and lifecycle
$aws configure --profile=ceph
...


$aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3api create-bucket --bucket testbucket
$
$aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3api list-buckets
{
    "Buckets": [
        {
            "Name": "testbucket",:
            "CreationDate": "2024-10-11T14:58:37.624000+00:00"
        }
    ],
    "Owner": {
        "DisplayName": "User",
        "ID": "user"
    }
}
(END)
$
$aws --profile=ceph --endpoint=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket
$
$dd if=/dev/random of=file01.bin bs=1M count=1
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 --acl=public-read-write s3 cp file01.bin s3://testbucket/file01.bin
upload: ./file01.bin to s3://testbucket/file01.bin                
$
### Configure versioning Status Suspended
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api get-bucket-versioning --bucket testbucket
{
    "Status": "Suspended",
    "MFADelete": "Disabled"
}
$
$cat lc-exp.json 
{
    "Rules": [
        {
            "Expiration": {
                "Date": "2024-10-09T00:00:00+00:00"
            },
            "ID": "Delete after 09-10",
            "Prefix": "",
            "Status": "Enabled"
        }
    ]
}
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lc-exp.json
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api get-bucket-lifecycle-configuration --bucket testbucket
{
    "Rules": [
        {
            "Expiration": {
                "Date": "2024-10-09T00:00:00+00:00"
            },
            "ID": "Delete after 09-10",
            "Prefix": "",
            "Status": "Enabled"
        }
    ]
}
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket
2024-10-11 17:02:52    1048576 file01.bin
$


### From cephadm process LC
[ceph: root@mgmt-0 /]# radosgw-admin lc list
[
    {
        "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2",
        "shard": "lc.0",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "UNINITIAL"
    }
]
[ceph: root@mgmt-0 /]# 
[ceph: root@mgmt-0 /]# radosgw-admin lc process --bucket testbucket
### On RHCS 5 outputs:
2024-10-15T17:34:16.121+0000 7fbf1affd700  0 lifecycle: ERROR: bucket shard callback failed. obj=file01.bin. ret=(125) Operation canceled
### But nothing on RHCS 7
[ceph: root@mgmt-0 /]# 
[ceph: root@mgmt-0 /]# radosgw-admin lc list
[
    {
        "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2",
        "shard": "lc.0",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "COMPLETE"
    }
]
### Check status COMPLETE but started timestamp hasn't changed


### From Bash verify object existence, it shouldn't exist
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket
2024-10-11 17:02:52    1048576 file01.bin
$



### Let's try with versioning Status Enabled
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Enabled --bucket testbucket
$

### From cephadm process LC
[ceph: root@mgmt-0 /]# radosgw-admin lc list
[
    {
        "bucket": ":testbucket:2d477110-e0f5-4b54-896d-6da177205552.267800.2",
        "shard": "lc.0",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "COMPLETE"
    }
]
[ceph: root@mgmt-0 /]# radosgw-admin lc process --bucket testbucket
### On RHCS 7 outputs
2024-10-11T15:18:20.962+0000 7f1a7d7fa640  0 lifecycle: ERROR: remove_expired_obj :testbucket[2d477110-e0f5-4b54-896d-6da177205552.267800.2]):file01.bin (2) No such file or directory wp_thrd: 0, 0
2024-10-11T15:18:20.962+0000 7f1a7d7fa640  0 lifecycle: ERROR: remove_expired_obj :testbucket[2d477110-e0f5-4b54-896d-6da177205552.267800.2]):file01.bin (2) No such file or directory wp_thrd: 0, 0
### But nothing on RHCS 5
[ceph: root@mgmt-0 /]#



### Again from Bash, object is listed but doesn't exist anymore
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket
### On RHCS7 the object is listed
2024-10-11 17:02:52    1048576 file01.bin
### But not on RHCS5
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 --acl=public-read-write s3 cp s3://testbucket/file01.bin download01.bin
fatal error: An error occurred (404) when calling the HeadObject operation: Key "file01.bin" does not exist
$


### Ceph log errors while trying to access the object
2024-10-11T15:50:01.020+0000 7f4095d17640 -1 req 3761530966336190639 0.002000019s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=deletebucket:2d477110-e0f5-4b54-896d-6da177205552.267800.1,z=0a875f53-f401-4c41-97e4-adea129ffef6,az=0},d={b=deletebucket:2d477110-e0f5-4b54-896d-6da177205552.267800.1,z=2d477110-e0f5-4b54-896d-6da177205552,az=0}} returned ret=-2
2024-10-11T15:50:01.475+0000 7f40a953e640 -1 req 14709733441495016299 0.002000019s :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe={s={b=yourbucket:2d477110-e0f5-4b54-896d-6da177205552.564426.2,z=0a875f53-f401-4c41-97e4-adea129ffef6,az=0},d={b=yourbucket:2d477110-e0f5-4b54-896d-6da177205552.564426.2,z=2d477110-e0f5-4b54-896d-6da177205552,az=0}} returned ret=-2


### Manually delete object
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 rm  s3://testbucket/file01.bin
delete: s3://testbucket/file01.bin
$
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 ls s3://testbucket
$



### RHCS 5 or RHCS 7, Delete bucket from cephadm, fails besides it's empty
[ceph: root@mgmt-0 /]# radosgw-admin bucket list
[
    "testbucket"
]
[ceph: root@mgmt-0 /]#
[ceph: root@mgmt-0 /]# radosgw-admin bucket rm --bucket testbucket
2024-10-11T15:33:59.642+0000 7f8251b66800 -1 ERROR: could not remove non-empty bucket testbucket


### Try again
$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3 rm  s3://testbucket/ --recursive --include "*"
$

[ceph: root@mgmt-0 /]# radosgw-admin bucket rm --bucket testbucket
2024-10-11T15:37:18.611+0000 7f46583ce800 -1 ERROR: could not remove non-empty bucket testbucket
Comment 1 Storage PM bot 2024-10-16 14:18:05 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 4 Rodrigo Capa 2024-10-22 08:51:19 UTC 
Hello, the issue is whenever bucket-versioning Status is Suspended:

$aws --profile=ceph --endpoint-url=http://rgws-0.pnq2.redhat.com:80 s3api put-bucket-versioning --versioning-configuration MFADelete=Disabled,Status=Suspended --bucket testbucket

If LC is processed with Status=Enabled, it goes ok, but once LC is processed with Status=Suspended something is lost.
Comment 6 Rodrigo Capa 2024-11-25 08:24:15 UTC 
Hello,


is there any update about this issue?

I see activity on the past for RHCS 7,is this also been worked on RHCS 5?
Comment 9 Rodrigo Capa 2024-12-09 13:49:41 UTC 
Hello, 5 would be nice, but at least RHCS 6 and 7