Bug 1890843 - [rgw] Objects with S3 Object Lock (WORM) getting overwritten
Summary: [rgw] Objects with S3 Object Lock (WORM) getting overwritten
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 4.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: 5.0
Assignee: Matt Benjamin (redhat)
QA Contact: Tejas
URL:
Whiteboard:
Depends On:
Blocks: 1797784
TreeView+ depends on / blocked
 
Reported: 2020-10-23 03:34 UTC by Deepu K S
Modified: 2024-01-09 05:24 UTC (History)
11 users (show)

Fixed In Version: ceph-16.1.0-486.el8cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-08-30 08:26:54 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Ceph Project Bug Tracker 48351 0 None None None 2021-02-25 14:59:51 UTC
Red Hat Issue Tracker RHCEPH-1178 0 None None None 2021-08-30 00:15:38 UTC
Red Hat Product Errata RHBA-2021:3294 0 None None None 2021-08-30 08:27:06 UTC

Description Deepu K S 2020-10-23 03:34:31 UTC
Description of problem:
S3 Object locking (WORM) not working as expected.

When trying out a test to implement a WORM with S3 Object locking by preventing an object from being deleted or overwritten for a fixed amount of time or indefinitely, it doesn't seem to work as expected.
I'm able to overwrite or delete the objects with lock enabled.

The feature is based on the AWS S3 functionality described here:
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html

I see BZ https://bugzilla.redhat.com/show_bug.cgi?id=1797784 where support for S3 Object-lock is integrated into RGW, but I request to consider this behaviour when testing.

Version-Release number of selected component (if applicable):
   "rgw": {
        "ceph version 14.2.8-91.el8cp (75b4845da7d469665bd48d1a49badcc3677bf5cd) nautilus (stable)": 2

How reproducible:
Always

Steps to Reproduce:
1. Adding in the next comments.
2.
3.

Actual results:
We notice that object CAN be overwritten by an upload with the same key name. If these “new” version doesn’t have a legal hold or lock-mode they can also be deleted.
So locking seems to be limited for deletion of version and can be overwritten.

In my tests locally, I was even able delete a file with legal hold. I'm adding the tests results in coming comments.

Expected results:
Objects should not be overwritten/deleted within this retention period.

Additional info:

Comment 1 Deepu K S 2020-10-23 03:36:18 UTC
Below are the test results done using aws client:

ceph version 14.2.8-89.el8cp (9ab115d618c72e7d9227441ec25ceb1487c76fb8) nautilus (stable)

[admin@admin-node object-lock-testing]$ ls -l
total 84480
-rw-rw-r--. 1 admin admin  5242880 Sep 22 14:50 compliance-test-orig.dd
-rw-rw-r--. 1 admin admin 52428800 Sep 22 14:49 compliance-test-replace.dd
-rw-rw-r--. 1 admin admin  2621440 Sep 22 14:49 legal-test-orig.dd
-rw-rw-r--. 1 admin admin 26214400 Sep 22 14:49 legal-test-replace.dd

1. Created a bucket with object lock enabled.
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api create-bucket --bucket object-lock-test-bucket --object-lock-enabled-for-bucket
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api list-buckets
{
    "Buckets": [
        {
            "Name": "object-lock-test-bucket",
            "CreationDate": "2020-09-22T19:04:15.130Z"
        }
    ],
    "Owner": {
        "DisplayName": "S3 Operator",
        "ID": "operator"
    }
}

2. Setting up the Bucket Retention mode for the bucket.
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object-lock-configuration --bucket object-lock-test-bucket --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 1 }}}'
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api get-object-lock-configuration --bucket object-lock-test-bucket
{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled",
        "Rule": {
            "DefaultRetention": {
                "Mode": "COMPLIANCE",
                "Days": 1
            }
        }
    }
}


3. Put an object with Retention:COMPLIANCE mode and retention period of 1 day.
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --object-lock-mode COMPLIANCE --object-lock-retain-until-date "2020-09-24" --key compliance-upload --body compliance-test-orig.dd 
{
    "ETag": "\"2b1dda510bdfe6612d0b894136ffd834\"",
    "VersionId": "OZme5xAU6wwXzp1Ons1NH1to4nrY38h"
}

[admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/
2020-09-22 19:09   5242880   s3://object-lock-test-bucket/compliance-upload

4. Replace the above object with another file and same key value as normal upload.
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --key compliance-upload --body compliance-test-replace.dd 
{
    "ETag": "\"e895015ad1c474fbb8ed9d2f65a35ee1\"",
    "VersionId": "2nzrinZVaNmZcz6YPhgbvv.lm9VtIvI"
}

[admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/
2020-09-22 19:17  52428800   s3://object-lock-test-bucket/compliance-upload

The object is getting overwritten.



5. Same test done with Legal hold enabled. Put an object with legal hold set on it.
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --object-lock-legal-hold-status ON --key legal-upload --body legal-test-orig.dd 
{
    "ETag": "\"a3459175525a9779fe72fc044a26b2a8\"",
    "VersionId": "QYYK-aJEcNiKSBFMDZNnr.eiwzdNJX6"
}
[admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/
2020-09-22 19:17  52428800   s3://object-lock-test-bucket/compliance-upload
2020-09-22 19:18   2621440   s3://object-lock-test-bucket/legal-upload

6. Replace it with a normal object.
[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --key legal-upload --body legal-test-replace.dd 
{
    "ETag": "\"9bc00415cabaa61a82a01cc303cde0bd\"",
    "VersionId": "mzvTqizgy474DOt73Ml2QFJ8MJ-pBNH"
}
[admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/
2020-09-22 19:17  52428800   s3://object-lock-test-bucket/compliance-upload
2020-09-22 19:19  26214400   s3://object-lock-test-bucket/legal-upload

The file is seen as overwritten.

[admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api list-objects --bucket object-lock-test-bucket
{
    "Contents": [
        {
            "Key": "compliance-upload",
            "LastModified": "2020-09-22T19:17:33.323Z",
            "ETag": "\"e895015ad1c474fbb8ed9d2f65a35ee1\"",
            "Size": 52428800,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "S3 Operator",
                "ID": "operator"
            }
        },
        {
            "Key": "legal-upload",
            "LastModified": "2020-09-22T19:19:49.903Z",
            "ETag": "\"9bc00415cabaa61a82a01cc303cde0bd\"",
            "Size": 26214400,
            "StorageClass": "STANDARD",
            "Owner": {
                "DisplayName": "S3 Operator",
                "ID": "operator"
            }
        }
    ]
}

7. The objects gets overwritten and can be deleted even with other clients such as s3cmd or aws s3api. In my case, I was able to delete even the first version of the object.

Comment 12 errata-xmlrpc 2021-08-30 08:26:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:3294


Note You need to log in before you can comment on or make changes to this bug.