Description of problem: S3 Object locking (WORM) not working as expected. When trying out a test to implement a WORM with S3 Object locking by preventing an object from being deleted or overwritten for a fixed amount of time or indefinitely, it doesn't seem to work as expected. I'm able to overwrite or delete the objects with lock enabled. The feature is based on the AWS S3 functionality described here: https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html I see BZ https://bugzilla.redhat.com/show_bug.cgi?id=1797784 where support for S3 Object-lock is integrated into RGW, but I request to consider this behaviour when testing. Version-Release number of selected component (if applicable): "rgw": { "ceph version 14.2.8-91.el8cp (75b4845da7d469665bd48d1a49badcc3677bf5cd) nautilus (stable)": 2 How reproducible: Always Steps to Reproduce: 1. Adding in the next comments. 2. 3. Actual results: We notice that object CAN be overwritten by an upload with the same key name. If these “new” version doesn’t have a legal hold or lock-mode they can also be deleted. So locking seems to be limited for deletion of version and can be overwritten. In my tests locally, I was even able delete a file with legal hold. I'm adding the tests results in coming comments. Expected results: Objects should not be overwritten/deleted within this retention period. Additional info:
Below are the test results done using aws client: ceph version 14.2.8-89.el8cp (9ab115d618c72e7d9227441ec25ceb1487c76fb8) nautilus (stable) [admin@admin-node object-lock-testing]$ ls -l total 84480 -rw-rw-r--. 1 admin admin 5242880 Sep 22 14:50 compliance-test-orig.dd -rw-rw-r--. 1 admin admin 52428800 Sep 22 14:49 compliance-test-replace.dd -rw-rw-r--. 1 admin admin 2621440 Sep 22 14:49 legal-test-orig.dd -rw-rw-r--. 1 admin admin 26214400 Sep 22 14:49 legal-test-replace.dd 1. Created a bucket with object lock enabled. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api create-bucket --bucket object-lock-test-bucket --object-lock-enabled-for-bucket [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api list-buckets { "Buckets": [ { "Name": "object-lock-test-bucket", "CreationDate": "2020-09-22T19:04:15.130Z" } ], "Owner": { "DisplayName": "S3 Operator", "ID": "operator" } } 2. Setting up the Bucket Retention mode for the bucket. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object-lock-configuration --bucket object-lock-test-bucket --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 1 }}}' [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api get-object-lock-configuration --bucket object-lock-test-bucket { "ObjectLockConfiguration": { "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 1 } } } } 3. Put an object with Retention:COMPLIANCE mode and retention period of 1 day. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --object-lock-mode COMPLIANCE --object-lock-retain-until-date "2020-09-24" --key compliance-upload --body compliance-test-orig.dd { "ETag": "\"2b1dda510bdfe6612d0b894136ffd834\"", "VersionId": "OZme5xAU6wwXzp1Ons1NH1to4nrY38h" } [admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/ 2020-09-22 19:09 5242880 s3://object-lock-test-bucket/compliance-upload 4. Replace the above object with another file and same key value as normal upload. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --key compliance-upload --body compliance-test-replace.dd { "ETag": "\"e895015ad1c474fbb8ed9d2f65a35ee1\"", "VersionId": "2nzrinZVaNmZcz6YPhgbvv.lm9VtIvI" } [admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/ 2020-09-22 19:17 52428800 s3://object-lock-test-bucket/compliance-upload The object is getting overwritten. 5. Same test done with Legal hold enabled. Put an object with legal hold set on it. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --object-lock-legal-hold-status ON --key legal-upload --body legal-test-orig.dd { "ETag": "\"a3459175525a9779fe72fc044a26b2a8\"", "VersionId": "QYYK-aJEcNiKSBFMDZNnr.eiwzdNJX6" } [admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/ 2020-09-22 19:17 52428800 s3://object-lock-test-bucket/compliance-upload 2020-09-22 19:18 2621440 s3://object-lock-test-bucket/legal-upload 6. Replace it with a normal object. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api put-object --bucket object-lock-test-bucket --key legal-upload --body legal-test-replace.dd { "ETag": "\"9bc00415cabaa61a82a01cc303cde0bd\"", "VersionId": "mzvTqizgy474DOt73Ml2QFJ8MJ-pBNH" } [admin@admin-node object-lock-testing]$ s3cmd ls s3://object-lock-test-bucket/ 2020-09-22 19:17 52428800 s3://object-lock-test-bucket/compliance-upload 2020-09-22 19:19 26214400 s3://object-lock-test-bucket/legal-upload The file is seen as overwritten. [admin@admin-node object-lock-testing]$ aws --endpoint=http://rgw-node.test.example.com:8080 s3api list-objects --bucket object-lock-test-bucket { "Contents": [ { "Key": "compliance-upload", "LastModified": "2020-09-22T19:17:33.323Z", "ETag": "\"e895015ad1c474fbb8ed9d2f65a35ee1\"", "Size": 52428800, "StorageClass": "STANDARD", "Owner": { "DisplayName": "S3 Operator", "ID": "operator" } }, { "Key": "legal-upload", "LastModified": "2020-09-22T19:19:49.903Z", "ETag": "\"9bc00415cabaa61a82a01cc303cde0bd\"", "Size": 26214400, "StorageClass": "STANDARD", "Owner": { "DisplayName": "S3 Operator", "ID": "operator" } } ] } 7. The objects gets overwritten and can be deleted even with other clients such as s3cmd or aws s3api. In my case, I was able to delete even the first version of the object.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 5.0 bug fix and enhancement), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3294