Bug 2266648

Summary: [rgw-multipart/encryption]: Object download fails with "MD5 signatures do not match" if transitioned to other storage class
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Hemanth Sai <hmaheswa>
Component: RGWAssignee: Matt Benjamin (redhat) <mbenjamin>
Status: NEW --- QA Contact: Hemanth Sai <hmaheswa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 5.3CC: bkunal, ceph-eng-bugs, cephqe-warriors, jcaratza, mkasturi, vimishra
Target Milestone: ---   
Target Release: 5.3z9   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2262741, 2262766, 2269342    
Bug Blocks:    

Description Hemanth Sai 2024-02-28 17:00:12 UTC 
This bug was initially created as a copy of Bug #2262741


Description of problem:


On downloading a multipart and encrypted object that was transitioned via LC policy to another storage class, we observed an MD5 signature mismatch error 

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd info  s3://test-transit1/logs-obj-20M-3
s3://test-transit1/logs-obj-20M-3 (object):
   File size: 20971520
   Last mod:  Mon, 05 Feb 2024 07:36:16 GMT
   MIME type: application/octet-stream
   Storage:   cold                          <<<<<<<<<<<<<<<
   MD5 sum:   8f4e33f3dc3e414ff94e5fb6905cba8c
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       user1: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1707114783/ctime:1707114565/gid:0/gname:root/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/mode:33188/mtime:1707114565/uid:0/uname:root
[root@ceph-vimpy-zpo62y-node6 ~]#

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd get s3://test-transit1/logs-obj-20M-3
download: 's3://test-transit1/logs-obj-20M-3' -> './logs-obj-20M-3'  [1 of 1]
 20971520 of 20971520   100% in    0s   232.76 MB/s  done
WARNING: MD5 signatures do not match: computed=2c2cb58ecd02171724877444eebd85d1, received=8f4e33f3dc3e414ff94e5fb6905cba8c

Version-Release number of selected component (if applicable):

ceph version 18.2.1-10.el9cp

How reproducible:
Always

Steps to Reproduce:
1. Create a ceph cluster with rgw configured.
2. Enable default encryption and restart rgw
[root@ceph-vimpy-zpo62y-node6 ~]# ceph config dump | grep rgw_crypt_default_encryption_key
client.rgw.rgw.1                                       dev       rgw_crypt_default_encryption_key       4YSmvJtBv0aZ7geVgAsdpRnLBEwWSWlMIGnRS8a9TSA=

3. create a bucket 'test-transit1' and perform multipart upload of size  20M objects.

4. Apply an LC rule to transition to a cold storage class in 3 days 

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd getlifecycle s3://test-transit1
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>example-id</ID>
		<Filter>
			<Prefix>logs</Prefix>
		</Filter>
		<Status>Enabled</Status>
		<Transition>
			<Days>3</Days>
			<StorageClass>cold</StorageClass>
		</Transition>
	</Rule>
</LifecycleConfiguration>

[root@ceph-vimpy-zpo62y-node6 ~]#

5. Try object download for the objects transitioned to the cold class. 

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd info  s3://test-transit1/logs-obj-20M-3
s3://test-transit1/logs-obj-20M-3 (object):
   File size: 20971520
   Last mod:  Mon, 05 Feb 2024 07:36:16 GMT
   MIME type: application/octet-stream
   Storage:   cold                          <<<<<<<<<<<<<<<
   MD5 sum:   8f4e33f3dc3e414ff94e5fb6905cba8c
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       user1: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1707114783/ctime:1707114565/gid:0/gname:root/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/mode:33188/mtime:1707114565/uid:0/uname:root
[root@ceph-vimpy-zpo62y-node6 ~]#

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd get s3://test-transit1/logs-obj-20M-3
download: 's3://test-transit1/logs-obj-20M-3' -> './logs-obj-20M-3'  [1 of 1]
 20971520 of 20971520   100% in    0s   232.76 MB/s  done
WARNING: MD5 signatures do not match: computed=2c2cb58ecd02171724877444eebd85d1, received=8f4e33f3dc3e414ff94e5fb6905cba8c


Actual results:

Object download for multipart and encrypted object fails with MD5 signature mismatch error when the object was transitioned to a cold storage class

Expected results:

Download should not fail

Additional info:
manual test logs on 5.3: https://docs.google.com/document/d/1K5zRHfLEC1LfwSpsLxFO28WKh3NyFWdum1s6jeOEHkU/edit?usp=sharing