+++ This bug was initially created as a clone of Bug #2262766 +++

+++ This bug was initially created as a clone of Bug #2262741 +++

Description of problem:


On downloading a multipart and encrypted object that was transitioned via LC policy to another storage class, we observed an MD5 signature mismatch error 

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd info  s3://test-transit1/logs-obj-20M-3
s3://test-transit1/logs-obj-20M-3 (object):
   File size: 20971520
   Last mod:  Mon, 05 Feb 2024 07:36:16 GMT
   MIME type: application/octet-stream
   Storage:   cold                          <<<<<<<<<<<<<<<
   MD5 sum:   8f4e33f3dc3e414ff94e5fb6905cba8c
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       user1: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1707114783/ctime:1707114565/gid:0/gname:root/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/mode:33188/mtime:1707114565/uid:0/uname:root
[root@ceph-vimpy-zpo62y-node6 ~]#

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd get s3://test-transit1/logs-obj-20M-3
download: 's3://test-transit1/logs-obj-20M-3' -> './logs-obj-20M-3'  [1 of 1]
 20971520 of 20971520   100% in    0s   232.76 MB/s  done
WARNING: MD5 signatures do not match: computed=2c2cb58ecd02171724877444eebd85d1, received=8f4e33f3dc3e414ff94e5fb6905cba8c

Version-Release number of selected component (if applicable):

ceph version 18.2.1-10.el9cp

How reproducible:
Always

Steps to Reproduce:
1. Create a ceph cluster with rgw configured.
2. Enable default encryption and restart rgw
[root@ceph-vimpy-zpo62y-node6 ~]# ceph config dump | grep rgw_crypt_default_encryption_key
client.rgw.rgw.1                                       dev       rgw_crypt_default_encryption_key       4YSmvJtBv0aZ7geVgAsdpRnLBEwWSWlMIGnRS8a9TSA=

3. create a bucket 'test-transit1' and perform multipart upload of size  20M objects.

4. Apply an LC rule to transition to a cold storage class in 3 days 

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd getlifecycle s3://test-transit1
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>example-id</ID>
		<Filter>
			<Prefix>logs</Prefix>
		</Filter>
		<Status>Enabled</Status>
		<Transition>
			<Days>3</Days>
			<StorageClass>cold</StorageClass>
		</Transition>
	</Rule>
</LifecycleConfiguration>

[root@ceph-vimpy-zpo62y-node6 ~]#

5. Try object download for the objects transitioned to the cold class. 

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd info  s3://test-transit1/logs-obj-20M-3
s3://test-transit1/logs-obj-20M-3 (object):
   File size: 20971520
   Last mod:  Mon, 05 Feb 2024 07:36:16 GMT
   MIME type: application/octet-stream
   Storage:   cold                          <<<<<<<<<<<<<<<
   MD5 sum:   8f4e33f3dc3e414ff94e5fb6905cba8c
   SSE:       none
   Policy:    none
   CORS:      none
   ACL:       user1: FULL_CONTROL
   x-amz-meta-s3cmd-attrs: atime:1707114783/ctime:1707114565/gid:0/gname:root/md5:8f4e33f3dc3e414ff94e5fb6905cba8c/mode:33188/mtime:1707114565/uid:0/uname:root
[root@ceph-vimpy-zpo62y-node6 ~]#

[root@ceph-vimpy-zpo62y-node6 ~]# s3cmd get s3://test-transit1/logs-obj-20M-3
download: 's3://test-transit1/logs-obj-20M-3' -> './logs-obj-20M-3'  [1 of 1]
 20971520 of 20971520   100% in    0s   232.76 MB/s  done
WARNING: MD5 signatures do not match: computed=2c2cb58ecd02171724877444eebd85d1, received=8f4e33f3dc3e414ff94e5fb6905cba8c


Actual results:

Object download for multipart and encrypted object fails with MD5 signature mismatch error when the object was transitioned to a cold storage class

Expected results:

Download should not fail

Additional info:

--- Additional comment from Vidushi Mishra on 2024-02-05 11:04:51 UTC ---

This issue is also observed in 5.3 and 6.1 versions.