Bug 2353296

Description of problem:
small objects download failing with checksum-mismatch error using client aws-cli/2.24.22
but if we disable checksum parameters and try to download the object again, download passed and the md5sum of the object is same as the local object which was uploaded.



log snippet:

[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ /usr/local/bin/aws --version
aws-cli/2.24.22 Python/3.12.9 Linux/5.14.0-503.31.1.el9_5.x86_64 exe/x86_64.rhel.9
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ 
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ /usr/local/bin/aws --endpoint-url http://10.0.67.106:80 s3 cp obj9KB s3://bkt1/small_obj2_9KB
upload: ./obj9KB to s3://bkt1/small_obj2_9KB                    
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ [cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ /usr/local/bin/aws --endpoint-url http://10.0.67.106:80 s3api get-object-attributes --bucket bkt1 --key small_obj2_9KB --object-attributes ObjectSize ObjectParts checksum etag StorageClass
{
    "LastModified": "2025-03-19T04:22:29+00:00",
    "ETag": "5420afa22f6423a9f59e669540656bb4",
    "Checksum": {
        "ChecksumCRC64NVME": "muxnKqokq0A=",
        "ChecksumType": "FULL_OBJECT"
    },
    "StorageClass": "STANDARD",
    "ObjectSize": 9000
}
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ /usr/local/bin/aws --endpoint-url http://10.0.67.106:80 s3 cp s3://bkt1/small_obj2_9KB small_obj2_9KB.download
download failed: s3://bkt1/small_obj2_9KB to ./small_obj2_9KB.download Expected checksum muxnKqokq0A= did not match calculated checksum: r1HUB0klT14=
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ 
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ export AWS_REQUEST_CHECKSUM_CALCULATION=when_required
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ export AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ /usr/local/bin/aws --endpoint-url http://10.0.67.106:80 s3 cp s3://bkt1/small_obj2_9KB small_obj2_9KB.download
download: s3://bkt1/small_obj2_9KB to ./small_obj2_9KB.download
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ 
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ ls -l | grep 9KB
-rw-r--r--. 1 cephuser cephuser       9000 Mar 18 04:59 obj9KB
-rw-r--r--. 1 cephuser cephuser       9000 Mar 19 04:22 small_obj2_9KB.download
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ 
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ md5sum obj9KB small_obj2_9KB.download
5420afa22f6423a9f59e669540656bb4  obj9KB
5420afa22f6423a9f59e669540656bb4  small_obj2_9KB.download
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ 



Version-Release number of selected component (if applicable):
ceph version 19.2.0-111.el9cp

How reproducible:
always

Steps to Reproduce:
1.create a local object of size 9KB
2.install awscli (aws-cli/2.24.22)
3.upload the above 9KB object into the bucket
4.try to download the object, download fails with checksum mismatch. but if we disable checksum validation with the client, download happens and the object is not actually corrupted. md5sum is the same

Actual results:
small object download failing with checksum mismatch

Expected results:
Expected small objects download pass even without disabling checksum validation with the client



Additional info:

test env details:
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ ceph orch host ls
HOST                                       ADDR         LABELS                    STATUS  
ceph-pri-hsm-scale-63cc9z-node1-installer  10.0.66.218  _admin,mon,mgr,installer          
ceph-pri-hsm-scale-63cc9z-node2            10.0.65.55   osd,mgr                           
ceph-pri-hsm-scale-63cc9z-node3            10.0.64.3    osd,mon                           
ceph-pri-hsm-scale-63cc9z-node4            10.0.65.231  osd,mon                           
ceph-pri-hsm-scale-63cc9z-node5            10.0.67.106  rgw,osd                           
5 hosts in cluster
[cephuser@ceph-pri-hsm-scale-63cc9z-node6 ~]$ 

client node: 10.0.64.233

creds: root/passwd , cephuser/cephuser