Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2247742

Summary: [rgw-ms][compression-encryption][usage][6.1]: Encrypted multipart uploads cause an inconsistency in size obtained via bucket stats
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: Vidushi Mishra <vimishra>
Component: RGW-MultisiteAssignee: Casey Bodley <cbodley>
Status: CLOSED ERRATA QA Contact: Vidushi Mishra <vimishra>
Severity: high Docs Contact: Disha Walvekar <dwalveka>
Priority: unspecified    
Version: 6.1CC: cbodley, ceph-eng-bugs, cephqe-warriors, dwalveka, mbenjamin, tserlin
Target Milestone: ---   
Target Release: 6.1z4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ceph-17.2.6-191.el9cp Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: 2236643 Environment:
Last Closed: 2024-02-08 18:13:09 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2236643    
Bug Blocks: 2261930    

Description Vidushi Mishra 2023-11-03 04:18:42 UTC 
+++ This bug was initially created as a clone of Bug #2236643 +++

Description of problem:


When performing encrypted multipart uploads, we observe that the size under rgw.main field (obtained via radosgw-admin bucket stats for a bucket) are inconsistent.

Note: This is observed when we have compression enabled on all the zones. 

Version-Release number of selected component (if applicable):

ceph version 16.2.10-208.el8cp 

How reproducible:

1/1

Steps to Reproduce:

1. Create a multisite and enable compression on all the zones (we enabled zlib) and enable default encryption.
2. Upload a 20M object from the primary site to the secondary site

[root@ceph-pri-mp-i2cdj2-node6 s3_swift]# s3cmd put 20M s3://testbucket1/zlib-20M 
upload: '20M' -> 's3://testbucket1/zlib-20M'  [part 1 of 2, 15MB] [1 of 1]
 15728640 of 15728640   100% in    0s    80.86 MB/s  done
upload: '20M' -> 's3://testbucket1/zlib-20M'  [part 2 of 2, 5MB] [1 of 1]
 5242880 of 5242880   100% in    0s    52.20 MB/s  done

[root@ceph-pri-mp-i2cdj2-node6 s3_swift]# s3cmd ls s3://testbucket1/zlib-20M
2023-09-01 04:38     20971520  s3://testbucket1/zlib-20M


3. Verify the object is synced to the secondary site

[root@ceph-sec-mp-i2cdj2-node6 reusables]#  s3cmd ls s3://testbucket1/zlib-20M
2023-09-01 04:38        20424  s3://testbucket1/zlib-20M

Actual results:

1. We observe the bucket stats are inconsistent with respect to the size across sites.

a) bucket stats at the primary site
-----------------------------------

[root@ceph-pri-mp-i2cdj2-node6 s3_swift]# radosgw-admin bucket stats --bucket testbucket1
{
    "bucket": "testbucket1",
    "num_shards": 11,
    "tenant": "",
    "zonegroup": "c642c114-eeb1-47d4-848c-d3fe37dfe4a1",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "d7d1627f-ee91-4b1f-b182-b95b5a9920d4.34792.1",
    "marker": "d7d1627f-ee91-4b1f-b182-b95b5a9920d4.34792.1",
    "index_type": "Normal",
    "owner": "christyj.741",
    "ver": "0#1,1#8,2#1,3#1,4#8,5#1,6#1,7#1,8#1,9#8,10#1",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
    "mtime": "2023-09-01T04:30:41.494112Z",
    "creation_time": "2023-09-01T04:30:41.478776Z",
    "max_marker": "0#,1#00000000007.178.6,2#,3#,4#00000000007.395.6,5#,6#,7#,8#,9#00000000007.357.6,10#",
    "usage": {
        "rgw.main": {
            "size": 62914560,
            "size_actual": 62914560,
            "size_utilized": 21012368,
            "size_kb": 61440,
            "size_kb_actual": 61440,
            "size_kb_utilized": 20520,
            "num_objects": 3


b) bucket stats at the secondary site
----------------------------------------

[root@ceph-sec-mp-i2cdj2-node6 reusables]# radosgw-admin bucket stats --bucket testbucket1
{
    "bucket": "testbucket1",
    "num_shards": 11,
    "tenant": "",
    "zonegroup": "c642c114-eeb1-47d4-848c-d3fe37dfe4a1",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "d7d1627f-ee91-4b1f-b182-b95b5a9920d4.34792.1",
    "marker": "d7d1627f-ee91-4b1f-b182-b95b5a9920d4.34792.1",
    "index_type": "Normal",
    "owner": "christyj.741",
    "ver": "0#1,1#2,2#1,3#1,4#2,5#1,6#1,7#1,8#1,9#2,10#1",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
    "mtime": "2023-09-01T04:30:41.494112Z",
    "creation_time": "2023-09-01T04:30:41.478776Z",
    "max_marker": "0#,1#00000000001.133.6,2#,3#,4#00000000001.190.6,5#,6#,7#,8#,9#00000000001.287.6,10#",
    "usage": {
        "rgw.main": {
            "size": 41963464,
            "size_actual": 41963520,
            "size_utilized": 21012984,
            "size_kb": 40980,
            "size_kb_actual": 40980,
            "size_kb_utilized": 20521,
            "num_objects": 3
        }


Expected results:

The bucket stats should be consistent across sites with respect to size.

Additional info:


The bucket stats become inconsistent when we have 
a) Compression enabled on all the zones and 
b) Encryption enabled on all the zones.

--- Additional comment from Casey Bodley on 2023-10-24 17:30:46 UTC ---

> ceph version 16.2.10-208.el8cp

this is targeted at rhcs 7, do we know if it reproduces there?

--- Additional comment from Casey Bodley on 2023-10-24 18:53:05 UTC ---

i was able to reproduce the issue on ceph-7.0-rhel-patches. after replication, the bucket index incorrectly uses the compressed size instead of the actual size

primary:
       "rgw.main": {                                                          
            "size": 20971520,                                                                                                                                 
            "size_actual": 20971520,
            "size_utilized": 20424,
            "size_kb": 20480,                                                  
            "size_kb_actual": 20480,  
            "size_kb_utilized": 20,
            "num_objects": 1                                                   
        },

~/ceph/build $ s3cmd -c c1.s3cfg ls s3://testbucket
2023-10-24 18:46     20971520  s3://testbucket/20m.iso

~/ceph/build $ s3cmd -c c1.s3cfg info s3://testbucket/20m.iso
s3://testbucket/20m.iso (object):
   File size: 20971520


secondary:
        "rgw.main": {
            "size": 20424,
            "size_actual": 20480,                                              
            "size_utilized": 20424,                                            
            "size_kb": 20,    
            "size_kb_actual": 20,
            "size_kb_utilized": 20,
            "num_objects": 1
        }

~/ceph/build $ s3cmd -c c2.s3cfg ls s3://testbucket
2023-10-24 18:46        20424  s3://testbucket/20m.iso

~/ceph/build $ s3cmd -c c2.s3cfg info s3://testbucket/20m.iso
s3://testbucket/20m.iso (object):
   File size: 20971520

the replicated object itself has the correct size and can be downloaded successfully:

~/ceph/build $ s3cmd -c c2.s3cfg get s3://testbucket/20m.iso 20m.iso.out
download: 's3://testbucket/20m.iso' -> '20m.iso.out'  [1 of 1]
 20971520 of 20971520   100% in    0s    44.11 MB/s  done

--- Additional comment from Casey Bodley on 2023-10-24 22:44:03 UTC ---

this will probably be needed on 5.3 and 6.1 too

--- Additional comment from errata-xmlrpc on 2023-10-25 01:54:56 UTC ---

This bug has been added to advisory RHBA-2023:118213 by Thomas Serlin (tserlin)

--- Additional comment from errata-xmlrpc on 2023-10-25 01:54:57 UTC ---

Bug report changed to ON_QA status by Errata System.
A QE request has been submitted for advisory RHBA-2023:118213-01
https://errata.devel.redhat.com/advisory/118213

--- Additional comment from Vidushi Mishra on 2023-10-25 05:32:13 UTC ---

The bug is verified on ceph version 18.2.0-99.el9cp.

primary site
===============
[root@folio15 hsbench]# radosgw-admin zone get | grep zlib
                        "compression_type": "zlib"
[root@folio15 hsbench]# ceph versions
{
    "mon": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 3
    },
    "mgr": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 3
    },
    "osd": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 24
    },
    "rgw": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 12
    },
    "rgw-nfs": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 1
    },
    "overall": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 43
    }
}
[root@folio15 hsbench]# radosgw-admin zone get | grep zlib
                        "compression_type": "zlib"
[root@folio15 hsbench]# radosgw-admin bucket stats --bucket testbucket1
{
    "bucket": "testbucket1",
    "num_shards": 11,
    "tenant": "",
    "versioning": "off",
    "zonegroup": "168926dc-aabe-4510-82ce-b1b3bab3448a",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "8dc62baa-afc5-496b-9ac1-692c719856ff.384645.1",
    "marker": "8dc62baa-afc5-496b-9ac1-692c719856ff.384645.1",
    "index_type": "Normal",
    "owner": "user1",
    "ver": "0#1,1#1,2#1,3#1,4#8,5#1,6#1,7#1,8#1,9#15,10#1",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
    "mtime": "2023-10-25T04:50:09.944840Z",
    "creation_time": "2023-10-25T04:50:09.941992Z",
    "max_marker": "0#,1#,2#,3#,4#00000000007.36161045.6,5#,6#,7#,8#,9#00000000014.41022577.6,10#",
    "usage": {
        "rgw.main": {
            "size": 52428800,
            "size_actual": 52428800,
            "size_utilized": 51042,
            "size_kb": 51200,
            "size_kb_actual": 51200,
            "size_kb_utilized": 50,
            "num_objects": 2
        },
        "rgw.multimeta": {
            "size": 0,
            "size_actual": 0,
            "size_utilized": 0,
            "size_kb": 0,
            "size_kb_actual": 0,
            "size_kb_utilized": 0,
            "num_objects": 0
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
[root@folio15 hsbench]# 




secondary site
=====================
[root@folio18 ~]# ceph versions
{
    "mon": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 3
    },
    "mgr": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 3
    },
    "osd": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 24
    },
    "rgw": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 6
    },
    "overall": {
        "ceph version 18.2.0-99.el9cp (3907976ec457510fcff7ac9039193741cf5c7d2b) reef (stable)": 36
    }
}
[root@folio18 ~]# radosgw-admin zone get | grep zlib
                        "compression_type": "zlib"
[root@folio18 ~]#  radosgw-admin bucket stats --bucket testbucket1
{
    "bucket": "testbucket1",
    "num_shards": 11,
    "tenant": "",
    "versioning": "off",
    "zonegroup": "168926dc-aabe-4510-82ce-b1b3bab3448a",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "8dc62baa-afc5-496b-9ac1-692c719856ff.384645.1",
    "marker": "8dc62baa-afc5-496b-9ac1-692c719856ff.384645.1",
    "index_type": "Normal",
    "owner": "user1",
    "ver": "0#1,1#1,2#1,3#1,4#2,5#1,6#1,7#1,8#1,9#3,10#1",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
    "mtime": "2023-10-25T04:50:09.944840Z",
    "creation_time": "2023-10-25T04:50:09.941992Z",
    "max_marker": "0#,1#,2#,3#,4#00000000001.29716368.6,5#,6#,7#,8#,9#00000000002.29861985.6,10#",
    "usage": {
        "rgw.main": {
            "size": 52428800,
            "size_actual": 52428800,
            "size_utilized": 51042,
            "size_kb": 51200,
            "size_kb_actual": 51200,
            "size_kb_utilized": 50,
            "num_objects": 2
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}
[root@folio18 ~]#
Comment 9 errata-xmlrpc 2024-02-08 18:13:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.1 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2024:0747
Comment 10 Red Hat Bugzilla 2024-06-08 04:25:15 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days