+++ This bug was initially created as a clone of Bug #2257373 +++

Observing output format change for radosgw-admin bucket check command in 6.1z4


Description of problem:
We are observing output format change for radosgw-admin bucket check command in 6.1z4

With 5.3z6, 6.1z3-async, 7.0: radosgw-admin bucket check command output will display only the list of orphan entries for multipart objects.

whereas with 6.1z4: radosgw-admin bucket check command output displays list of orphan entries for multipart objects along with "existing_header" and "calculated_header" data.(Parsing output as json throws error: json.decoder.JSONDecodeError: Extra data)
log: http://magna002.ceph.redhat.com/ceph-qe-logs/Chaithra/cephci-run-MDFYBZ/Test_functionality_of_bucket_check_fix_0.log

Version-Release number of selected component (if applicable):
ceph version 17.2.6-181.el9cp

How reproducible:
3/3

Steps to Reproduce:
1. Deploy cluster with ceph version 17.2.6-181.el9cp
2. Upload multipart object such a way that orphen entries are available
3. execute: "radosgw-admin bucket check --bucket <bucket.name>"

Actual results:
radosgw-admin bucket check command output displays list of orphan entries for multipart objects along with "existing_header" and "calculated_header" data.

Expected results:
radosgw-admin bucket check command output will display only the list of orphan entries for multipart objects

Additional info:

Outputs:
*******************************************
With 6.1z4: ceph version 17.2.6-174.el9cp, ceph version 17.2.6-181.el9cp
# radosgw-admin bucket check --bucket=davidl.956-bucky-4567-0
[
    "_multipart_key_davidl.956-bucky-4567-0_0.2~GCeFgWr-XapcVgxpMxhdzB7kAFEyx4f.1",
    "_multipart_key_davidl.956-bucky-4567-0_0.2~GCeFgWr-XapcVgxpMxhdzB7kAFEyx4f.2",
    "_multipart_key_davidl.956-bucky-4567-0_0.2~GCeFgWr-XapcVgxpMxhdzB7kAFEyx4f.3"
]
{
    "existing_header": {
        "usage": {
            "rgw.main": {
                "size": 629145600,
                "size_actual": 629145600,
                "size_utilized": 629145600,
                "size_kb": 614400,
                "size_kb_actual": 614400,
                "size_kb_utilized": 614400,
                "num_objects": 3
            },
            "rgw.multimeta": {
                "size": 0,
                "size_actual": 0,
                "size_utilized": 65,
                "size_kb": 0,
                "size_kb_actual": 0,
                "size_kb_utilized": 1,
                "num_objects": 1
            }
        }
    },
    "calculated_header": {
        "usage": {
            "rgw.main": {
                "size": 629145600,
                "size_actual": 629145600,
                "size_utilized": 629145600,
                "size_kb": 614400,
                "size_kb_actual": 614400,
                "size_kb_utilized": 614400,
                "num_objects": 3
            }
        }
    }
}

*******************************************

With 6.1z3-async: ceph version 17.2.6-170.el9cp
#radosgw-admin bucket check --bucket=georgek.991-bucky-287-0
[
"_multipart_key_georgek.991-bucky-287-0_0.2~q7mUnDqn6zOO934S4a-LZ7GFEKW-IV6.1",
"_multipart_key_georgek.991-bucky-287-0_0.2~q7mUnDqn6zOO934S4a-LZ7GFEKW-IV6.2",
"_multipart_key_georgek.991-bucky-287-0_0.2~q7mUnDqn6zOO934S4a-LZ7GFEKW-IV6.3",
"_multipart_key_georgek.991-bucky-287-0_0.2~q7mUnDqn6zOO934S4a-LZ7GFEKW-IV6.4",
"_multipart_key_georgek.991-bucky-287-0_0.2~q7mUnDqn6zOO934S4a-LZ7GFEKW-IV6.5",
"_multipart_key_georgek.991-bucky-287-0_0.2~q7mUnDqn6zOO934S4a-LZ7GFEKW-IV6.6"
]

*******************************************

With 7.0: ceph version 18.2.0-131.el9cp
#radosgw-admin bucket check --bucket=cathyb.308-bucky-4524-0
[
"_multipart_key_cathyb.308-bucky-4524-0_0.2~uZb6cOzGLMoV2axwtmW19H-AVDe7HKH.1",
"_multipart_key_cathyb.308-bucky-4524-0_0.2~uZb6cOzGLMoV2axwtmW19H-AVDe7HKH.2",
"_multipart_key_cathyb.308-bucky-4524-0_0.2~uZb6cOzGLMoV2axwtmW19H-AVDe7HKH.3",
"_multipart_key_cathyb.308-bucky-4524-0_0.2~uZb6cOzGLMoV2axwtmW19H-AVDe7HKH.4"
]

*******************************************

With 5.3z6: ceph version 16.2.10-233.el8cp
#radosgw-admin bucket check --bucket=jamess.943-bucky-4518-0
[
"_multipart_key_jamess.943-bucky-4518-0_0.2~pRCo7Maf8eP4vuVLzeGY5t0i0SZXhbR.1"'
]

*******************************************

--- Additional comment from Casey Bodley on 2024-01-09 17:22:27 UTC ---

changes to this command's output were expected from https://github.com/ceph/ceph/pull/52576. i see that those commits were cherry-picked for https://bugzilla.redhat.com/show_bug.cgi?id=2167318

however, it's missing followup commits from https://github.com/ceph/ceph/pull/53607. without those additional commits, the command output can't be parsed as a json document because it contains two json documents (an array and a dictionary) one after the other. this regression was tracked in https://tracker.ceph.com/issues/62956

--- Additional comment from Chaithra on 2024-01-10 07:12:56 UTC ---

Thanks Casey for the confirmation.
Adding Regression keyword and targeting to 6.1z4.

Thanks,
Chaithra

--- Additional comment from RHEL Program Management on 2024-01-10 07:13:07 UTC ---

This bug report has Keywords: Regression or TestBlocker.

Since no regressions or test blockers are allowed between releases, it is being proposed as a blocker for this release.

Please resolve \triage ASAP.

--- Additional comment from Matt Benjamin (redhat) on 2024-01-11 22:05:37 UTC ---

Hi Chaithra,

1. I don't think the severity of this bz is in at all high;  it doesn't matter whether it "used to work"
2. this should have been created as a clone of the bz in 7.1, not just moved to 6.1z4, since we still need to fix it there--and maybe in 7.0z

thanks,

Matt