Bug 2329523 - [rgw][listing]: list-object-versions fails on versioned bucket, with error marker failed to make forward progress [NEEDINFO]
Summary: [rgw][listing]: list-object-versions fails on versioned bucket, with error ma...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RGW
Version: 7.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 8.1
Assignee: Casey Bodley
QA Contact: sohan singh
Rivka Pollack
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-11-29 18:26 UTC by Vidushi Mishra
Modified: 2025-06-26 12:19 UTC (History)
7 users (show)

Fixed In Version: ceph-19.2.1-117.el9cp
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2025-06-26 12:19:44 UTC
Embargoed:
mbenjamin: needinfo? (ivancich)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-10306 0 None None None 2024-11-29 18:30:19 UTC
Red Hat Product Errata RHSA-2025:9775 0 None None None 2025-06-26 12:19:47 UTC

Description Vidushi Mishra 2024-11-29 18:26:45 UTC
Description of problem:

While performing aws s3api list-object-versions on a versioned bucket (testlistver1) with a total of 3 million objects (300K unique objects, 10 versions per object), the operation fails and logs the following error in the RGW logs: 
ERROR: list_objects_ordered marker failed to make forward progress

------------ snippet --------- 

2024-11-29T17:59:41.883+0000 7fd5fb924640  0 req 637514305345681421 0.545995593s s3:list_bucket ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=data/N27/good/193_tasking/2024-10-31/25/compacted-part-f55a5b45-f11f-4dd7-91e0-79658ca61548-0-187125[z8lfR.CsJLmA.hAKxEp6m-JHEAKlV.s], cur_marker=data/N27/good/193_tasking/2024-10-31/25/compacted-part-f55a5b45-f11f-4dd7-91e0-79658ca61548-0-187125[lwFYLniFq9EzkfmqMAtRm7HhXTlrbmz]

Version-Release number of selected component (if applicable):

ceph version 18.2.1-229.0.hotfix.Partnerbz2327880.el9cp 

How reproducible:
2/2

Steps to Reproduce:
1. create a bucket 'testlistver1' and enable versioning.

2. Populate the bucket with 300K unique objects, each having approximately 10 versions, resulting in a total of 3 million objects.

the objects are written in the path 
data/N27/good/193_tasking/2024-10-31/25/compacted-part-f55a5b45-f11f-4dd7-91e0-79658ca61548-0

3. After the upload is complete, we have an overall 3M objects in the bucket and the bucket is resharded to 1999 shards.

[root@depressa016 ~]#  date; time radosgw-admin bucket   stats --bucket testlistver1  | egrep 'num_objects|num_shards|id|bucket'
Fri Nov 29 06:18:28 PM UTC 2024
    "bucket": "testlistver1",
    "num_shards": 1999,
    "id": "019a4b7a-9ed6-444e-8eef-30e6292ef70b.454745.2",
            "num_objects": 3002575
    "bucket_quota": {

real	0m0.693s
user	0m0.122s
sys	0m0.052s


4. Perform list-object-versions on the bucket
#  aws --endpoint http://rgw_endpoint:80 s3api list-object-versions --bucket testlistver1

5. Observe the operation's behavior and monitor the RGW logs for errors.

Actual results:

- The operation fails to return results.
- RGW logs throw ERROR: list_objects_ordered marker failed to make forward progress

Expected results:

- The list-object-versions should fail

Follow the next comment for Additional info:

Comment 9 errata-xmlrpc 2025-06-26 12:19:44 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2025:9775


Note You need to log in before you can comment on or make changes to this bug.