Bug 2329523
| Summary: | [rgw][listing]: list-object-versions fails on versioned bucket, with error marker failed to make forward progress | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Vidushi Mishra <vimishra> |
| Component: | RGW | Assignee: | Casey Bodley <cbodley> |
| Status: | CLOSED ERRATA | QA Contact: | sohan singh <sohsingh> |
| Severity: | high | Docs Contact: | Rivka Pollack <rpollack> |
| Priority: | unspecified | ||
| Version: | 7.1 | CC: | ceph-eng-bugs, cephqe-warriors, ivancich, mbenjamin, mkasturi, rpollack, tserlin |
| Target Milestone: | --- | Flags: | mbenjamin:
needinfo?
(ivancich) |
| Target Release: | 8.1 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-19.2.1-117.el9cp | Doc Type: | No Doc Update |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2025-06-26 12:19:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Ceph Storage 8.1 security, bug fix, and enhancement updates), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2025:9775 |
Description of problem: While performing aws s3api list-object-versions on a versioned bucket (testlistver1) with a total of 3 million objects (300K unique objects, 10 versions per object), the operation fails and logs the following error in the RGW logs: ERROR: list_objects_ordered marker failed to make forward progress ------------ snippet --------- 2024-11-29T17:59:41.883+0000 7fd5fb924640 0 req 637514305345681421 0.545995593s s3:list_bucket ERROR: list_objects_ordered marker failed to make forward progress; attempt=2, prev_marker=data/N27/good/193_tasking/2024-10-31/25/compacted-part-f55a5b45-f11f-4dd7-91e0-79658ca61548-0-187125[z8lfR.CsJLmA.hAKxEp6m-JHEAKlV.s], cur_marker=data/N27/good/193_tasking/2024-10-31/25/compacted-part-f55a5b45-f11f-4dd7-91e0-79658ca61548-0-187125[lwFYLniFq9EzkfmqMAtRm7HhXTlrbmz] Version-Release number of selected component (if applicable): ceph version 18.2.1-229.0.hotfix.Partnerbz2327880.el9cp How reproducible: 2/2 Steps to Reproduce: 1. create a bucket 'testlistver1' and enable versioning. 2. Populate the bucket with 300K unique objects, each having approximately 10 versions, resulting in a total of 3 million objects. the objects are written in the path data/N27/good/193_tasking/2024-10-31/25/compacted-part-f55a5b45-f11f-4dd7-91e0-79658ca61548-0 3. After the upload is complete, we have an overall 3M objects in the bucket and the bucket is resharded to 1999 shards. [root@depressa016 ~]# date; time radosgw-admin bucket stats --bucket testlistver1 | egrep 'num_objects|num_shards|id|bucket' Fri Nov 29 06:18:28 PM UTC 2024 "bucket": "testlistver1", "num_shards": 1999, "id": "019a4b7a-9ed6-444e-8eef-30e6292ef70b.454745.2", "num_objects": 3002575 "bucket_quota": { real 0m0.693s user 0m0.122s sys 0m0.052s 4. Perform list-object-versions on the bucket # aws --endpoint http://rgw_endpoint:80 s3api list-object-versions --bucket testlistver1 5. Observe the operation's behavior and monitor the RGW logs for errors. Actual results: - The operation fails to return results. - RGW logs throw ERROR: list_objects_ordered marker failed to make forward progress Expected results: - The list-object-versions should fail Follow the next comment for Additional info: