Description of problem: almost all of the log objects download failing with 404, except few objects this is on 9.0 build (ceph version 20.1.0-30.el9cp), which is greater than this bz fixed in version (ceph-20.1.0-26) https://bugzilla.redhat.com/show_bug.cgi?id=2387075 [cephuser@depressa018 hsm4-90-dest-bkt1]$ date; aws --endpoint-url http://10.1.172.218:80 --profile hsm4 s3 sync s3://hsm4-90-dest-bkt1 . | grep failed Mon Oct 6 06:07:33 PM UTC 2025 download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-24-15-0000000000NE75WQ to ./hsm4-90-src-bkt1-logs-2025-10-06-12-24-15-0000000000NE75WQ argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-25-30-0000000001C6W5AK to ./hsm4-90-src-bkt1-logs-2025-10-06-12-25-30-0000000001C6W5AK argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-28-00-0000000003OP39MX to ./hsm4-90-src-bkt1-logs-2025-10-06-12-28-00-0000000003OP39MX argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-30-30-00000000058T8DJM to ./hsm4-90-src-bkt1-logs-2025-10-06-12-30-30-00000000058T8DJM argument of type 'NoneType' is not iterable [cephuser@depressa018 hsm4-90-dest-bkt1]$ I have triggered a 2M workload agin now of small objects(1-2KB) gc list is now populated with 50 entries [cephuser@depressa018 ~]$ radosgw-admin gc list --include-all | grep "tag" | wc -l 50 [cephuser@depressa018 ~]$ [cephuser@depressa018 ~]$ date; radosgw-admin gc list --include-all Mon Oct 6 06:28:06 PM UTC 2025 [ { "tag": "_4m3vAOAY3LAE9dQTJxQ6SwB-exZQSSA\u0000", "time": "2025-10-06T20:21:24.497207+0000", "objs": [ { "pool": "default.rgw.buckets.data", "oid": "cde4c7d8-83bc-45df-a6d5-25d5fb1d2220.244207.3__shadow_hsm4-90-src-bkt1-logs-2025-10-06-18-20-09-00000000138I0NDV0", "key": "", "instance": "" } ] }, .... { "tag": "_FuewayRuHk_TDFwnh6uch4ee-bwzkHb\u0000", "time": "2025-10-06T20:16:24.159636+0000", "objs": [ { "pool": "default.rgw.buckets.data", "oid": "cde4c7d8-83bc-45df-a6d5-25d5fb1d2220.244207.3__shadow_hsm4-90-src-bkt1-logs-2025-10-06-18-15-09-0000000009YIV8T10", "key": "", "instance": "" } ] } ] [cephuser@depressa018 ~]$ once gc process them, all of the above listed objects download is failing [cephuser@depressa018 log_objects]$ date; aws --endpoint-url http://10.1.172.218:80 --profile hsm4 s3 ls s3://hsm4-90-dest-bkt1 Tue Oct 7 06:18:01 AM UTC 2025 2025-10-06 12:25:30 16485437 hsm4-90-src-bkt1-logs-2025-10-06-12-24-15-0000000000NE75WQ 2025-10-06 12:26:45 16581005 hsm4-90-src-bkt1-logs-2025-10-06-12-25-30-0000000001C6W5AK 2025-10-06 12:28:00 15117063 hsm4-90-src-bkt1-logs-2025-10-06-12-26-45-0000000002VJ6M9B 2025-10-06 12:29:15 15938721 hsm4-90-src-bkt1-logs-2025-10-06-12-28-00-0000000003OP39MX 2025-10-06 12:30:30 15896820 hsm4-90-src-bkt1-logs-2025-10-06-12-29-15-0000000004RRG3D7 2025-10-06 12:31:45 17036303 hsm4-90-src-bkt1-logs-2025-10-06-12-30-30-00000000058T8DJM 2025-10-06 12:33:00 17303023 hsm4-90-src-bkt1-logs-2025-10-06-12-31-45-0000000006RRR4DQ 2025-10-06 18:13:54 8629979 hsm4-90-src-bkt1-logs-2025-10-06-12-33-00-0000000007UL3NQA 2025-10-06 18:15:09 14556608 hsm4-90-src-bkt1-logs-2025-10-06-18-13-54-0000000008F9DHJW 2025-10-06 18:16:24 15318960 hsm4-90-src-bkt1-logs-2025-10-06-18-15-09-0000000009YIV8T1 2025-10-06 18:17:39 15507440 hsm4-90-src-bkt1-logs-2025-10-06-18-16-24-0000000010ENNEGT 2025-10-06 18:18:54 15154784 hsm4-90-src-bkt1-logs-2025-10-06-18-17-39-00000000111GWK9I 2025-10-06 18:20:09 14156956 hsm4-90-src-bkt1-logs-2025-10-06-18-18-54-00000000125IENE6 2025-10-06 18:21:24 13662692 hsm4-90-src-bkt1-logs-2025-10-06-18-20-09-00000000138I0NDV 2025-10-06 18:22:39 14546440 hsm4-90-src-bkt1-logs-2025-10-06-18-21-24-0000000014QD5CNJ 2025-10-06 18:23:50 13295404 hsm4-90-src-bkt1-logs-2025-10-06-18-22-39-0000000015WOOP6L 2025-10-06 18:25:05 13080884 hsm4-90-src-bkt1-logs-2025-10-06-18-23-50-00000000160IICYS 2025-10-06 18:26:20 13977528 hsm4-90-src-bkt1-logs-2025-10-06-18-25-05-0000000017WEVCPI 2025-10-06 18:27:35 13970956 hsm4-90-src-bkt1-logs-2025-10-06-18-26-20-0000000018WI9X1K 2025-10-06 18:28:50 13344756 hsm4-90-src-bkt1-logs-2025-10-06-18-27-35-0000000019UW0E8W 2025-10-06 18:30:05 13721964 hsm4-90-src-bkt1-logs-2025-10-06-18-28-50-00000000201T6SWN 2025-10-06 18:31:20 13676456 hsm4-90-src-bkt1-logs-2025-10-06-18-30-05-0000000021NE332R 2025-10-06 18:32:35 13857744 hsm4-90-src-bkt1-logs-2025-10-06-18-31-20-0000000022D8PRCT 2025-10-06 18:33:50 13958308 hsm4-90-src-bkt1-logs-2025-10-06-18-32-35-00000000231UX5BJ 2025-10-06 18:35:05 13785576 hsm4-90-src-bkt1-logs-2025-10-06-18-33-50-0000000024FOMRFY [cephuser@depressa018 log_objects]$ [cephuser@depressa018 hsm4-90-dest-bkt1_temp1]$ date; aws --endpoint-url http://10.1.172.218:80 --profile hsm4 s3 sync s3://hsm4-90-dest-bkt1 . Tue Oct 7 06:18:38 AM UTC 2025 download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-24-15-0000000000NE75WQ to ./hsm4-90-src-bkt1-logs-2025-10-06-12-24-15-0000000000NE75WQ argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-28-00-0000000003OP39MX to ./hsm4-90-src-bkt1-logs-2025-10-06-12-28-00-0000000003OP39MX argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-30-30-00000000058T8DJM to ./hsm4-90-src-bkt1-logs-2025-10-06-12-30-30-00000000058T8DJM argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-33-00-0000000007UL3NQA to ./hsm4-90-src-bkt1-logs-2025-10-06-12-33-00-0000000007UL3NQA argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-13-54-0000000008F9DHJW to ./hsm4-90-src-bkt1-logs-2025-10-06-18-13-54-0000000008F9DHJW argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-25-30-0000000001C6W5AK to ./hsm4-90-src-bkt1-logs-2025-10-06-12-25-30-0000000001C6W5AK argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-15-09-0000000009YIV8T1 to ./hsm4-90-src-bkt1-logs-2025-10-06-18-15-09-0000000009YIV8T1 argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-17-39-00000000111GWK9I to ./hsm4-90-src-bkt1-logs-2025-10-06-18-17-39-00000000111GWK9I argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-21-24-0000000014QD5CNJ to ./hsm4-90-src-bkt1-logs-2025-10-06-18-21-24-0000000014QD5CNJ argument of type 'NoneType' is not iterable download: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-26-45-0000000002VJ6M9B to ./hsm4-90-src-bkt1-logs-2025-10-06-12-26-45-0000000002VJ6M9B download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-20-09-00000000138I0NDV to ./hsm4-90-src-bkt1-logs-2025-10-06-18-20-09-00000000138I0NDV argument of type 'NoneType' is not iterable download: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-31-45-0000000006RRR4DQ to ./hsm4-90-src-bkt1-logs-2025-10-06-12-31-45-0000000006RRR4DQ download: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-12-29-15-0000000004RRG3D7 to ./hsm4-90-src-bkt1-logs-2025-10-06-12-29-15-0000000004RRG3D7 download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-18-54-00000000125IENE6 to ./hsm4-90-src-bkt1-logs-2025-10-06-18-18-54-00000000125IENE6 argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-27-35-0000000019UW0E8W to ./hsm4-90-src-bkt1-logs-2025-10-06-18-27-35-0000000019UW0E8W argument of type 'NoneType' is not iterable download: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-16-24-0000000010ENNEGT to ./hsm4-90-src-bkt1-logs-2025-10-06-18-16-24-0000000010ENNEGT download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-23-50-00000000160IICYS to ./hsm4-90-src-bkt1-logs-2025-10-06-18-23-50-00000000160IICYS argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-25-05-0000000017WEVCPI to ./hsm4-90-src-bkt1-logs-2025-10-06-18-25-05-0000000017WEVCPI argument of type 'NoneType' is not iterable download: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-22-39-0000000015WOOP6L to ./hsm4-90-src-bkt1-logs-2025-10-06-18-22-39-0000000015WOOP6L download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-26-20-0000000018WI9X1K to ./hsm4-90-src-bkt1-logs-2025-10-06-18-26-20-0000000018WI9X1K argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-30-05-0000000021NE332R to ./hsm4-90-src-bkt1-logs-2025-10-06-18-30-05-0000000021NE332R argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-28-50-00000000201T6SWN to ./hsm4-90-src-bkt1-logs-2025-10-06-18-28-50-00000000201T6SWN argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-31-20-0000000022D8PRCT to ./hsm4-90-src-bkt1-logs-2025-10-06-18-31-20-0000000022D8PRCT argument of type 'NoneType' is not iterable download failed: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-32-35-00000000231UX5BJ to ./hsm4-90-src-bkt1-logs-2025-10-06-18-32-35-00000000231UX5BJ argument of type 'NoneType' is not iterable download: s3://hsm4-90-dest-bkt1/hsm4-90-src-bkt1-logs-2025-10-06-18-33-50-0000000024FOMRFY to ./hsm4-90-src-bkt1-logs-2025-10-06-18-33-50-0000000024FOMRFY [cephuser@depressa018 hsm4-90-dest-bkt1_temp1]$ Version-Release number of selected component (if applicable): ceph version 20.1.0-30.el9cp How reproducible: always Steps to Reproduce: 1.deploy rhcs9.0 cluster with rgw daemons 2.create src and dest buckets, configure bucket logging 3.trigger a 1M or 2M workload of small sized objects(1-10KB). Actual results: gc list shows entries for log-objects once the worload is completed. and the object download fails after gc process them Expected results: expected no race condition occur during parallel upload that deletes the log objects later with gc Additional info: env details: [cephuser@depressa018 ~]$ ceph orch host ls HOST ADDR LABELS STATUS depressa015 10.1.172.215 _admin,mgr,installer,osd,mon depressa016 10.1.172.216 mgr,osd,rgw,mon depressa017 10.1.172.217 mgr,osd,rgw,mon depressa018 10.1.172.218 client,mgr,rgw,osd,mon 4 hosts in cluster [cephuser@depressa018 ~]$ creds: root/r
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2026:1536