Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

This project is now read‑only. Starting Monday, February 2, please use https://ibm-ceph.atlassian.net/ for all bug tracking management.

Bug 2415656

Summary:	[RGW][Dedup]: Dedup background thread crashes while processing 500K identical 5MB objects
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Vidushi Mishra <vimishra>
Component:	RGW	Assignee:	Gabriel BenHanokh <gbenhano>
Status:	CLOSED ERRATA	QA Contact:	Vidushi Mishra <vimishra>
Severity:	urgent	Docs Contact:	Rivka Pollack <rpollack>
Priority:	unspecified
Version:	9.0	CC:	ceph-eng-bugs, cephqe-warriors, gbenhano, rpollack, tserlin
Target Milestone:	---
Target Release:	9.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-20.1.0-98	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2026-01-29 07:03:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Vidushi Mishra 2025-11-18 17:01:08 UTC

Description of problem:

RGW dedup background thread crashes with ceph_assert(val.count > 1) while deduplicating 500K identical 5 MB objects (2.4 TB total, ~5 MB unique), and dedup never completes.


Backtrace 
----------

2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: add_entry::COUNT=65534
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::testdedup1/dedup-obj-220858 was added successfully to table::loaded_objects=65534
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: validate::success
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: remap::Existing key: STANDARD is mapped to idx=0
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::bucket=testdedup1, obj=dedup-obj-220869, block_id=2047, rec_id=30, shared_manifest=0::num_parts=0::size_4k_units=1280::ETAG=b8fc857a25e7958868c2f003d5e0952d
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: add_entry::COUNT=65535
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::testdedup1/dedup-obj-220869 was added successfully to table::loaded_objects=65535
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: validate::success
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: remap::Existing key: STANDARD is mapped to idx=0
2025-11-18T12:26:19.053+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::bucket=testdedup1, obj=dedup-obj-220891, block_id=2047, rec_id=31, shared_manifest=0::num_parts=0::size_4k_units=1280::ETAG=b8fc857a25e7958868c2f003d5e0952d
2025-11-18T12:26:19.055+0000 7fef72c7b640 -1 /builddir/build/BUILD/ceph-20.1.0/src/rgw/rgw_dedup_table.cc: In function 'int rgw::dedup::dedup_table_t::add_entry(rgw::dedup::key_t*, rgw::dedup::disk_block_id_t, rgw::dedup::record_id_t, bool)' thread 7fef72c7b640 time 2025-11-18T12:26:19.054815+0000
/builddir/build/BUILD/ceph-20.1.0/src/rgw/rgw_dedup_table.cc: 140: FAILED ceph_assert(val.count > 1)

 ceph version 20.1.0-93.el9cp (a9ee38bff2b79ebf12e9894690705036936aad46) tentacle (rc - RelWithDebInfo)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7ff0e8e16f86]
 2: /usr/bin/radosgw(+0x4077d5) [0x55dc2dd397d5]
 3: (rgw::dedup::Background::add_record_to_dedup_table(rgw::dedup::dedup_table_t*, rgw::dedup::disk_record_t const*, rgw::dedup::disk_block_id_t, unsigned char, rgw::dedup::md5_stats_t*, rgw::dedup::remapper_t*)+0x2b2) [0x55dc2e2d5d52]
 4: (rgw::dedup::Background::process_all_slabs(rgw::dedup::dedup_table_t*, rgw::dedup::Background::dedup_step_t, unsigned short, unsigned short, unsigned int*, rgw::dedup::md5_stats_t*, rgw::dedup::disk_block_seq_t*, rgw::dedup::remapper_t*)+0x2f0) [0x55dc2e2ded60]
 5: (rgw::dedup::Background::objects_dedup_single_md5_shard(rgw::dedup::dedup_table_t*, unsigned short, rgw::dedup::md5_stats_t*, unsigned short)+0x183) [0x55dc2e2dfaa3]
 6: (rgw::dedup::Background::f_dedup_md5_shard(unsigned int, unsigned char*, unsigned long, unsigned short, unsigned short)+0x109) [0x55dc2e2e0149]
 7: (rgw::dedup::Background::process_all_shards(bool, int (rgw::dedup::Background::*)(unsigned int, unsigned char*, unsigned long, unsigned short, unsigned short), unsigned char*, unsigned long, unsigned short, unsigned short)+0x220) [0x55dc2e2d7500]
 8: (rgw::dedup::Background::run()+0xc22) [0x55dc2e2e7a72]
 9: /lib64/libstdc++.so.6(+0xdbae4) [0x7ff0e7e0cae4]
 10: /lib64/libc.so.6(+0x8b2ea) [0x7ff0e7abb2ea]
 11: /lib64/libc.so.6(+0x1103c0) [0x7ff0e7b403c0]

2025-11-18T12:26:19.056+0000 7fef72c7b640 -1 *** Caught signal (Aborted) **
 in thread 7fef72c7b640 thread_name:dedup_bg

 ceph version 20.1.0-93.el9cp (a9ee38bff2b79ebf12e9894690705036936aad46) tentacle (rc - RelWithDebInfo)
 1: /lib64/libc.so.6(+0x3fc30) [0x7ff0e7a6fc30]
 2: /lib64/libc.so.6(+0x8d02c) [0x7ff0e7abd02c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18d) [0x7ff0e8e16fe5]
 6: /usr/bin/radosgw(+0x4077d5) [0x55dc2dd397d5]
 7: (rgw::dedup::Background::add_record_to_dedup_table(rgw::dedup::dedup_table_t*, rgw::dedup::disk_record_t const*, rgw::dedup::disk_block_id_t, unsigned char, rgw::dedup::md5_stats_t*, rgw::dedup::remapper_t*)+0x2b2) [0x55dc2e2d5d52]
 8: (rgw::dedup::Background::process_all_slabs(rgw::dedup::dedup_table_t*, rgw::dedup::Background::dedup_step_t, unsigned short, unsigned short, unsigned int*, rgw::dedup::md5_stats_t*, rgw::dedup::disk_block_seq_t*, rgw::dedup::remapper_t*)+0x2f0) [0x55dc2e2ded60]
 9: (rgw::dedup::Background::objects_dedup_single_md5_shard(rgw::dedup::dedup_table_t*, unsigned short, rgw::dedup::md5_stats_t*, unsigned short)+0x183) [0x55dc2e2dfaa3]
 10: (rgw::dedup::Background::f_dedup_md5_shard(unsigned int, unsigned char*, unsigned long, unsigned short, unsigned short)+0x109) [0x55dc2e2e0149]
 11: (rgw::dedup::Background::process_all_shards(bool, int (rgw::dedup::Background::*)(unsigned int, unsigned char*, unsigned long, unsigned short, unsigned short), unsigned char*, unsigned long, unsigned short, unsigned short)+0x220) [0x55dc2e2d7500]
 12: (rgw::dedup::Background::run()+0xc22) [0x55dc2e2e7a72]
 13: /lib64/libstdc++.so.6(+0xdbae4) [0x7ff0e7e0cae4]
 14: /lib64/libc.so.6(+0x8b2ea) [0x7ff0e7abb2ea]
 15: /lib64/libc.so.6(+0x1103c0) [0x7ff0e7b403c0]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
 -9999> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: validate::success
 -9998> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: remap::Existing key: STANDARD is mapped to idx=0
 -9997> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::bucket=testdedup1, obj=dedup-obj-199017, block_id=1985, rec_id=16, shared_manifest=0::num_parts=0::size_4k_units=1280::ETAG=b8fc857a25e7958868c2f003d5e0952d
 -9996> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_entry::COUNT=63537
 -9995> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::testdedup1/dedup-obj-199017 was added successfully to table::loaded_objects=63537
 -9994> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: validate::success
 -9993> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: remap::Existing key: STANDARD is mapped to idx=0
 -9992> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::bucket=testdedup1, obj=dedup-obj-199028, block_id=1985, rec_id=17, shared_manifest=0::num_parts=0::size_4k_units=1280::ETAG=b8fc857a25e7958868c2f003d5e0952d
 -9991> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_entry::COUNT=63538
 -9990> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::testdedup1/dedup-obj-199028 was added successfully to table::loaded_objects=63538
 -9989> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: validate::success
 -9988> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: remap::Existing key: STANDARD is mapped to idx=0
 -9987> 2025-11-18T12:26:19.022+0000 7fef72c7b640 20 dedup background: add_record_to_dedup_table::bucket=testdedup1, obj=dedup-obj-199031, block_id=1985, rec_id=18, shared_manifest=0::num_parts=0::siz
Version-Release number of selected component (if applicable):
ceph version 20.1.0-93.el9cp (a9ee38bff2b79ebf12e9894690705036936aad46) tentacle (rc - RelWithDebInfo)


Dataset:
----------

Bucket: testdedup1

Number of objects: 500000

Object-size: 5 MB

Total overall size updated: 500K*5MB = ~2.4 TB

All objects have identical size and content (only one unique object)

How reproducible:
1/1

Steps to Reproduce:
1. Create bucket, 'testdedup1'
2. Upload 500K objects, each 5 MB, all with the same content.
3. cehck bucket stats that show ~2.4 TB and 500K objects.
4. Trigger dedup estimate
# radosgw-admin --rgw-realm usa dedup estimate
5. Periodically check dedup stats (over 30+ minutes)
radosgw-admin --rgw-realm usa dedup stats


Actual results:

Observed that dedup does not progress and eventually the dedup background thread crashes with an assert in rgw_dedup_table.cc.

Expected results:

- Dedup successfully processes all 500K objects.
- Deduplicate the 2.4 TB dataset down to ~5 MB of unique data

Additional info:

Comment 11 errata-xmlrpc 2026-01-29 07:03:52 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Ceph Storage 9.0 Security and Enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2026:1536