.The OSD no longer times out when backfilling
Previously, backfilling objects with hundreds of thousands of `omap` entries could cause the OSD to time out. With this release, the backfilling process now reads a maximum of 8096 `omap` entries at a time, allowing the OSDs to continue running.
Description of problem:
[RHCS 2.y] OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled
Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 2.4
How reproducible:
Always in the scale lab
Additional info:
Testing in the scale lab uncovered an area where osds were
hitting heartbeat timeouts during backfill.
This turned out to be due to too many omap entries read in each 'chunk'
being backfilled, and only resetting the heartbeat once an entire chunk was assembled. We should ask Athena to tune this down for their
bucket index osds - setting:
Max number of omap entries per chunk:
osd_recovery_max_omap_entries_per_chunk = 1024
Josh is planning to add a patch for 3.0 and 2.5 to reset the heartbeat in middle of gathering a chunk, which should make the current default of 64000
entries per backfill chunk work even with large stores.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2018:0340
Description of problem: [RHCS 2.y] OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled Version-Release number of selected component (if applicable): Red Hat Ceph Storage 2.4 How reproducible: Always in the scale lab Additional info: Testing in the scale lab uncovered an area where osds were hitting heartbeat timeouts during backfill. This turned out to be due to too many omap entries read in each 'chunk' being backfilled, and only resetting the heartbeat once an entire chunk was assembled. We should ask Athena to tune this down for their bucket index osds - setting: Max number of omap entries per chunk: osd_recovery_max_omap_entries_per_chunk = 1024 Josh is planning to add a patch for 3.0 and 2.5 to reset the heartbeat in middle of gathering a chunk, which should make the current default of 64000 entries per backfill chunk work even with large stores.