Bug 1505561 – [RHCS 2.y] OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1505561 - [RHCS 2.y] OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled

Summary:	[RHCS 2.y] OSD heartbeat timeout due to too many omap entries read in each 'c...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	RADOS (Show other bugs)
Sub Component:
Version:	2.4
Hardware:	x86_64 Linux

Priority	high Severity medium
Target Milestone:	rc
Target Release:	2.5
Assigned To:	Josh Durgin
QA Contact:	Manohar Murthy
Docs Contact:	Aron Gunn

URL:
Whiteboard:
Keywords:

Depends On:	1505559
Blocks:	1536401
	Show dependency tree / graph

Reported:	2017-10-23 17:13 EDT by Vikhyat Umrao
Modified:	2018-02-21 14:44 EST (History)
CC List:	9 users (show)

See Also:	1505559
Fixed In Version:	RHEL: ceph-10.2.10-11.el7cp Ubuntu: ceph_10.2.10-8redhat1xenial
Doc Type:	Bug Fix
Doc Text:	.The OSD no longer times out when backfilling Previously, backfilling objects with hundreds of thousands of `omap` entries could cause the OSD to time out. With this release, the backfilling process now reads a maximum of 8096 `omap` entries at a time, allowing the OSDs to continue running.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-02-21 14:44:55 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	3222691	None	None	None	2017-10-23 17:36 EDT
Ceph Project Bug Tracker	21897	None	None	None	2017-10-23 17:23 EDT
Red Hat Product Errata	RHBA-2018:0340	normal	SHIPPED_LIVE	Red Hat Ceph Storage 2.5 bug fix and enhancement update	2018-02-21 19:50:32 EST

Groups: None (edit)

Description Vikhyat Umrao 2017-10-23 17:13:58 EDT

Description of problem:
[RHCS 2.y] OSD heartbeat timeout due to too many omap entries read in each 'chunk' being backfilled


Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 2.4 

How reproducible:
Always in the scale lab


Additional info:

Testing in the scale lab uncovered an area where osds were
hitting heartbeat timeouts during backfill.

This turned out to be due to too many omap entries read in each 'chunk'
being backfilled, and only resetting the heartbeat once an entire chunk was assembled. We should ask Athena to tune this down for their
bucket index osds - setting:

Max number of omap entries per chunk:

    osd_recovery_max_omap_entries_per_chunk = 1024

Josh is planning to add a patch for 3.0 and 2.5 to reset the heartbeat in middle of gathering a chunk, which should make the current default of 64000
entries per backfill chunk work even with large stores.

Comment 16 errata-xmlrpc 2018-02-21 14:44:55 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0340

Note You need to log in before you can comment on or make changes to this bug.