Bug 1325322 – OSD hit suicide timeout or OSD goes down and comes back, When deleting large RBD images with features: striping, exclusive, object map

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1325322 - OSD hit suicide timeout or OSD goes down and comes back, When deleting large RBD images with features: striping, exclusive, object map

Summary:	OSD hit suicide timeout or OSD goes down and comes back, When deleting large ...

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	RBD (Show other bugs)
Sub Component:
Version:	1.3.2
Hardware:	x86_64 Linux

Priority	medium Severity medium
Target Milestone:	z2
Target Release:	3.0
Assigned To:	Jason Dillaman
QA Contact:	Manohar Murthy
Docs Contact:	Erin Donnelly

URL:
Whiteboard:
Keywords:

Depends On:	1548067
Blocks:	1557269 1348597 1372735
	Show dependency tree / graph

Reported:	2016-04-08 08:49 EDT by Vikhyat Umrao
Modified:	2018-04-26 13:39 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:	RHEL: ceph-12.2.4-1.el7cp Ubuntu: ceph_12.2.4-2redhat1
Doc Type:	Bug Fix
Doc Text:	* Previously, an attempt to delete a large RBD image with the "object map" feature enabled could cause the OSD nodes to trigger the "suicide_timeout" and self-terminate. With this update, deleting large RBD images with "object map" no longer causes OSDs to crash.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-04-26 13:38:39 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	2247541	None	None	None	2016-04-08 09:22 EDT
Ceph Project Bug Tracker	15121	None	None	None	2017-02-22 11:18 EST
Ceph Project Bug Tracker	17356	None	None	None	2017-02-22 10:34 EST
Ceph Project Bug Tracker	21797	None	None	None	2017-10-16 08:47 EDT
Red Hat Product Errata	RHBA-2018:1259	None	None	None	2018-04-26 13:39 EDT

Groups: None (edit)

Description Vikhyat Umrao 2016-04-08 08:49:10 EDT

Description of problem:
OSD hit suicide timeout or sometime OSD goes down and comes back, When deleting large RBD  images with features: striping, exclusive, object map

Version-Release number of selected component (if applicable):
Red Hat Ceph Storage 1.3.2 
ceph version 0.94.5-9.el7cp (deef183a81111fa5e128ec88c90a32c9587c615d)

How reproducible:
As per customer always 

Steps to Reproduce:
In a big cluster, try creating several 100TB+ RBD images with features striping, exclusive-lock object map. Some of them should produce the same behavior when you attempt to delete them (at least after writing data). Customer did not observe this on RBD images without object map. 

rbd image 'test':
        size 102400 TB in 26843545600 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.b66fc9238e1f29
        format: 2
        features: striping, exclusive, object map
        flags: object map invalid
        stripe unit: 512 kB
        stripe count: 8

Comment 2 Jason Dillaman 2016-04-08 08:57:14 EDT

An object map that is tracking 26843545600 objects will require >6GB of memory to store.

In the attached upstream ticket, we added a guard to prevent the use of object map on extremely large RBD images (>1PB) -- which tops out at 64MB of memory.  In the example above, that is a 100PB image.

Comment 9 Jason Dillaman 2016-04-12 21:20:42 EDT

Upstream PR: https://github.com/ceph/ceph/pull/8401

Comment 45 Ken Dreyer (Red Hat) 2018-02-22 13:20:10 EST

resolving with the v12.2.3 rebase

Comment 56 errata-xmlrpc 2018-04-26 13:38:39 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1259

Note You need to log in before you can comment on or make changes to this bug.