Bug 1425115 – MON - OSD communication, OSDs are not marked down when co-located with MONs

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1425115 - MON - OSD communication, OSDs are not marked down when co-located with MONs

Summary:	MON - OSD communication, OSDs are not marked down when co-located with MONs

Status:	CLOSED ERRATA

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	RADOS (Show other bugs)
Sub Component:
Version:	2.1
Hardware:	x86_64 Linux

Priority	high Severity high
Target Milestone:	rc
Target Release:	2.3
Assigned To:	Kefu Chai
QA Contact:	vidushi
Docs Contact:	Erin Donnelly

URL:
Whiteboard:
Keywords:	Regression

Depends On:
Blocks:	1437916
	Show dependency tree / graph

Reported:	2017-02-20 10:25 EST by Tomas Petr
Modified:	2017-07-30 11:16 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:	RHEL: ceph-10.2.7-2.el7cp Ubuntu: ceph_10.2.7-3redhat1xenial
Doc Type:	Bug Fix
Doc Text:	.An OSD failure no longer causes significant delay Previously, when an OSD and a Monitor were colocated on the same node and the node failed, Ceph waited some time before sending the note to the Monitor so the Monitor could decide if it wanted to mark the OSD as down. This could lead to a significant delay. With this update, when an OSD is known to be down, the cluster becomes aware immediately after the failure report, and Ceph sends the note to the Monitor right away.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-06-19 09:29:50 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Knowledge Base (Solution)	2937431	None	None	None	2017-02-27 08:49 EST
Ceph Project Bug Tracker	18104	None	None	None	2017-02-23 06:28 EST
Red Hat Product Errata	RHBA-2017:1497	normal	SHIPPED_LIVE	Red Hat Ceph Storage 2.3 bug fix and enhancement update	2017-06-19 13:24:11 EDT

Groups: None (edit)

Description Tomas Petr 2017-02-20 10:25:06 EST

Description of problem:
MON - OSD communication, OSDs are not marked down when co-located with MONs:
In the RHCS 2.1 co-located environment MONs and OSDs, if the co-located node is shuted down, the osds are NOT marked as down. The terminated monitor leaves quorum correctly. 
If is is during HW outage and the noout flag is set, the osds stay marked as "up" forever. If we manually perform "ceph osd down X", after 15 minutes f.e., all osds from this node are recognized as down.
If the noout flag is not set, the osds are marked as down and out after 300s.
Nodes that are pure ODS only are marked down correctly every time. 

Version-Release number of selected component (if applicable):
ceph version 10.2.3-17.el7cp

How reproducible:
Every time

Steps to Reproduce:
1. create  co-located RHCS 2 
2. shutdown one co-located node
3. osds are not recognized as down

Actual results:
osds are not recognized as down

Expected results:
osds are recognized as down

Additional info:
on RHCS 1.3 works fine

Comment 2 Vikhyat Umrao 2017-02-20 10:33:46 EST

Tested in RHCS 1.3.3(0.94.9-8.el7cp) all works fine. Added 'regression' keyword.

Comment 11 Shinobu KINJO 2017-02-20 22:26:25 EST

Under collocation both MONs and OSDs situation,

Comment 22 Vikhyat Umrao 2017-02-23 10:54:30 EST

jewel backport tracker: http://tracker.ceph.com/issues/18104
jewel backport patch: https://github.com/ceph/ceph/pull/12677/commits/dcc9483cbd605997876983ec65f6fb80d5da8a52

Comment 23 Josh Durgin 2017-03-31 21:33:39 EDT

Included in upstream 10.2.6.

Comment 33 errata-xmlrpc 2017-06-19 09:29:50 EDT

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497

Note You need to log in before you can comment on or make changes to this bug.