Bug 1488149 – [ceph-container] - dmcrypt - osds failed to start after node reboot

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1488149 - [ceph-container] - dmcrypt - osds failed to start after node reboot

Summary:	[ceph-container] - dmcrypt - osds failed to start after node reboot

Status:	CLOSED CURRENTRELEASE

Aliases:	None

Product:	Red Hat Ceph Storage
Classification:	Red Hat
Component:	Container (Show other bugs)
Sub Component:
Version:	2.4
Hardware:	Unspecified Unspecified

Priority	unspecified Severity high
Target Milestone:	rc
Target Release:	2.4
Assigned To:	leseb
QA Contact:	Harish NV Rao
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:	1503598
Blocks:	1473436 1479701
	Show dependency tree / graph

Reported:	2017-09-04 08:59 EDT by Vasishta
Modified:	2018-05-30 11:28 EDT (History)
CC List:	9 users (show)

See Also:
Fixed In Version:	ceph-2-rhel-7-docker-candidate-30274-20170914211241
Doc Type:	Bug Fix
Doc Text:	.Encrypted containerized OSDs starts as expected after a reboot Encrypted containerized OSD daemons failed to start after a reboot. In addition, the following log message was added to the OSD log file: ---- filestore(/var/lib/ceph/osd/bb-1) mount failed to open journal /var/lib/ceph/osd/bb-1/journal: (2) No such file or directory ---- This bug has been fixed, and such OSDs start as expected in this situation.
Story Points:	---
Clone Of:
Environment:
Last Closed:	2018-05-30 11:28:13 EDT
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
File contains log snippet of an OSD service (5.90 KB, text/plain) 2017-09-04 08:59 EDT, Vasishta	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

Groups: None (edit)

Description Vasishta 2017-09-04 08:59:00 EDT

Created attachment 1321841 [details]
File contains log snippet of an OSD service

Description of problem:

After upgrading cluster (OSDs - dmcrypt & dedicated journal) from 2.3 to 2.4, OSDs are failing to get started on rebooting the OSD node, saying mount 'failed to open journal'. 

Not sure whether this issue is dependent on upgrade or not, as upgrade worked fine. OSD services started and was running fine after following upgrade procedure (service start).

Version-Release number of selected component (if applicable):
ceph version 10.2.7-32.el7cp
brew-pulp-docker01.web.<-->:8888/rhceph:2.4

How reproducible:
Always (2/2)

Steps to Reproduce:
1. Upgrade a containerized ceph cluster with encrypted OSDs and dedicated journals from 2.3 to 2.4
2. Reboot an OSD Node 


Actual results:
OSD services are not getting started after node reboot
Log snippet - filestore(/var/lib/ceph/osd/bb-1) mount failed to open journal /var/lib/ceph/osd/bb-1/journal: (2) No such file or directory

Expected results:
OSD services must get started after node reboot

Comment 3 seb 2017-09-05 09:11:49 EDT

There is fix for this in 3.0.
Not sure how I can do a backport for this.

@Ken, which branch should I use to backport a fix?
Thanks!

Comment 5 seb 2017-09-05 10:13:14 EDT

lgtm.

Comment 10 leseb 2017-10-18 05:03:14 EDT

LGTM, thanks

Comment 11 Jon Schlueter 2017-10-18 10:14:35 EDT

Latest build of ceph2 container image is broken rhceph-rhel7-docker-2.4-3 see bug 1503598  The revert was not sufficient and complete to get us back to something that works.


$ docker run  -it  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhceph:2.4-3  version
common_functions.sh: line 3: disk_list.sh: No such file or directory

Comment 14 Ken Dreyer (Red Hat) 2018-05-30 11:28:13 EDT

please reopen if this is still an issue

Note You need to log in before you can comment on or make changes to this bug.