Bug 1869797

Summary:	'ceph-volume raw prepare' fails to prepare because ceph-osd cannot acquire lock on device
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Sébastien Han <shan>
Component:	Ceph-Volume	Assignee:	Sébastien Han <shan>
Status:	CLOSED ERRATA	QA Contact:	Ameena Suhani S H <amsyedha>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.1	CC:	aschoen, assingh, ceph-eng-bugs, ceph-qe-bugs, tserlin, vereddy
Target Milestone:	---
Target Release:	4.2
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-14.2.11-88.el8cp, ceph-14.2.11-88.el7cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-01-12 14:56:31 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Sébastien Han 2020-08-18 16:40:26 UTC

When preaparing the osd device with --mkfs, the ceph-osd binary tries to
acquire an exclusive lock on the device (soon to become an OSD).
Unfortunately, when running in containers, we have seen cases where
there is a race between ceph-osd and systemd-udevd to acquire a lock on
the device. Sometimes systemd-udevd gets the lock and releases it soon
so that the ceph-osd gets sometimes the lock is still held and because
ceph-osd uses LOCK_NB the command fails.

Beofre running --mkfs, ceph-volume call ceph-bluestore-tool which opens the devices seeking bluestore label, then close, this is where udev might be triggering its scan on the block.

This commit retries if the lock cannot be acquired, up to 5 times for 5
seconds, this should be more than enough to acquire the lock and
proceed with the OSD mkfs.

Unfortunately, this is so transient that we cannot lock earlier from c-v,
this won't do anything.

Ceph tracker bug: https://tracker.ceph.com/issues/47010

Comment 1 RHEL Program Management 2020-08-18 16:40:34 UTC

Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.

Comment 2 Sébastien Han 2020-08-18 16:42:53 UTC

This race impacts OCS.

Comment 3 Sébastien Han 2020-08-19 13:00:18 UTC

Moving to z3 since we worked around it in Rook and it seems to close to make it for z2.

Comment 4 Andrew Schoen 2020-11-05 16:38:58 UTC

Moving this to 4.2. It doesn't seem to be a rush as there is already a workaround in rook and an upstream fix.

Comment 5 Sébastien Han 2020-11-13 14:08:20 UTC

Andrew, there is no workaround in Rook, just the fix upstream.

Comment 12 Sébastien Han 2021-01-04 10:10:08 UTC

We cannot introduce such a race, just verify the deployment is successful, also now the code retries so by looking at the prepare job logs you might see it retrying (meaning we are hitting the issue and retrying).

Comment 13 Sébastien Han 2021-01-04 10:10:38 UTC

Checking for regression is fine too.

Comment 14 Ameena Suhani S H 2021-01-05 05:35:43 UTC

As per comment #11 #12 #13

Verified using

Ceph Version: 14.2.11-95.el8cp
Ceph Ansible Version: 4.0.41-1.el8cp.noarch

Comment 16 errata-xmlrpc 2021-01-12 14:56:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0081