Bug 1869797 - 'ceph-volume raw prepare' fails to prepare because ceph-osd cannot acquire lock on device
Summary: 'ceph-volume raw prepare' fails to prepare because ceph-osd cannot acquire lo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: Ceph-Volume
Version: 4.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.2
Assignee: Sébastien Han
QA Contact: Ameena Suhani S H
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-08-18 16:40 UTC by Sébastien Han
Modified: 2021-01-12 14:56 UTC (History)
6 users (show)

Fixed In Version: ceph-14.2.11-88.el8cp, ceph-14.2.11-88.el7cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-12 14:56:31 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github ceph ceph pull 36700 0 None closed ceph-volume: retry when acquiring lock fails 2021-02-03 06:18:22 UTC
Red Hat Product Errata RHSA-2021:0081 0 None None None 2021-01-12 14:56:52 UTC

Description Sébastien Han 2020-08-18 16:40:26 UTC 
When preaparing the osd device with --mkfs, the ceph-osd binary tries to
acquire an exclusive lock on the device (soon to become an OSD).
Unfortunately, when running in containers, we have seen cases where
there is a race between ceph-osd and systemd-udevd to acquire a lock on
the device. Sometimes systemd-udevd gets the lock and releases it soon
so that the ceph-osd gets sometimes the lock is still held and because
ceph-osd uses LOCK_NB the command fails.

Beofre running --mkfs, ceph-volume call ceph-bluestore-tool which opens the devices seeking bluestore label, then close, this is where udev might be triggering its scan on the block.

This commit retries if the lock cannot be acquired, up to 5 times for 5
seconds, this should be more than enough to acquire the lock and
proceed with the OSD mkfs.

Unfortunately, this is so transient that we cannot lock earlier from c-v,
this won't do anything.

Ceph tracker bug: https://tracker.ceph.com/issues/47010
Comment 1 RHEL Program Management 2020-08-18 16:40:34 UTC 
Please specify the severity of this bug. Severity is defined here:
https://bugzilla.redhat.com/page.cgi?id=fields.html#bug_severity.
Comment 2 Sébastien Han 2020-08-18 16:42:53 UTC 
This race impacts OCS.
Comment 3 Sébastien Han 2020-08-19 13:00:18 UTC 
Moving to z3 since we worked around it in Rook and it seems to close to make it for z2.
Comment 4 Andrew Schoen 2020-11-05 16:38:58 UTC 
Moving this to 4.2. It doesn't seem to be a rush as there is already a workaround in rook and an upstream fix.
Comment 5 Sébastien Han 2020-11-13 14:08:20 UTC 
Andrew, there is no workaround in Rook, just the fix upstream.
Comment 12 Sébastien Han 2021-01-04 10:10:08 UTC 
We cannot introduce such a race, just verify the deployment is successful, also now the code retries so by looking at the prepare job logs you might see it retrying (meaning we are hitting the issue and retrying).
Comment 13 Sébastien Han 2021-01-04 10:10:38 UTC 
Checking for regression is fine too.
Comment 14 Ameena Suhani S H 2021-01-05 05:35:43 UTC 
As per comment #11 #12 #13

Verified using

Ceph Version: 14.2.11-95.el8cp
Ceph Ansible Version: 4.0.41-1.el8cp.noarch
Comment 16 errata-xmlrpc 2021-01-12 14:56:31 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Ceph Storage 4.2 Security and Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:0081
Note You need to log in before you can comment on or make changes to this bug.