Bug 1491780
| Summary: | ceph: Racing between partition creation & device node creation | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | Ceph-Disk | Assignee: | Kefu Chai <kchai> |
| Status: | CLOSED ERRATA | QA Contact: | Warren <wusui> |
| Severity: | urgent | Docs Contact: | Bara Ancincova <bancinco> |
| Priority: | urgent | ||
| Version: | 2.4 | CC: | dyasny, gfidente, gmeno, jdurgin, johfulto, kchai, lhh, nlevine, sasha, shan, srevivo, tserlin, vakulkar, wusui |
| Target Milestone: | rc | Keywords: | Triaged |
| Target Release: | 2.4 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | RHEL: ceph-10.2.7-34.el7cp Ubuntu: ceph_10.2.7-35redhat1 | Doc Type: | Bug Fix |
| Doc Text: |
.`ceph-disk` retries up to ten times to find files that represents newly created OSD partitions
When deploying a new OSD with the `ceph-ansible` playbook, the file under the `/sys/` directory that represents a newly created OSD partition failed to show up right after the `partprobe` utility returned it. Consequently, the `ceph-disk` utility failed to activate the OSD, and `ceph-ansible` could not deploy the OSD successfully. With this update, if `ceph-disk` cannot find the file, it retries up to ten times to find it before it terminates. As a result, `ceph-disk` can activate the newly prepared OSD as expected.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-10-17 18:12:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1479701, 1496985 | ||
|
Description
Alexander Chuzhoy
2017-09-14 16:08:27 UTC
Background on this is below. Question we should verify: Does the ceph-disk that is being used in the puddle have the fix in tracker.ceph.com/issues/19428? ---------- Forwarded message ---------- From: John Fulton <johfulto> Date: Wed, Sep 13, 2017 at 6:09 PM Subject: ceph-disk race condition To: Sasha Chuzhoy <sasha> Cc: Giulio Fidente <gfidente>, Sebastien Han <shan> Hi Sasha, Following up our conversation today. You used ceph-ansible-3.0.0-0.1.rc6.el7cp.noarch and manually applied the following patch to your THT, which you should do as that ceph-ansible version requires it. https://review.openstack.org/#/c/501983/3/docker/services/ceph-ansible/ceph-osd.yaml You then deployed 3 ceph-storage in your overcloud. 2 succeeded and 1 failed. The one that failed had the following error: http://sprunge.us/fMGj The above error is a ceph-disk error race condition which was fixed in a newer version of ceph: http://tracker.ceph.com/issues/19428 Sasha, Which version of ceph-disk are you using as provided by the ceph-osd package on your Ceph Storage node. Here is an example of it on my upstream system: [root@overcloud-cephstorage-0 ~]# yum whatprovides */ceph-disk Loaded plugins: fastestmirror, priorities Loading mirror speeds from cached hostfile 1020 packages excluded due to repository priority protections 1:ceph-osd-10.2.7-0.el7.x86_64 : Ceph Object Storage Daemon Repo : quickstart-centos-ceph-jewel Matched from: Filename : /usr/sbin/ceph-disk 1:ceph-osd-10.2.7-0.el7.x86_64 : Ceph Object Storage Daemon Repo : @quickstart-centos-ceph-jewel Matched from: Filename : /usr/sbin/ceph-disk [root@overcloud-cephstorage-0 ~]# Thanks John So there's no ceph-disk on the ceph storage node. It's in the respective container (ceph-osd-overcloud-cephstorage-0-devvdb): [root@overcloud-cephstorage-0 /]# rpm -qf `which ceph-disk` ceph-osd-10.2.7-28.el7cp.x86_64 > Does the ceph-disk that is being used in the puddle have the fix in tracker.ceph.com/issues/19428? no, the upstream backport PR[0] targeting jewel was included by 10.2.8[1] and up. i also cross checked using git cli by checking the two commits in the backport PR: $ git tag --contains a20d2b89ee13e311cf1038c54ecadae79b68abd5 v10.2.8 v10.2.9 $ git tag --contains 2d5d0aec60ec9689d44a53233268e9b9dd25df95 v10.2.8 v10.2.9 since rhcs 3.0 will be based on luminous, we can consider this issue addressed. -- [0] https://github.com/ceph/ceph/pull/14329 [1] http://tracker.ceph.com/issues/19493, see target version. On a downstream system (sealusa3) with this issue ceph -v returns the following [root@20eceada4d09 /]# ceph -v ceph version 10.2.7-28.el7cp (216cda64fd9a9b43c4b0c2f8c402d36753ee35f7) [root@20eceada4d09 /]# without this fix, user might not be able to deploy OSD successfully. so i updated the Doc Text field. Automated tests have not produced a problem. Marking as verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2903 |