Bug 1336756
| Summary: | ceph-disk prepare: Error: partprobe /dev/vdb failed : Error: Error informing the kernel about modifications to partition /dev/vdb1 -- Device or resource busy. | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Ceph Storage | Reporter: | Daniel Horák <dahorak> | |
| Component: | Ceph-Disk | Assignee: | Ken Dreyer (Red Hat) <kdreyer> | |
| Status: | CLOSED ERRATA | QA Contact: | Daniel Horák <dahorak> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | 2.0 | CC: | ceph-qe-bugs, dahorak, hnallurv, kdreyer, mbukatov, nlevine, rcyriac | |
| Target Milestone: | rc | |||
| Target Release: | 2.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | parted-3.1-26.el7 ceph-10.2.1-11.el7cp | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1339705 (view as bug list) | Environment: | ||
| Last Closed: | 2016-08-23 19:38:42 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1339705 | |||
Thanks for the steps to reproduce, that's very helpful. I think I have all the information I need now. small update/note: I didn't see this issue and wasn't able to reproduce it on VMs in OpenStack environment, but I saw it (and can reproduce it) on kvm VMs running on our (different) physical servers. I've tried to update parted from Fedora 22 (to version parted-3.2-16.fc22) - as it was suggested in the upstream issue[1] - and I can confirm that it fixes the issue. Originally with parted-3.1-23.el7.x86_64 it was failing. [1] http://tracker.ceph.com/issues/15918 https://github.com/ceph/ceph/pull/9195 is the PR to master; still undergoing review upstream. Hi Daniel, Would you be so kind as to provide me with access to a machine where I can reproduce the problem. I've collected enough expertise now to make use of it. And I can't seem to reproduce it on a CentOS 7.2 (VM or bare metal). Thanks ! See comment 7. This BZ needs to be resolved ASAP as it is a blocker for Beta 1 (5/31). I've opened bug 1339705 against parted to track an improvement in partprobe. qa_ack given Tested and VERIFIED on VMs accordingly to comment 0 and comment 9 with following packages: # rpm -qa parted ceph-osd ceph-osd-10.2.2-5.el7cp.x86_64 parted-3.1-26.el7.x86_64 I'll try to retest it also on real HW. Tested on physical HW server without any problem.
With following packages:
parted-3.1-26.el7.x86_64
ceph-osd-10.2.2-2.el7cp.x86_64
>> VERIFIED
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2016-1755.html |
Description of problem: Command `ceph-disk prepare ...` sometimes fails to prepare disk for a Ceph OSD with following error: ceph-disk: Error: partprobe /dev/vdb failed : Error: Error informing the kernel about modifications to partition /dev/vdb1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/vdb1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting. Error: Failed to add partition 1 (Device or resource busy) Version-Release number of selected component (if applicable): ceph-base-10.2.0-1.el7cp.x86_64 ceph-common-10.2.0-1.el7cp.x86_64 ceph-osd-10.2.0-1.el7cp.x86_64 ceph-selinux-10.2.0-1.el7cp.x86_64 libcephfs1-10.2.0-1.el7cp.x86_64 python-cephfs-10.2.0-1.el7cp.x86_64 How reproducible: 40% on our VMs Steps to Reproduce: 1. Create and install node for Ceph OSD with at least two spare disks. 2. Run command for disk preparation for a Ceph OSD. Device /dev/vdb is targeted for journal, /dev/vdc for OSD data. If you have more spare disks, you might try to repeat this command for each "OSD data" device. # ceph-disk prepare --cluster ceph /dev/vdc /dev/vdb 3. Before trying again, clean up both the journal and OSD data devices: # sgdisk --zap-all --clear --mbrtogpt -g -- /dev/vdb # sgdisk --zap-all --clear --mbrtogpt -g -- /dev/vdc Actual results: Sometimes the ceph-disk command fails with following (or similar) error: # ceph-disk prepare --cluster ceph /dev/vdc /dev/vdb prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data The operation has completed successfully. ceph-disk: Error: partprobe /dev/vdb failed : Error: Error informing the kernel about modifications to partition /dev/vdb1 -- Device or resource busy. This means Linux won't know about any changes you made to /dev/vdb1 until you reboot -- so you shouldn't mount it or use it in any way before rebooting. Error: Failed to add partition 1 (Device or resource busy) # echo $? 1 Expected results: Command ceph-disk should properly prepare the disk for Ceph OSD. Additional info: I discovered this issue while testing USM.