|Summary:||ceph-installer creates RBDs with currently unmappable features activated|
|Product:||Red Hat Ceph Storage||Reporter:||Chris Blum <cblum>|
|Component:||Ceph-Installer||Assignee:||Christina Meno <gmeno>|
|Status:||CLOSED WONTFIX||QA Contact:||ceph-qe-bugs <ceph-qe-bugs>|
|Severity:||unspecified||Docs Contact:||Bara Ancincova <bancinco>|
|Version:||3.0||CC:||adeza, aschoen, bancinco, ceph-eng-bugs, flucifre, gmeno, jdillama, khartsoe, mhackett, nthomas, sankarshan|
|Fixed In Version:||Doc Type:||Known Issue|
.Certain image features are not supported with the RBD kernel module The following image features are not supported with the current version of the RADOS Block Device (RBD) kernel module (`krbd`) that is included in Red Hat Enterprise Linux 7.4: * `object-map` * `deep-flatten` * `journaling` * `fast-diff` However, by default the `ceph-installer` utility creates RBDs with the aforementioned features enabled. As a consequence, an attempt to map the kernel RBDs by running the `rbd map` command fails. To work around this issue, disable the unsupported features by setting the `rbd_default_features = 1` option in the Ceph configuration file for kernel RBDs or dynamically disable them by running the following command: ---- rbd feature disable <image> <feature> ---- This issue is a limitation only in kernel RBDs, and the features work as expected with user-space RBDs.
|Last Closed:||2018-02-15 21:59:16 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
|Bug Depends On:|
|Bug Blocks:||1322504, 1383917, 1412948, 1494421|
Description Chris Blum 2016-05-26 12:20:05 UTC
Description of problem: Currently the RBDs created with the RHS-C look like this: [root@rhcs1 ceph]# rbd info testRBD rbd image 'testRBD': size 10240 MB in 2560 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.105b74b0dc51 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten <---- flags: Because of https://access.redhat.com/solutions/2258301 this will prevent rbds to be mapped with these features activated until the GA of RHEL7.3 (in Autumn) Suggestion: Create new RBDs without those features to make them mappable in RHEL7.2. In a non-RHS-C setup the default RBD looks like this: [root@mon ceph]# rbd create testRBD1 --size 10 [root@mon ceph]# rbd info testRBD1 rbd image 'testRBD1': size 10240 kB in 3 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.101f74b0dc51 format: 2 features: layering flags: [root@mon ceph]# ceph --version ceph version 0.94.5-9.el7cp (deef183a81111fa5e128ec88c90a32c9587c615d) Version-Release number of selected component (if applicable): Recent
Comment 2 Chris Blum 2016-05-26 14:34:25 UTC
Same thing tested n Jewel Ceph: [root@mon vagrant]# rbd create testRBD1 --size 10 [root@mon vagrant]# rbd info testRBD1 rbd image 'testRBD1': size 10240 kB in 3 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.101f2ae8944a format: 2 features: layering flags: [root@mon vagrant]# ceph --version ceph version 10.2.0-1.el7cp (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
Comment 3 Nishanth Thomas 2016-06-21 14:28:18 UTC
Calamari provides no options to unset this hile pool is created. so moving to calamari
Comment 5 Nishanth Thomas 2016-06-21 14:31:47 UTC
ceph-installer creates the cluster in such a way that these features enabled by default
Comment 7 Christina Meno 2016-06-21 20:51:20 UTC
These are set using the key rbd_default_features in ceph.conf The list of features that don't work till RHEL7.3 in kRBD are: object map, deep flatten, journaling, exclusive lock, fast diff These work just fine in userspace RBD. My take on this is we should include docs that explain what action to take if you are a kRBD user. Chris what do you think about that?
Comment 8 Christina Meno 2016-06-22 15:54:50 UTC
kRBD users are few. We're not going to make code changes in 2.0 for this. Ken would you please add an entry in the 2.0 docs/release notes based on Comment 7?
Comment 9 Chris Blum 2016-06-22 21:34:17 UTC
I would try to design it in a way that it 'just works' for the regular user - meaning that I would deactivate the features unless there is a good reason to keep them active. Once 7.3 is released, we could bring the features back as a small update? This is all based on the assumption that these features are not essential for something important... If the features will stay activated, I would vote for a better help message that clearly tells the user that kRBD can't currently map this RBD (if that is somehow possible?)
Comment 13 Christina Meno 2016-07-13 16:10:13 UTC
This doc text should really mention that these features work fine with userspace RBD and this is only a limitation in krbd (kernel rbd)
Comment 17 Christina Meno 2017-03-02 00:19:31 UTC
Jason, Do we have an updated kRBD module shipped in RHEL7.3 that supports object map, deep flatten, journaling, exclusive lock, fast diff? i.e. can we close this BZ as works in current-release? Also in future is there something sane we can do with the value of rbd_default_features? cheers, G
Comment 18 Jason Dillaman 2017-03-02 02:22:34 UTC
The RHEL 7.3 krbd module supports layering and exclusive-lock, but still doesn't support object-map, fast-diff, deep-flatten, and journaling (not enabled by default). For the users of krbd, they can set "rbd default features = 1" in the ceph.conf file where they create images. Alternatively, a user can also run "rbd feature disable <image-spec> exclusive-lock,object-map,fast-diff,deep-flatten".
Comment 24 Jason Dillaman 2017-03-02 14:20:09 UTC
Comment 25 Christina Meno 2017-03-03 16:29:04 UTC
Bara, Looks good except one change I suggest: In that section it says "The ceph-installer utility enables the following features by default when creating a Ceph Block Device:" It's true that ceph-installer configures the cluster that way e.g. the defaults are "object-map, fast-diff, deep-flatten" It is misleading to say that ceph-installer creates the block device. For clarity I would say this instead "The default configuration of Ceph includes the following RBD features" cheers, G
Comment 26 Christina Meno 2017-03-03 16:35:33 UTC
Jason, What I'm hearing is that we'd need to take a lowest-common-denominator approach here and just set "rbd default features = 1" in the ceph.conf since ceph-installer(ansible) has no way of knowing what clients will be used or what RBDs will be created My concern is that my understanding is kRBD users are a minority when compared to user-space RBD. That means that this change would hurt the 80%(made up number) use case. What other options do we have here? Could the kRBD client warn that certain block devices have unsupported features? Can they be changed after the block-device is created? thanks, G
Comment 27 Jason Dillaman 2017-03-03 16:39:23 UTC
Greg, I believe the verbiage added by Bara in the doc answers your question re: can they be changed after it's created (yup). Since this config suggestion is only in the section related to krbd block devices, I would also think it would appear to be applicable to those users only.