Bug 1340080

Summary: ceph-installer creates RBDs with currently unmappable features activated
Product: Red Hat Ceph Storage Reporter: Chris Blum <cblum>
Component: Ceph-InstallerAssignee: Christina Meno <gmeno>
Status: CLOSED WONTFIX QA Contact: ceph-qe-bugs <ceph-qe-bugs>
Severity: unspecified Docs Contact: Bara Ancincova <bancinco>
Priority: unspecified    
Version: 3.0CC: adeza, aschoen, bancinco, ceph-eng-bugs, flucifre, gmeno, jdillama, khartsoe, mhackett, nthomas, sankarshan
Target Milestone: rc   
Target Release: 3.*   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Known Issue
Doc Text:
.Certain image features are not supported with the RBD kernel module The following image features are not supported with the current version of the RADOS Block Device (RBD) kernel module (`krbd`) that is included in Red Hat Enterprise Linux 7.4: * `object-map` * `deep-flatten` * `journaling` * `fast-diff` However, by default the `ceph-installer` utility creates RBDs with the aforementioned features enabled. As a consequence, an attempt to map the kernel RBDs by running the `rbd map` command fails. To work around this issue, disable the unsupported features by setting the `rbd_default_features = 1` option in the Ceph configuration file for kernel RBDs or dynamically disable them by running the following command: ---- rbd feature disable <image> <feature> ---- This issue is a limitation only in kernel RBDs, and the features work as expected with user-space RBDs.
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-15 21:59:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1322504, 1383917, 1412948, 1494421    

Description Chris Blum 2016-05-26 12:20:05 UTC
Description of problem:
Currently the RBDs created with the RHS-C look like this:
[root@rhcs1 ceph]# rbd info testRBD
rbd image 'testRBD':
    size 10240 MB in 2560 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.105b74b0dc51
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten   <----

Because of https://access.redhat.com/solutions/2258301 this will prevent rbds to be mapped with these features activated until the GA of RHEL7.3 (in Autumn)

Suggestion: Create new RBDs without those features to make them mappable in RHEL7.2.

In a non-RHS-C setup the default RBD looks like this:
[root@mon ceph]# rbd create testRBD1 --size 10
[root@mon ceph]# rbd info testRBD1
rbd image 'testRBD1':
	size 10240 kB in 3 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.101f74b0dc51
	format: 2
	features: layering
[root@mon ceph]# ceph --version
ceph version 0.94.5-9.el7cp (deef183a81111fa5e128ec88c90a32c9587c615d)

Version-Release number of selected component (if applicable):

Comment 2 Chris Blum 2016-05-26 14:34:25 UTC
Same thing tested n Jewel Ceph:

[root@mon vagrant]# rbd create testRBD1 --size 10
[root@mon vagrant]# rbd info testRBD1
rbd image 'testRBD1':
    size 10240 kB in 3 objects
    order 22 (4096 kB objects)
    block_name_prefix: rbd_data.101f2ae8944a
    format: 2
    features: layering
[root@mon vagrant]# ceph --version
ceph version 10.2.0-1.el7cp (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)

Comment 3 Nishanth Thomas 2016-06-21 14:28:18 UTC
Calamari provides no options to unset this hile pool is created. so moving to calamari

Comment 5 Nishanth Thomas 2016-06-21 14:31:47 UTC
ceph-installer creates the cluster in such a way that these features enabled by default

Comment 7 Christina Meno 2016-06-21 20:51:20 UTC
These are set using the key rbd_default_features in ceph.conf

The list of features that don't work till RHEL7.3 in kRBD are:
object map, deep flatten, journaling, exclusive lock, fast diff

These work just fine in userspace RBD. My take on this is we should include docs that explain what action to take if you are a kRBD user.

Chris what do you think about that?

Comment 8 Christina Meno 2016-06-22 15:54:50 UTC
kRBD users are few. We're not going to make code changes in 2.0 for this.

Ken would you please add an entry in the 2.0 docs/release notes based on Comment 7?

Comment 9 Chris Blum 2016-06-22 21:34:17 UTC
I would try to design it in a way that it 'just works' for the regular user - meaning that I would deactivate the features unless there is a good reason to keep them active. Once 7.3 is released, we could bring the features back as a small update?
This is all based on the assumption that these features are not essential for something important...
If the features will stay activated, I would vote for a better help message that clearly tells the user that kRBD can't currently map this RBD (if that is somehow possible?)

Comment 13 Christina Meno 2016-07-13 16:10:13 UTC
This doc text should really mention that these features work fine with userspace RBD and this is only a limitation in krbd (kernel rbd)

Comment 17 Christina Meno 2017-03-02 00:19:31 UTC

Do we have an updated kRBD module shipped in RHEL7.3 that supports object map, deep flatten, journaling, exclusive lock, fast diff?

i.e. can we close this BZ as works in current-release?

Also in future is there something sane we can do with the value of rbd_default_features?


Comment 18 Jason Dillaman 2017-03-02 02:22:34 UTC
The RHEL 7.3 krbd module supports layering and exclusive-lock, but still doesn't support object-map, fast-diff, deep-flatten, and journaling (not enabled by default).

For the users of krbd, they can set "rbd default features = 1" in the ceph.conf file where they create images. Alternatively, a user can also run "rbd feature disable <image-spec> exclusive-lock,object-map,fast-diff,deep-flatten".

Comment 24 Jason Dillaman 2017-03-02 14:20:09 UTC
@Bara: lgtm

Comment 25 Christina Meno 2017-03-03 16:29:04 UTC

Looks good except one change I suggest: In that section it says "The ceph-installer utility enables the following features by default when creating a Ceph Block Device:"

It's true that ceph-installer configures the cluster that way e.g. the defaults  are "object-map, fast-diff, deep-flatten"

It is misleading to say that ceph-installer creates the block device.

For clarity I would say this instead "The default configuration of Ceph includes the following RBD features"


Comment 26 Christina Meno 2017-03-03 16:35:33 UTC

What I'm hearing is that we'd need to take a lowest-common-denominator approach here and just set "rbd default features = 1" in the ceph.conf since ceph-installer(ansible) has no way of knowing what clients will be used or what RBDs will be created

My concern is that my understanding is kRBD users are a minority when compared to user-space RBD. That means that this change would hurt the 80%(made up number) use case.

What other options do we have here? Could the kRBD client warn that certain block devices have unsupported features? Can they be changed after the block-device is created?


Comment 27 Jason Dillaman 2017-03-03 16:39:23 UTC
Greg, I believe the verbiage added by Bara in the doc answers your question re: can they be changed after it's created (yup). Since this config suggestion is only in the section related to krbd block devices, I would also think it would appear to be applicable to those users only.