Bug 1690086

(In reply to Scoots Hamilton from comment #0)
> Description of problem:
> 
> Current documentation for Ceph config overrides fails to be successfully
> implemented: 
> 
> Following instructions here: 
> 
> https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/12/
> html-single/deploying_an_overcloud_with_containerized_red_hat_ceph/
> #Configuring_Ceph_Storage_Cluster_Settings
> 
> Editing this file: /home/stack/templates/ceph-config.yaml
> 
> In the documented manner: 
> parameter_defaults:
>   CephConfigOverrides:
>     KEY: VALUE
> 
> Results in the override value to show up in the [global] section of the
> config file, allowing the default values to overwrite that values in the
> [osd] section.
> 
> 
> Version-Release number of selected component (if applicable):
> 12
> 
> How reproducible: 100%
> 
> Steps to Reproduce:
> - Edit the file mentioned in the docs: /home/stack/templates/ceph-config.yaml
> - Deploy Ceph 
> 
> Actual results:
> 
> EXAMPLE ceph-config.yml FILE: 
> 
> parameter_defaults:
>   CephConfigOverrides:
>     journal_size: 40960  <====== DESIRED CUSTOM VALUE 
>     max_open_files: 262144 <====== CUSTOM VALUE
>  
> 
> 2. Resultant ceph.conf 
> [global]
> ...
> journal_size = 40960 <========= THE CUSTOM VALUE 
> log file = /dev/null
> max_open_files = 262144 <======= CUSTOM VALUE
> mon cluster log file = /dev/null
> osd_pool_default_pg_num = 128
> osd_pool_default_pgp_num = 128
> ...
> 
> [osd]
> osd journal size = 5120 <== DEFAULT VALUE WHICH OVERRIDES VALUE IN 
> osd mkfs options xfs = -f -i size=2048
> osd mkfs type = xfs
> osd mount options xfs = noatime,largeio,inode64,swalloc
> 
> Expected results:
> There is an expectation that following the documented steps will allow the
> user to specify a custom value in for their Ceph environment and that the
> value will be populated in correct sections to ensure implementation. 
> 
> Any value in the [global] section is second to a values in a service
> component section ([osd], [mon], [rgw], etc.)

I think having the journal_size in this example documentation is misleading and I agree with you that it should be removed. 
The documentation does state:

"""
You can arbitrarily apply global Ceph cluster settings using the following syntax in the parameter_defaults section of your environment file: 

parameter_defaults:
  CephConfigOverrides:
    KEY: VALUE
"""

This is true, but the docs should emphasize that this ONLY applies to the [global] section and should only be used for variables in the global section. Perhaps it should have a warning that variables which would be set in the [osd] section shouldn't be set using this method. It should follow up to say that ceph-ansible has variables to modify other sections, e.g. the [osd] section, which you can override with the CephAnsibleExtraConfig and provide an example of that. I'll give an example below.
 
> Additional info:
> 
> Currently to work around this we modified the
> /usr/share/ceph-ansible/group_vars/all.yml and removed the default journal
> size value with the custom one: 

I see. Going forward, please never modify any values in /usr/share/ceph-ansible/group_vars/all.yml directly. Instead, any ceph-ansible variable that you would write in there can be overridden in a Heat environment file when using Director like this:

"""
parameter_defaults:
  CephAnsibleExtraConfig:
    journal_size: 40960
"""

The documentation didn't state that but it should. I'll request the same update as per this docbug. Thanks for pointing this out. 

> EX: journal_size: 40960 # OSD journal size in MB
> 
> This resulted in the desired outcome: 
> 
> [osd]
> osd journal size = 40960

*** Bug 1547534 has been marked as a duplicate of this bug. ***

Updated and merged for:

Osp13-queens: 
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/deploying_an_overcloud_with_containerized_red_hat_ceph/index#Configuring_Ceph_Storage_Cluster_Settings

Osp14-rocky:
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/deploying_an_overcloud_with_containerized_red_hat_ceph/index#Configuring_Ceph_Storage_Cluster_Settings

and master.

Note that from OSP16 it will be possible to provide custom key/values to use in any section of the Ceph config file [1], the related doc bug is [2]

1. https://bugzilla.redhat.com/show_bug.cgi?id=1666973
2. https://bugzilla.redhat.com/show_bug.cgi?id=1780757