Bug 1438590

Description of problem:

At large scale, it is difficult, tedious, repetitive and sometimes impossible to specify Ceph OSDs by device name, and to specify for each OSD a SSD journal (bz 1438572 must be fixed first!).  Installation issues can prevent scalability just as much as other implementation issues.  This is difficult because:

- device names are not guaranteed to be stable across reboots
- at large scale, the number and type of devices may differ due to purchasing hardware at different times
- at large scale, there is a high probability that one or more HDDs may have failed and not been replaced, and we need the deployment to succeed anyway

These causes are discussed below in "Additional Info" section.  Use of OpenStack directory for HCI (hyperconverged) deployment has made this problem more visible, because there is no option to use an external Ceph cluster (deployed via ceph-ansible) in this configuration.

possible form of solution:

To support scalability of OpenStack-on-Ceph configurations, including HCI (hyperconverged) storage, OpenStack needs to support discovery of Ceph OSDs by a rule, using either a device naming pattern, including /dev/disk/by-path/ names, drive size, or other attribute.  

The discovery process should convert the rule into a set of /dev/disk/by-path names and attributes for each node, and this should be logged by puppet-ceph as the deployment is performed.  It is implied in this syntax that whatever drive is used for the operating system or for a Ceph SSD journal would not be used as a Ceph OSD.

This benefits the installation by getting rid of most of the ceph YAML file and potential sources of user error.  More importantly, if there are nodes that deviate either in the number or type of storage hardware from the norm, these nodes can still be deployed.  The bigger the deployment, the more installation complexity is removed by this feature.

Here's a thumbnail of how this could work.  Instead of specifying "ceph::profile::params::osds:" in the ceph YAML file, we could specify:

ceph::profile::params::osd_selection_rule:
  size: '4 TB'

or a regular expression for device name, such as:

ceph::profile::params::osd_selection_rule:
  name: '/dev/sd*[a-z]'   

or, if necessary (see additional info below), 

ceph::profile::params::osd_selection_rule:
  name: '/dev/disk/by-path/pci-0000:03:00.0-sas-0x1221000000000000-lun-*'

Version-Release number of selected component (if applicable):

RHOSP 10 (Newton)

How reproducible:

The documentation below describes the current situation:

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html-single/red_hat_ceph_storage_for_the_overcloud/#Mapping_the_Ceph_Storage_Node_Disk_Layout

Actual results:

In the scale lab using hardware from an OpenStack partner, SuperMicro ( https://www.supermicro.com/solutions/Cloud.cfm ), OpenStack repeatedly failed to deploy with only 8 cephstorage nodes, let alone what would happen with 30 nodes (which we intend to test).  We worked around problem by deploying using /dev/disk/by-path names, but we had to cut our drive count from 36 to 34 (because of failed drives in some of the hosts)to get it to deploy, wasting over 10 drives.

Expected results:

RHOSP OpenStack+Ceph deploys every time, successfully and easily, at scale, using all of the hardware made available to it.

Additional info:

We should look at ceph-ansible support for a discovery process like this, since there is talk of getting support for ceph-ansible into OpenStack.

device-name stability: OpenStack, unlike ceph-ansible, reboots nodes as part of normal deployment process.  Device names are not guaranteed by Linux to be stable across reboots, and they are not stable across reboots when there are two storage controllers, as we can demonstrate with SuperMicro 6048Rs in the scale lab).  This can be worked around using /dev/disk/by-path specifications and the bug fix done by Joe Talerico et al in:

https://review.openstack.org/#/c/451826

non-identical hardware:  If an OpenStack site is not purchased in its entirety at the exact same time with the exact same hardware specifications, then it may have different types of storage controllers in different slots and/or different number and types of drives.  We don't want to place unreasonable restrictions on the homogeneity of cephstorage nodes within an OpenStack cluster.

disk failure probability:  When you get to a scale where you have on the order of 1000 HDDs in a configuration, a few of them are going to fail - this is normal behavior given the drive MTBF etc for a cluster with so many drives in it.  We don't want the OpenStack deploy for the entire node to fail because of this.