Bug 1817228

Created attachment 1673619 [details]
dittybopper graph of block device throughput

Description of problem 

For baremetal install, OCS does not distinguish between NVMe devices and HDD devices.   Consequently it treats NVM as just another OSD and puts NVM OSDs in same storage pool as HDD OSDs, even though they have different sizes.   Normally, since NVM SSDs are smaller (in TB) than HDDs, each NVM would thus do less I/O work (IOPS, MB/s) than each HDD, although each NVM can do at least 2 orders of magnitude more IOPS than an HDD.

This is a regression compared to RHCS, in which ceph-ansible installer is able to utilize NVM devices intelligently (more below).

Why do we care?   HDDs are not needed for low-density-high-IOPS configurations, but are still needed for high-density-low-IOPS configurations.  All the performance data showing the conclusions below has already been collected for RHCS years ago and nothing about it has changed.


Version:

OpenShift (OCP) 4.3.3
quay.io/rhceph-dev/ocs-olm-operator:4.3.0-rc2
Ceph nautilus 14.2.4-125 
quay.io/rhceph-dev/rhceph@sha256:1ec55227084f058c468df5cfff2cd55623668a72ec742af3e8b1c05b52d44d0a


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

Yes, from a performance perspective.


Is there any workaround available to the best of your knowledge?

No.   I can most likely go into toolbox ceph command and hack on the cluster to make it work better, but that is not how OCS installer is supposed to work.


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

1 - vanilla install on commodity server hardware


Can this issue reproducible?

Every time


Can this issue reproduce from the UI?

never tried it.


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Run baremetal OCS install (described below)
2. run ripsaw fio workload with default CR, modifying storage class to be one provided by OCS
3. monitor cluster block device throughput with cloud-bulldozer/dittybopper (attachment included)

detailed install method at bottom.


Actual results:

/dev/nvme0n1 was not doing nearly as much throughput as it was capable of.  Consequently cluster performance was well below what it could have been.


Expected results:

/dev/nvme0n1 is used intelligently in 2 ways:

a) as a metadata device and journal device for the HDD OSDs, avoiding double-write penalty and metadata-access seek time.

To do this correctly, you have to allocate enough storage space on the NVM partitions to handle the metadata for the HDD.   This can be done if you have an average object size.  

- first allocate a 1-GiB WAL (write-ahead-log) partition on an NVM device for the HDD OSD
- next allocate the RocksDB partition for that OSD.  By default, you take a percentage (Ceph docs say 4%) of the HDD size and allocate an NVM partition for RocksDB of that size.  But if a user has unusually small objects/files, or uses erasure coding extensively, additional space may be needed - we should ask them what their average object/file size is and what kind of replication (3-way or ECk+m) and calculate it from that.


b) as a separate all-flash storage pool that can achieve an order of magnitude more random or small file/object IOPS than a collection of HDDs can.


One can make OCS do b) all-flash pool post-install by retargeting the pool at the "ssd" device class, this can be done without OCS being aware of that.  But you could argue that this should already be done by default for cephfs metadata pool and for certain RGW pools that take very little space but require high IOPS.   It also would cost nothing to create certain pools targeted at SSD and HDD.   This is an opinionated installer!

Suggestions for mixed HDD+SSD baremetal configurations:
RBD pool - have 2 pools, 1 on SSD, and another on HDD
cephfs data pool - have 2 pools, 1 on SSD and another on HDD
cephfs metadata pool - 1 pool on SSD
RGW - have 2 data pools, 1 on SSD and another on HDD.
RGW metadata pools should be on SSD

By default, Cephfs storage class should land your data on HDD and your metadata on SSD.
By default, RGW should land your data on HDD and all metadata on SSD

Of course, if there are only SSD devices available, then you don't have to create both kinds of pools.  And if there are only HDD devices (not recommended) then all pools live there, obviously.


Additional info:

Ceph understands which devices are NVM or HDD.  

sh-4.2$ ceph df
RAW STORAGE:
    CLASS     SIZE       AVAIL      USED        RAW USED     %RAW USED 
    hdd       83 TiB     83 TiB     128 GiB      179 GiB          0.21 
    ssd       13 TiB     13 TiB      21 GiB       26 GiB          0.19 
    TOTAL     97 TiB     97 TiB     149 GiB      205 GiB          0.21 
 
And it's easy for Ceph to create a pool that either lives entirely on SSD or lives entirely on HDD.  This ancient article documents how to use Ceph device classes to create such storage pools.

https://ceph.io/community/new-luminous-crush-device-classes/

Here are examples of pools for RBD and Cephfs:

# ocos rsh $(ocos get pod | awk '/tools/{print $1}') ceph osd pool ls detail
...
pool 1 'example-storagecluster-cephblockpool' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 190 lfor 0/0/179 flags hashpspool,selfmanaged_snaps stripe_width 0 target_size_ratio 0.49 application rbd
	removed_snaps [1~3]
...
pool 4 'example-storagecluster-cephfilesystem-metadata' replicated size 3 min_size 2 crush_rule 3 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode on last_change 160 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
...
pool 6 'example-storagecluster-cephfilesystem-data0' replicated size 3 min_size 2 crush_rule 5 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 190 lfor 0/0/188 flags hashpspool stripe_width 0 target_size_ratio 0.49 application cephfs

Each pool has its own corresponding CRUSH rule, but if you look at the crush rules, they are all the same.  For example, pool 6 the cephfs data pool uses crush rule 5, but they all look the same and all blindly select OSDs based on rack and not on device type.


Installation methods:

I used this repo to install OCP4 on baremetal Alias lab cluster:

https://github.com/bengland2/ocp4_upi_baremetal

I then used Alex Calhoun's document to learn how to install OCS on baremetal, it comes down to this script, which uses various files in the same URL directory:

http://perf1.perf.lab.eng.bos.redhat.com/pub/bengland/public/openshift/upi/ocs-bringup/all-of-it.sh

Once OCS was up and running, I use benchmarks from this URL to test:

https://github.com/cloud-bulldozer/ripsaw

this particular problem can be observed with the fio benchmark, the attached graph was from an fio run.