Bug 1583839

Description of problem:

This bug is sort of a tracker bug that may be linked to other more specific problems in Ceph (such as ceph-volume or ceph-ansible or Ceph trackers).  it also requires performance work.  

RHCS 3.2 will be the release that supports Bluestore for all customers, if I understand PM's intent.  It should be *easy* to install in the sense that a minimal and comprehensible set of tuning knobs should be exposed to the sysadmin.  It must have reasonable defaults.  If we fail, it can be almost impossible to correct some of the problems, such as the RocksDB partition size, at a later date in a production environment (i.e. by rebuilding OSDs).   Other problems, such as poor defaults, will cause the customer to conclude that Bluestore performance is inferior to, or no better than, Filestore, although clearly Bluestore represents a major performance improvement.

Here are several areas where Bluestore may require tuning today:

- RocksDB partition size
- reducing write amplification through RocksDB tuning (pg log for example)
- reducing tail latency (increasing fairness) to multiple clients
- memory allocation among competing Bluestore OSD processes

We need this to be much easier for the average system administrator.  Specifically:

- The Ceph installer s/w could accept an user input for average RADOS object size on an OSD or subset of OSDs.   It could then calculate how big to make the RocksDB partition based on knowledge of device sizes, assuming each RADOS object requires roughly the same amount of metadata per object (I'm trying to measure that now).  The default might be based on assumption of 4-MB object size (i.e. RBD, Cephfs with large files, or RGW with large objects).

- The Ceph installer s/w could calculate how much RAM to assign to each OSD. 
  The user input here could be the fraction of physical RAM to assign to OSD caching.  The installer would then divide this by the number of OSDs, assuming all OSDs are the same size (we can also weight OSD RAM allocation by size of device.   A default might be 3/4  physical memory on a large server (hyperconvergence of apps and OSDs could require lowering this).  This default allows for co-resident monitors, MDS, etc.  By specifying a fraction, we help to avoid situation where the user can oversubscribe memory.  

- if necessary a web-app calculator or spreadsheet similar to the PG calculator can be used to help people derive the correct values for complicated configurations.

- Testing and analysis is needed to show how much of Ryan Meredith's (and Mark Nelson's) Micron tunings are necessary and, if they are, can be made default behavior.  For example, should these tunings be Luminous defaults, and if not, why not?


How reproducible:

John Harrigan's work in the Red Hat scale lab shows effect of RocksDB spillover to HDD on increasing RGW latency over time.  This in part was coupled with slowness of RGW garbage collection, but still could happen whenever you fill up the OSD.

https://docs.google.com/presentation/d/1wwtYf9ymHwd8B1Utjn2JsqXRCTlAdeTm8VDoxdIWYxQ/edit#slide=id.g35e2252ad7_0_94

(available to non-Red-Hat upon request)

Micron had initial difficulties with tuning Bluestore to outperform Filestore, and Micron is an experienced Ceph user.  In the end they got great results, particularly for lower tail latency.

https://www.openstack.org/summit/vancouver-2018/summit-schedule/events/21803/unleashing-the-power-of-flash-for-ceph-data-stores-an-all-nvme-ceph-performance-deep-dive

https://www.micron.com/about/blogs/2018/may/ceph-bluestore-vs-filestoreblock-performance-comparison-when-leveraging-micron-nvme-ssds


Steps to Reproduce:

1.  Install Ceph with Bluestore using the documented RHCS installer, without tuning.
2.  Compare performance to filestore using RBD or RGW, for an aged, fully-populated workload.


Actual results:

Bluestore can be slower than filestore when default caching per OSD (1 GB) and default RocksDB partition size of 1 GB are used.  


Expected results:

Bluestore should never be worse, and should be significantly better than Filestore, as shown by Micron (Ryan Meredith) article.  Mark Nelson is the authority on Bluestore tuning, and has published a great deal of performance data to the Ceph community.  

Alex Calhoun measured big speedups for Bluestore on HDD when write size was multiple of bluestore_min_alloc_size (can avoid double-write penalty), and Ben England measured 4-5x speedup on RADOS object create speed for small 4-KB objects.  See:

https://mojo.redhat.com/groups/product-performance-scale-community-of-practice/content?query=Bluestore


Additional info:

  https://ceph.com/community/new-luminous-bluestore/
  http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/

There has been some difficulty in measuring metadata per-object cost:

   http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026333.html

Li Xiaoyan from Intel discusses Bluestore RocksDB I/O in:
  https://www.youtube.com/watch?v=jKdNFaZHrf0

In her talk, she mentions that about half of the writes to Bluestore seem to be "pg log" writes, some of which may be transitory states and therefore maybe shouldn't be written to RocksDB persistent storage.