DescriptionKritik Sachdeva
2022-09-23 16:55:27 UTC
Description of problem:
In RHCS 5.x there is no blueFS spillover health warning generated when the RocksDB starts consuming block (slower) device space.
Version-Release number of selected component (if applicable): RHCS 5.0z4 and RHCS 5.2
How reproducible: Always
Steps to Reproduce:
1. Deploy a fresh RHCS 5 or upgrade a cluster from RHCS 4 to RHCS 5 with smaller block DB size (Like 10 Mib or 30 Mib)
- For example:
~~~
service_type: osd
service_id: osd_nodeXY_paths
service_name: osd.osd_nodeXY_paths
placement:
hosts:
- nodeX
- nodeY
spec:
block_db_size: 10485760 <----
data_devices:
paths:
- /dev/sdb
- /dev/sdc
db_devices:
paths:
- /dev/sdd
filter_logic: AND
objectstore: bluestore
~~~
2. Add some data into the cluster using RBD
3. Collect the output of the below command and look for the "slow_used_bytes" parameter.
~~~
$ ceph daemon osd.<id> perf dump bluefs
~~~
- If using non-colocated OSDs, then also verify using the below command and look for "SLOW" Column
~~~
$ ceph daemon osd.<id> bluefs stats
~~~
*NOTE*: non-colocated: OSDs having DB and Data on separate devices
Actual results: No bluefs spillover health warning
Expected results: It should show the bluefs spillover health warning
Additional info:
Tried to reproduce this issue in RHCS 4.2z4 and successfully able to get the bluefs spillover health warning
Description of problem: In RHCS 5.x there is no blueFS spillover health warning generated when the RocksDB starts consuming block (slower) device space. Version-Release number of selected component (if applicable): RHCS 5.0z4 and RHCS 5.2 How reproducible: Always Steps to Reproduce: 1. Deploy a fresh RHCS 5 or upgrade a cluster from RHCS 4 to RHCS 5 with smaller block DB size (Like 10 Mib or 30 Mib) - For example: ~~~ service_type: osd service_id: osd_nodeXY_paths service_name: osd.osd_nodeXY_paths placement: hosts: - nodeX - nodeY spec: block_db_size: 10485760 <---- data_devices: paths: - /dev/sdb - /dev/sdc db_devices: paths: - /dev/sdd filter_logic: AND objectstore: bluestore ~~~ 2. Add some data into the cluster using RBD 3. Collect the output of the below command and look for the "slow_used_bytes" parameter. ~~~ $ ceph daemon osd.<id> perf dump bluefs ~~~ - If using non-colocated OSDs, then also verify using the below command and look for "SLOW" Column ~~~ $ ceph daemon osd.<id> bluefs stats ~~~ *NOTE*: non-colocated: OSDs having DB and Data on separate devices Actual results: No bluefs spillover health warning Expected results: It should show the bluefs spillover health warning Additional info: Tried to reproduce this issue in RHCS 4.2z4 and successfully able to get the bluefs spillover health warning