While testing snapshot based rbd-mirror with a random rw workload, Paul Cuzner noticed that at the start of every replication interval the ceph-osd cpu consumption spikes dramatically, and continues to grow over time. For example, at the beginning of a run the cpu spike was 60% of a core...but after 24 hours with the same randrw workload running this grows to 1.5-2 cores. The cpu overhead appears worse for 4KB block sizes, than for IO sizes of 16KB or more. The change rate within the snapshot is only 250MB every 5 mins - The workload is just 20 rbd images, using rate limited fio which caps each rbd image to 50 IOPS (40 read + 10 write) If the host is not capping the OSD this issue, will likely go unnoticed but in environments like k8s where the OSD is capped this is more of a problem. This translates to high latency for clients during these spikes, and with the growth over time, it means continual performance degradation.
> Malay, during an upgrade, the OCS operator will update that configmap with any new settings, correct? Yes Travis, The changes will be picked up after the upgrade. So if some particular config is decided here, we can put that in that configmap. But keep in mind ocs-operator will always reconcile the configmap and will always apply those configs. So if someone wants to remove those configs for any reason that will be impossible. So we have to be a little careful.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat Ceph Storage 6.1 Bug Fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:4473
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days