Bug 2119217

Summary:	CPU and latency spikes during periodic snaptrim operations
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Josh Durgin <jdurgin>
Component:	RADOS	Assignee:	Adam Kupczyk <akupczyk>
Status:	CLOSED ERRATA	QA Contact:	Pawan <pdhiran>
Severity:	urgent	Docs Contact:	Akash Raj <akraj>
Priority:	urgent
Version:	6.0	CC:	adking, akraj, akupczyk, amagrawa, amathuri, bhubbard, ceph-eng-bugs, cephqe-warriors, choffman, ekuric, idryomov, kdreyer, kmanohar, kseeger, ksirivad, lflores, ngangadh, nojha, pdhange, pdhiran, rfriedma, rzarzyns, skanta, srangana, sseshasa, tnielsen, torkil, tpetr, tserlin, vdas, vereddy, vumrao
Target Milestone:	---	Keywords:	TestBlocker
Target Release:	6.1z1
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ceph-17.2.6-94.el9cp	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2218013 (view as bug list)		Environment:
Last Closed:	2023-08-03 16:45:09 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2077047, 2120239, 2150996, 2154351, 2169499, 2190382, 2218013, 2221020

Description Josh Durgin 2022-08-17 22:40:45 UTC

While testing snapshot based rbd-mirror with a random rw workload, Paul Cuzner noticed that at the start of every replication interval the ceph-osd cpu consumption spikes dramatically, and continues to grow over time. For example, at the beginning of a run the cpu spike was 60% of a core...but after 24 hours with the same randrw workload running this grows to 1.5-2 cores.

The cpu overhead appears worse for 4KB block sizes, than for IO sizes of 16KB or more.

The change rate within the snapshot is only 250MB every 5 mins - The workload is just 20 rbd images, using rate limited fio which caps each rbd image to 50 IOPS (40 read + 10 write)

If the host is not capping the OSD this issue, will likely go unnoticed but in environments like k8s where the OSD is capped this is more of a problem.

This translates to high latency for clients during these spikes, and with the growth over time, it means continual performance degradation.

Comment 44 Malay Kumar parida 2023-04-05 05:54:37 UTC

> Malay, during an upgrade, the OCS operator will update that configmap with any new settings, correct? 

Yes Travis, The changes will be picked up after the upgrade. So if some particular config is decided here,
we can put that in that configmap. But keep in mind ocs-operator will always reconcile the configmap and will always apply those 
configs. So if someone wants to remove those configs for any reason that will be impossible. So we have to be a little careful.

Comment 82 errata-xmlrpc 2023-08-03 16:45:09 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat Ceph Storage 6.1 Bug Fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:4473

Comment 83 Red Hat Bugzilla 2023-12-02 04:25:59 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days