Bug 2196272

Summary:	volumesnaphot creation taking 40s instead of 1s , with ceph-rbd , using csi plugin (OADP)
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Tzahi Ashkenazi <tashkena>
Component:	csi-driver	Assignee:	Rakshith <rar>
Status:	CLOSED NOTABUG	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	4.11	CC:	ocs-bugs, odf-bz-bot, whayutin
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-05-16 04:59:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tzahi Ashkenazi 2023-05-08 14:12:59 UTC

Description of problem (please be detailed as possible and provide log
snippests):

During OADP testing (backup & restore), while running backup flow with single name space with 100 pods , PVC size of 32MB on ceph-rbd.
using CSI plugin which create single snapshot per PVC.
100 busy box pods should take 8 1/2 mins, and when this issue happens it can take 4 hours
 
[root@f07-h27-000-r640 ~]# oc get volumesnapshotcontents -A  |wc -l
289
[root@f07-h27-000-r640 ~]# oc get volumesnapshot -A  |wc -l
90


original OADP ticket : 
   https://issues.redhat.com/browse/OADP-1719

Version of all relevant components (if applicable):
OCP 4.11.7
ODF 4.11.7
OADP 1.2.0-63

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
yes its a  blocker for out testing 

Is there any workaround available to the best of your knowledge?
no

Actual results:
snapshot creation of single snapshot takes more then 40 sec for 100 pods its 4H

Expected results:
snapshot creation of single snapshot takes around 4 sec for 100 pods its around 8 min 

Additional info:

must-gather output can be found here :

  https://drive.google.com/drive/folders/1qS82YIp1zraTFpvTsmeV_eHvnM9klQ2e?usp=sharing