Bug 2044114

Summary:	AWS - degradation in RBD clone creation times in ODF 4.10 vs ODF 4.9 for 1GB and 25GB sized clones
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Yuli Persky <ypersky>
Component:	csi-driver	Assignee:	Rakshith <rar>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Elad <ebenahar>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.10	CC:	alayani, jopinto, kramdoss, madam, mmuench, muagarwa, ocs-bugs, odf-bz-bot, rar
Target Milestone:	---	Keywords:	Automation, Performance
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-07 02:40:21 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Yuli Persky 2022-01-24 00:52:26 UTC

Description of problem:

AWS platform - There is a degradation in RBD clone creation times in  ODF 4.10 vs ODF 4.9 fpr 1GB and 25GB sized clones 

1GB RBD Clone creation times are: 

ODF 4.9 : 1.63 sec
ODF 4.10: 2.09 sec
 
25GB RBD Clone creation times are: 

ODF 4.9 : 1.75
ODF 4.10: 1.9

Note that there is no degradation in RBD clone creation times for 50 and 100 GB clones. 

Version-Release number of selected component (if applicable):

ODF 4.10.0.50

Note : you may find additional details in the following Jenkins job : https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/view/Performance/job/qe-trigger-aws-ipi-3az-rhcos-3m-3w-performance/58/



How reproducible:

Steps to Reproduce:
1. Run tests/e2e/performance/csi_tests/test_pvc_clone_performance.py test
2. Compare its results ( average reaattach time of 10 samples) to 4.9 results ( available in this report: https://docs.google.com/document/d/1vyufd55iDyvKeYOwoXwKSsNoRK2VR41QNTuH-iERR8s/edit ) 
3.


Actual results:

Clone creation times for 1GB and 25GB sized clones are longer in ODF 4.10 vs ODF 4.9

Expected results:

Clone creation time should be same or shorter in ODF 4.10

Additional info:

Relevant Jenkins job:

https://ocs4-jenkins-csb-odf-qe.apps.ocp-c1.prod.psi.redhat.com/view/Performance/job/qe-trigger-aws-ipi-3az-rhcos-3m-3w-performance/58/

Comparison report: 

https://docs.google.com/document/d/1OJfARHBAJs6bkYqri_HpSNM_N5gchUQ6P-lKe6ujQ6o/edit#


Must gather logs are available here: 

http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-058ai3c33-p/j-058ai3c33-p_20220105T230912/logs/testcases_1641427511/

Comment 3 Yuli Persky 2022-01-25 09:10:32 UTC

@Rakshith,

I'll 5 samples creation and average calculation for the test and will update on the results.

Comment 4 Yuli Persky 2022-02-02 10:31:25 UTC

PLease note that must gather logs are available here: 


http://magna002.ceph.redhat.com/ocsci-jenkins/openshift-clusters/j-058ai3c33-p/j-058ai3c33-p_20220105T230912/logs/testcases_1641427511/

As for the samples - the test is under modification,will post here the results after it's merged.

Comment 5 Yuli Persky 2022-03-06 22:33:29 UTC

@Rakshith,

Just want to update that the test modification is still not done due to other tasks priorities. However, this is going to be my next task since this is a very much needed fix. 
I think that this BZ can be closed , I did see number of executions of the clone performance test ( on AWS and also on VMware platforms) with different results reported. 
Therefore we'll rerun this test when the fix is ready and will report a new BZ if the AVERAGE snapshot creation time will still look bad.