Bug 2127724

Summary:	Regression degradation in dv cloning duration
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	guy chen <guchen>
Component:	csi-driver	Assignee:	Madhu Rajanna <mrajanna>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	krishnaram Karthick <kramdoss>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.11	CC:	bniver, jhopper, mmuench, mrajanna, muagarwa, ocs-bugs, odf-bz-bot, sbennert, sostapov, tmuthami, ycui
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-04-17 10:33:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description guy chen 2022-09-18 16:00:53 UTC

Description of problem (please be detailed as possible and provide log
snippests):

A degradation in cloning duration of 10 fedora DVs with copy cloneStrategy - from ~5 minutes at ODF 4.10.5 to ~9 minutes at ODF 4.11.1.
At larger numbers it is slower, for 100 DVs on ODF 4.11 it got up to 33 minutes.

Version of all relevant components (if applicable):
ODF 4.11.1

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, cloning got much slower.

Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
1

Can this issue reproducible?
Yes

Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
Yes

Steps to Reproduce:
1. Build OCP 4.11.0-rc.6 with 3 masters and 3 workers
2. Install CNV 4.11
3. Install local storage
4. Install ODF 4.11.1
5. Change cloneStrategy to Copy
6. Create 10 fedora VMS with a template 


Actual results:
DVS Cloning take ~9 minutes

Expected results:
Should take ~5 minutes, as in 4.10.5

Additional info: will be added

Comment 10 Madhu Rajanna 2022-10-04 07:57:40 UTC

At initial glance From the rbdplugin logs i can see below

cat current.log |grep err
2022-09-28T12:24:45.988776907Z E0928 12:24:45.988761       1 utils.go:200] ID: 13048 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:24:46.991721772Z E0928 12:24:46.991698       1 utils.go:200] ID: 13049 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:24:48.994549940Z E0928 12:24:48.994541       1 utils.go:200] ID: 13050 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:24:52.997178786Z E0928 12:24:52.997164       1 utils.go:200] ID: 13052 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:25:01.000189450Z E0928 12:25:01.000181       1 utils.go:200] ID: 13053 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:25:17.002706943Z E0928 12:25:17.002695       1 utils.go:200] ID: 13054 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:25:49.005530784Z E0928 12:25:49.005491       1 utils.go:200] ID: 13055 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:26:53.008542197Z E0928 12:26:53.008515       1 utils.go:200] ID: 13058 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:29:01.011394246Z E0928 12:29:01.011377       1 utils.go:200] ID: 13061 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:29:53.962274399Z E0928 12:29:53.962259       1 utils.go:200] ID: 13063 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:33:17.014674154Z E0928 12:33:17.014340       1 utils.go:200] ID: 13067 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists
2022-09-28T12:38:17.017602028Z E0928 12:38:17.017590       1 utils.go:200] ID: 13073 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists


ReqID 0001-0011-openshift-storage-0000000000000002-408eeba9-3819-11ed-a950-0a580a80041d took 0:00:00.039935 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-b31f4b59-3819-11ed-a950-0a580a80041d took 0:00:00.032579 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f373b86e-3819-11ed-a950-0a580a80041d took 0:00:00.143195 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f706e231-3819-11ed-a950-0a580a80041d took 0:00:00.149968 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f743e7a4-3819-11ed-a950-0a580a80041d took 0:00:00.147614 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f8957c50-3819-11ed-a950-0a580a80041d took 0:00:00.161886 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f94b8677-3819-11ed-a950-0a580a80041d took 0:00:00.156478 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f98784d7-3819-11ed-a950-0a580a80041d took 0:00:00.145785 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-fa20ca10-3819-11ed-a950-0a580a80041d took 0:00:00.149611 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-fad70096-3819-11ed-a950-0a580a80041d took 0:00:00.145393 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-fb5179bb-3819-11ed-a950-0a580a80041d took 0:00:00.146231 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-fc26966c-3819-11ed-a950-0a580a80041d took 0:00:00.155550 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-86a8b42c-381a-11ed-a950-0a580a80041d took 0:00:00.026904 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-86ee16a0-381a-11ed-a950-0a580a80041d took 0:00:00.024317 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-87733c2e-381a-11ed-a950-0a580a80041d took 0:00:00.031198 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-87f53bcd-381a-11ed-a950-0a580a80041d took 0:00:00.028428 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-887e5672-381a-11ed-a950-0a580a80041d took 0:00:00.030102 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-896325cf-381a-11ed-a950-0a580a80041d took 0:00:00.029635 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-89a04c86-381a-11ed-a950-0a580a80041d took 0:00:00.028530 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-89f2e4b6-381a-11ed-a950-0a580a80041d took 0:00:00.026261 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-8a0355c0-381a-11ed-a950-0a580a80041d took 0:00:00.026017 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-8a391619-381a-11ed-a950-0a580a80041d took 0:00:00.029473 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-73e917f0-381c-11ed-a950-0a580a80041d took 0:00:00.031857 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e20df102-381c-11ed-a950-0a580a80041d took 0:00:00.033644 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e25effc9-381c-11ed-a950-0a580a80041d took 0:00:00.032092 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e2e8cff3-381c-11ed-a950-0a580a80041d took 0:00:00.031281 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e36a0cbf-381c-11ed-a950-0a580a80041d took 0:00:00.031579 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e3ebc1d2-381c-11ed-a950-0a580a80041d took 0:00:00.026815 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e4f01ec2-381c-11ed-a950-0a580a80041d took 0:00:00.034416 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e54bcb56-381c-11ed-a950-0a580a80041d took 0:00:00.031419 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e5f27718-381c-11ed-a950-0a580a80041d took 0:00:00.028581 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d took 0:00:00.030709 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e6b1ae76-381c-11ed-a950-0a580a80041d took 0:00:00.029786 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e6c050bb-381c-11ed-a950-0a580a80041d took 0:00:00.028795 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e7b4ed59-381c-11ed-a950-0a580a80041d took 0:00:00.027939 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e7c44d98-381c-11ed-a950-0a580a80041d took 0:00:00.026886 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e8652201-381c-11ed-a950-0a580a80041d took 0:00:00.025077 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-e9f20823-381c-11ed-a950-0a580a80041d took 0:00:00.028138 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-89214d96-39bd-11ed-a950-0a580a80041d took 0:00:00.033437 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-89830b61-39bd-11ed-a950-0a580a80041d took 0:00:00.034117 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-8a0c46ea-39bd-11ed-a950-0a580a80041d took 0:00:00.032255 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-8c55110d-39bd-11ed-a950-0a580a80041d took 0:00:00.025290 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-8c551654-39bd-11ed-a950-0a580a80041d took 0:00:00.025014 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-0d46224f-39bf-11ed-a950-0a580a80041d took 0:00:00.031979 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-0daf49f7-39bf-11ed-a950-0a580a80041d took 0:00:00.031563 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-0e29770d-39bf-11ed-a950-0a580a80041d took 0:00:00.033158 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-15be6320-39bf-11ed-a950-0a580a80041d took 0:00:00.027820 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-15be648d-39bf-11ed-a950-0a580a80041d took 0:00:00.028793 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bd028d23-3f2e-11ed-a950-0a580a80041d took 0:00:00.032941 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-aaff17f3-3f31-11ed-a950-0a580a80041d took 0:00:00.031123 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-a15e0522-3f32-11ed-a950-0a580a80041d took 0:00:00.028350 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-ee31f15b-3f32-11ed-a950-0a580a80041d took 0:00:00.030680 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-ee7f8a28-3f32-11ed-a950-0a580a80041d took 0:00:00.029028 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-ef21b218-3f32-11ed-a950-0a580a80041d took 0:00:00.030983 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-ef7374d7-3f32-11ed-a950-0a580a80041d took 0:00:00.027169 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-f1d66839-3f32-11ed-a950-0a580a80041d took 0:00:00.037241 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-ba37502b-3f38-11ed-a950-0a580a80041d took 0:00:00.034181 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-ba86f7da-3f38-11ed-a950-0a580a80041d took 0:00:00.034535 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bb181e39-3f38-11ed-a950-0a580a80041d took 0:00:00.030155 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bb99c7c3-3f38-11ed-a950-0a580a80041d took 0:00:00.033312 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bc232b0e-3f38-11ed-a950-0a580a80041d took 0:00:00.034820 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bc8dae88-3f38-11ed-a950-0a580a80041d took 0:00:00.033322 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bd72b843-3f38-11ed-a950-0a580a80041d took 0:00:00.026598 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bdbf208e-3f38-11ed-a950-0a580a80041d took 0:00:00.030395 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-bdd622bc-3f38-11ed-a950-0a580a80041d took 0:00:00.031117 seconds
ReqID 0001-0011-openshift-storage-0000000000000002-be0f5dd5-3f38-11ed-a950-0a580a80041d took 0:00:00.025589 seconds

ReqID snapshot-970cf739-598b-4ea5-9fcb-250730037394 took 0:00:02.491645 seconds
ReqID snapshot-f0b6d5aa-492d-4449-a322-91c177d3174e took 0:00:02.116530 seconds
ReqID snapshot-f5989252-5258-4fbe-a8d7-572af1fb2cb0 took 0:00:01.922300 seconds
ReqID snapshot-9bee2fb0-4cd0-45c9-9297-2ce9887860c3 took 0:00:01.532367 seconds
ReqID snapshot-2bd939b0-99de-4378-9bc7-b890a31a80b4 took 0:00:01.541056 seconds
ReqID snapshot-d42e9471-05b8-42b8-896f-85783ca76eb6 took 0:00:01.360864 seconds
ReqID snapshot-3e309ed9-6537-4c19-9f83-fc8614a1b758 took 0:00:01.580551 seconds
ReqID snapshot-f076d306-2a39-4503-a24d-e17ff07e824b took 0:00:02.994352 seconds
ReqID snapshot-b335e980-9db9-4dae-a84f-058768cae0c8 took 0:00:02.593810 seconds
ReqID snapshot-84225e83-de90-4113-bb5f-ea352b53b619 took 0:00:03.011112 seconds

None of the PVC took more than a second. can you please check is that any problem with CNV itself? if we are sure we have problem with ODF i will dig further.