Description of problem (please be detailed as possible and provide log snippests): A degradation in cloning duration of 10 fedora DVs with copy cloneStrategy - from ~5 minutes at ODF 4.10.5 to ~9 minutes at ODF 4.11.1. At larger numbers it is slower, for 100 DVs on ODF 4.11 it got up to 33 minutes. Version of all relevant components (if applicable): ODF 4.11.1 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Yes, cloning got much slower. Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Yes Steps to Reproduce: 1. Build OCP 4.11.0-rc.6 with 3 masters and 3 workers 2. Install CNV 4.11 3. Install local storage 4. Install ODF 4.11.1 5. Change cloneStrategy to Copy 6. Create 10 fedora VMS with a template Actual results: DVS Cloning take ~9 minutes Expected results: Should take ~5 minutes, as in 4.10.5 Additional info: will be added
At initial glance From the rbdplugin logs i can see below cat current.log |grep err 2022-09-28T12:24:45.988776907Z E0928 12:24:45.988761 1 utils.go:200] ID: 13048 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:24:46.991721772Z E0928 12:24:46.991698 1 utils.go:200] ID: 13049 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:24:48.994549940Z E0928 12:24:48.994541 1 utils.go:200] ID: 13050 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:24:52.997178786Z E0928 12:24:52.997164 1 utils.go:200] ID: 13052 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:25:01.000189450Z E0928 12:25:01.000181 1 utils.go:200] ID: 13053 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:25:17.002706943Z E0928 12:25:17.002695 1 utils.go:200] ID: 13054 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:25:49.005530784Z E0928 12:25:49.005491 1 utils.go:200] ID: 13055 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:26:53.008542197Z E0928 12:26:53.008515 1 utils.go:200] ID: 13058 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:29:01.011394246Z E0928 12:29:01.011377 1 utils.go:200] ID: 13061 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:29:53.962274399Z E0928 12:29:53.962259 1 utils.go:200] ID: 13063 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:33:17.014674154Z E0928 12:33:17.014340 1 utils.go:200] ID: 13067 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists 2022-09-28T12:38:17.017602028Z E0928 12:38:17.017590 1 utils.go:200] ID: 13073 Req-ID: 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d already exists ReqID 0001-0011-openshift-storage-0000000000000002-408eeba9-3819-11ed-a950-0a580a80041d took 0:00:00.039935 seconds ReqID 0001-0011-openshift-storage-0000000000000002-b31f4b59-3819-11ed-a950-0a580a80041d took 0:00:00.032579 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f373b86e-3819-11ed-a950-0a580a80041d took 0:00:00.143195 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f706e231-3819-11ed-a950-0a580a80041d took 0:00:00.149968 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f743e7a4-3819-11ed-a950-0a580a80041d took 0:00:00.147614 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f8957c50-3819-11ed-a950-0a580a80041d took 0:00:00.161886 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f94b8677-3819-11ed-a950-0a580a80041d took 0:00:00.156478 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f98784d7-3819-11ed-a950-0a580a80041d took 0:00:00.145785 seconds ReqID 0001-0011-openshift-storage-0000000000000002-fa20ca10-3819-11ed-a950-0a580a80041d took 0:00:00.149611 seconds ReqID 0001-0011-openshift-storage-0000000000000002-fad70096-3819-11ed-a950-0a580a80041d took 0:00:00.145393 seconds ReqID 0001-0011-openshift-storage-0000000000000002-fb5179bb-3819-11ed-a950-0a580a80041d took 0:00:00.146231 seconds ReqID 0001-0011-openshift-storage-0000000000000002-fc26966c-3819-11ed-a950-0a580a80041d took 0:00:00.155550 seconds ReqID 0001-0011-openshift-storage-0000000000000002-86a8b42c-381a-11ed-a950-0a580a80041d took 0:00:00.026904 seconds ReqID 0001-0011-openshift-storage-0000000000000002-86ee16a0-381a-11ed-a950-0a580a80041d took 0:00:00.024317 seconds ReqID 0001-0011-openshift-storage-0000000000000002-87733c2e-381a-11ed-a950-0a580a80041d took 0:00:00.031198 seconds ReqID 0001-0011-openshift-storage-0000000000000002-87f53bcd-381a-11ed-a950-0a580a80041d took 0:00:00.028428 seconds ReqID 0001-0011-openshift-storage-0000000000000002-887e5672-381a-11ed-a950-0a580a80041d took 0:00:00.030102 seconds ReqID 0001-0011-openshift-storage-0000000000000002-896325cf-381a-11ed-a950-0a580a80041d took 0:00:00.029635 seconds ReqID 0001-0011-openshift-storage-0000000000000002-89a04c86-381a-11ed-a950-0a580a80041d took 0:00:00.028530 seconds ReqID 0001-0011-openshift-storage-0000000000000002-89f2e4b6-381a-11ed-a950-0a580a80041d took 0:00:00.026261 seconds ReqID 0001-0011-openshift-storage-0000000000000002-8a0355c0-381a-11ed-a950-0a580a80041d took 0:00:00.026017 seconds ReqID 0001-0011-openshift-storage-0000000000000002-8a391619-381a-11ed-a950-0a580a80041d took 0:00:00.029473 seconds ReqID 0001-0011-openshift-storage-0000000000000002-73e917f0-381c-11ed-a950-0a580a80041d took 0:00:00.031857 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e20df102-381c-11ed-a950-0a580a80041d took 0:00:00.033644 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e25effc9-381c-11ed-a950-0a580a80041d took 0:00:00.032092 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e2e8cff3-381c-11ed-a950-0a580a80041d took 0:00:00.031281 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e36a0cbf-381c-11ed-a950-0a580a80041d took 0:00:00.031579 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e3ebc1d2-381c-11ed-a950-0a580a80041d took 0:00:00.026815 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e4f01ec2-381c-11ed-a950-0a580a80041d took 0:00:00.034416 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e54bcb56-381c-11ed-a950-0a580a80041d took 0:00:00.031419 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e5f27718-381c-11ed-a950-0a580a80041d took 0:00:00.028581 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e6373fab-381c-11ed-a950-0a580a80041d took 0:00:00.030709 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e6b1ae76-381c-11ed-a950-0a580a80041d took 0:00:00.029786 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e6c050bb-381c-11ed-a950-0a580a80041d took 0:00:00.028795 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e7b4ed59-381c-11ed-a950-0a580a80041d took 0:00:00.027939 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e7c44d98-381c-11ed-a950-0a580a80041d took 0:00:00.026886 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e8652201-381c-11ed-a950-0a580a80041d took 0:00:00.025077 seconds ReqID 0001-0011-openshift-storage-0000000000000002-e9f20823-381c-11ed-a950-0a580a80041d took 0:00:00.028138 seconds ReqID 0001-0011-openshift-storage-0000000000000002-89214d96-39bd-11ed-a950-0a580a80041d took 0:00:00.033437 seconds ReqID 0001-0011-openshift-storage-0000000000000002-89830b61-39bd-11ed-a950-0a580a80041d took 0:00:00.034117 seconds ReqID 0001-0011-openshift-storage-0000000000000002-8a0c46ea-39bd-11ed-a950-0a580a80041d took 0:00:00.032255 seconds ReqID 0001-0011-openshift-storage-0000000000000002-8c55110d-39bd-11ed-a950-0a580a80041d took 0:00:00.025290 seconds ReqID 0001-0011-openshift-storage-0000000000000002-8c551654-39bd-11ed-a950-0a580a80041d took 0:00:00.025014 seconds ReqID 0001-0011-openshift-storage-0000000000000002-0d46224f-39bf-11ed-a950-0a580a80041d took 0:00:00.031979 seconds ReqID 0001-0011-openshift-storage-0000000000000002-0daf49f7-39bf-11ed-a950-0a580a80041d took 0:00:00.031563 seconds ReqID 0001-0011-openshift-storage-0000000000000002-0e29770d-39bf-11ed-a950-0a580a80041d took 0:00:00.033158 seconds ReqID 0001-0011-openshift-storage-0000000000000002-15be6320-39bf-11ed-a950-0a580a80041d took 0:00:00.027820 seconds ReqID 0001-0011-openshift-storage-0000000000000002-15be648d-39bf-11ed-a950-0a580a80041d took 0:00:00.028793 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bd028d23-3f2e-11ed-a950-0a580a80041d took 0:00:00.032941 seconds ReqID 0001-0011-openshift-storage-0000000000000002-aaff17f3-3f31-11ed-a950-0a580a80041d took 0:00:00.031123 seconds ReqID 0001-0011-openshift-storage-0000000000000002-a15e0522-3f32-11ed-a950-0a580a80041d took 0:00:00.028350 seconds ReqID 0001-0011-openshift-storage-0000000000000002-ee31f15b-3f32-11ed-a950-0a580a80041d took 0:00:00.030680 seconds ReqID 0001-0011-openshift-storage-0000000000000002-ee7f8a28-3f32-11ed-a950-0a580a80041d took 0:00:00.029028 seconds ReqID 0001-0011-openshift-storage-0000000000000002-ef21b218-3f32-11ed-a950-0a580a80041d took 0:00:00.030983 seconds ReqID 0001-0011-openshift-storage-0000000000000002-ef7374d7-3f32-11ed-a950-0a580a80041d took 0:00:00.027169 seconds ReqID 0001-0011-openshift-storage-0000000000000002-f1d66839-3f32-11ed-a950-0a580a80041d took 0:00:00.037241 seconds ReqID 0001-0011-openshift-storage-0000000000000002-ba37502b-3f38-11ed-a950-0a580a80041d took 0:00:00.034181 seconds ReqID 0001-0011-openshift-storage-0000000000000002-ba86f7da-3f38-11ed-a950-0a580a80041d took 0:00:00.034535 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bb181e39-3f38-11ed-a950-0a580a80041d took 0:00:00.030155 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bb99c7c3-3f38-11ed-a950-0a580a80041d took 0:00:00.033312 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bc232b0e-3f38-11ed-a950-0a580a80041d took 0:00:00.034820 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bc8dae88-3f38-11ed-a950-0a580a80041d took 0:00:00.033322 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bd72b843-3f38-11ed-a950-0a580a80041d took 0:00:00.026598 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bdbf208e-3f38-11ed-a950-0a580a80041d took 0:00:00.030395 seconds ReqID 0001-0011-openshift-storage-0000000000000002-bdd622bc-3f38-11ed-a950-0a580a80041d took 0:00:00.031117 seconds ReqID 0001-0011-openshift-storage-0000000000000002-be0f5dd5-3f38-11ed-a950-0a580a80041d took 0:00:00.025589 seconds ReqID snapshot-970cf739-598b-4ea5-9fcb-250730037394 took 0:00:02.491645 seconds ReqID snapshot-f0b6d5aa-492d-4449-a322-91c177d3174e took 0:00:02.116530 seconds ReqID snapshot-f5989252-5258-4fbe-a8d7-572af1fb2cb0 took 0:00:01.922300 seconds ReqID snapshot-9bee2fb0-4cd0-45c9-9297-2ce9887860c3 took 0:00:01.532367 seconds ReqID snapshot-2bd939b0-99de-4378-9bc7-b890a31a80b4 took 0:00:01.541056 seconds ReqID snapshot-d42e9471-05b8-42b8-896f-85783ca76eb6 took 0:00:01.360864 seconds ReqID snapshot-3e309ed9-6537-4c19-9f83-fc8614a1b758 took 0:00:01.580551 seconds ReqID snapshot-f076d306-2a39-4503-a24d-e17ff07e824b took 0:00:02.994352 seconds ReqID snapshot-b335e980-9db9-4dae-a84f-058768cae0c8 took 0:00:02.593810 seconds ReqID snapshot-84225e83-de90-4113-bb5f-ea352b53b619 took 0:00:03.011112 seconds None of the PVC took more than a second. can you please check is that any problem with CNV itself? if we are sure we have problem with ODF i will dig further.