Description of problem (please be detailed as possible and provide log
snippests):
In ODF 4.16 cluster, configured to use custom snapshotter image, encountered an issue while attempting to create VolumeGroupSnapshot.
Stale volumesnapshot were observed when creating second volumegroupsnapshot.

Version of all relevant components (if applicable):
OCP: 4.16.0-0.nightly-2024-04-08-024331
ODF: 4.16.0-73.stable
ROOK_CSI_SNAPSHOTTER_IMAGE: "madhupr001/csi-snapshotter:v1" 

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?
No

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2

Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:

Steps to Reproduce:

1. Enable feature-gate and other temporary pre-req to use custom snapshotter image
2. Create volumegroupsnapshotclass
3. Create PVC (1Gi) with desired label (In my case "group: testing")
4. Create volumegroupsnapshot with label selector and check volumesnapshot output
$ oc get volumesnapshot -A
NAMESPACE           NAME                                                                                            READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT                                                                              RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                                                                    CREATIONTIME   AGE
openshift-storage   snapshot-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26   true                     snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26   1Gi                           snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26   19m            19m

5. Create another PVC (3Gi) with same label used above
6. Create another volumegroupsnapshot with same label selector and check volumesnapshot output
$ oc get volumesnapshot -A
NAMESPACE           NAME                                                                                            READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT                                                                              RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                                                                    CREATIONTIME   AGE
openshift-storage   snapshot-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.54   true                     snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.54                                 snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.54   26s            26s
openshift-storage   snapshot-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.55   true                     snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.55   1Gi                           snapcontent-18f81c102e931e187735cb8a019ffc4aa394fbd321349a1f6b3815b12afd86f8-2024-04-10-11.50.55   25s            25s
openshift-storage   snapshot-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26   true                     snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26   1Gi                           snapcontent-54d7fff698021f1c53caf9fe996b45acee60c70bf69f9538c280c7b9529e0298-2024-04-10-11.19.26   31m            31m
openshift-storage   snapshot-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.53   true                     snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.53   3Gi                           snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.53   27s            27s
openshift-storage   snapshot-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.55   true                     snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.55   3Gi                           snapcontent-7d174422e869f08d9669129d70f8d965ccaacedef753c3f41f357d9ea253ada4-2024-04-10-11.50.55   25s            25s


Actual results:
Stale resources in volumesnapshot output

Expected results:
No stale resources in volumesnapshot output

Additional info: