Description of problem (please be detailed as possible and provide log snippests): When assigning DRPolicy to Subscription type applications, if multiple applications are selected at once, then DR protection is not applied for those applications. Version of all relevant components (if applicable): OCP 4.13.0-0.nightly-2023-06-20-224158 ODF 4.13.0-219.snaptrim ceph version 17.2.6-70.0.TEST.bz2119217.el9cp (6d74fefa15d1216867d1d112b47bb83c4913d28f) quincy (stable) ACM 2.8.0 Submariner 0.15.1 (Globalnet enabled) VolSync 0.7.1 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Configure RDR setup 2. Deploy multiple Subscription type applications in different namespaces 3. Go to Disaster recovery page -> Assign policy to Subscriptions 4. Select all applications and enable `Protect all PVCs within the application's namespace` to proceed. 5. Observe DRPC status. DR protection is not applied and VR, VRG resources not created Actual results: Applications are not DR protected when multiple applications are selected during policy assignment Expected results: All applications should be DR protected Additional Info: DRPC output: NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-1 busybox-1-placement-1-drpc 15m sagrawal-nc1 busybox-2 busybox-2-placement-1-drpc 15m sagrawal-nc1 $ oc get drpc -A -o yaml apiVersion: v1 items: - apiVersion: ramendr.openshift.io/v1alpha1 kind: DRPlacementControl metadata: creationTimestamp: "2023-07-11T13:11:40Z" generation: 1 labels: app: busybox-1 name: busybox-1-placement-1-drpc namespace: busybox-1 resourceVersion: "13679988" uid: 04413d0b-ed35-47e4-8738-7082f300d7f5 spec: drPolicyRef: apiVersion: ramendr.openshift.io/v1alpha1 kind: DRPolicy name: odr-policy-5m placementRef: apiVersion: apps.open-cluster-management.io/v1 kind: Placement name: busybox-1-placement-1 namespace: busybox-1 preferredCluster: sagrawal-nc1 pvcSelector: matchLabels: {} - apiVersion: ramendr.openshift.io/v1alpha1 kind: DRPlacementControl metadata: creationTimestamp: "2023-07-11T13:11:40Z" generation: 1 labels: app: busybox-2 name: busybox-2-placement-1-drpc namespace: busybox-2 resourceVersion: "13679991" uid: 2100756e-78e2-4ceb-9d8d-7e5fc7c63ea7 spec: drPolicyRef: apiVersion: ramendr.openshift.io/v1alpha1 kind: DRPolicy name: odr-policy-5m placementRef: apiVersion: apps.open-cluster-management.io/v1 kind: Placement name: busybox-2-placement-1 namespace: busybox-2 preferredCluster: sagrawal-nc1 pvcSelector: matchLabels: {} kind: List metadata: resourceVersion: "" >> No VRG/ VR resources created for any application $ oc get pods,pvc,vr,vrg -o wide -n busybox-1 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/busybox-5699775f99-mffd4 1/1 Running 0 26m 10.131.0.253 compute-1 <none> <none> NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE persistentvolumeclaim/busybox-pvc Bound pvc-6ad5e5cb-ae61-4f4d-86df-4e923d863040 5Gi RWO ocs-storagecluster-ceph-rbd 26m Filesystem $ oc get pods,pvc,vr,vrg -o wide -n busybox-2 NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/busybox-5699775f99-h95vb 1/1 Running 0 24m 10.128.3.101 compute-2 <none> <none> NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE persistentvolumeclaim/busybox-pvc Bound pvc-99174080-1130-4add-8b72-3a276888b7ac 5Gi RWO ocs-storagecluster-ceph-rbd 24m Filesystem >> ramen-hub-operator pod logs show following error messages: 2023-07-11T13:17:08.755Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1070 Using placement &{{Placement cluster.open-cluster-management.io/v1beta1} {busybox-1-placement-1 busybox-1 6ca74377-0a7b-460c-be47-052f5e61a622 13679990 2 2023-07-11 13:04:41 +0000 UTC <nil> <nil> map[app:busybox-1] map[cluster.open-cluster-management.io/experimental-scheduling-disable:true drplacementcontrol.ramendr.openshift.io/drpc-name:busybox-1-placement-1-drpc drplacementcontrol.ramendr.openshift.io/drpc-namespace:busybox-1] [] [] [{placement Update cluster.open-cluster-management.io/v1beta1 2023-07-11 13:04:41 +0000 UTC FieldsV1 {"f:status":{".":{},"f:conditions":{},"f:numberOfSelectedClusters":{}}} status} {unknown Update cluster.open-cluster-management.io/v1beta1 2023-07-11 13:04:41 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{".":{},"f:app":{}}},"f:spec":{".":{},"f:clusterSets":{}}} } {Mozilla Update cluster.open-cluster-management.io/v1beta1 2023-07-11 13:11:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:cluster.open-cluster-management.io/experimental-scheduling-disable":{}}}} } {manager Update cluster.open-cluster-management.io/v1beta1 2023-07-11 13:11:40 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:drplacementcontrol.ramendr.openshift.io/drpc-name":{},"f:drplacementcontrol.ramendr.openshift.io/drpc-namespace":{}}},"f:spec":{"f:predicates":{},"f:prioritizerPolicy":{".":{},"f:mode":{}},"f:spreadPolicy":{}}} }]} {[submariner] <nil> [{{{map[] [{name In [sagrawal-nc1]}]} {[]}}}] {Additive []} {[]} []} {1 [{PlacementMisconfigured False 0 2023-07-11 13:04:41 +0000 UTC Succeedconfigured Placement configurations check pass} {PlacementSatisfied True 0 2023-07-11 13:04:41 +0000 UTC AllDecisionsScheduled All cluster decisions scheduled}]}} {"DRPC": "busybox-1/busybox-1-placement-1-drpc", "rid": "70fb387e-e0aa-4267-bb8d-bffd818f20b4"} 2023-07-11T13:17:08.757Z DEBUG controller-runtime.webhook.webhooks admission/http.go:96 received request {"webhook": "/validate-ramendr-openshift-io-v1alpha1-drplacementcontrol", "UID": "fe9e0521-6180-48a0-ac21-98cb0d139b8d", "kind": "ramendr.openshift.io/v1alpha1, Kind=DRPlacementControl", "resource": {"group":"ramendr.openshift.io","version":"v1alpha1","resource":"drplacementcontrols"}} 2023-07-11T13:17:08.757Z INFO drplacementcontrol-resource v1alpha1/drplacementcontrol_webhook.go:39 validate update {"name": "busybox-1-placement-1-drpc"} 2023-07-11T13:17:08.757Z INFO drplacementcontrol-resource v1alpha1/drplacementcontrol_webhook.go:64 detected PVCSelector updates, which is disallowed {"name": "busybox-1-placement-1-drpc", "old": {}, "new": {}} 2023-07-11T13:17:08.757Z DEBUG controller-runtime.webhook.webhooks admission/http.go:143 wrote response {"webhook": "/validate-ramendr-openshift-io-v1alpha1-drplacementcontrol", "code": 403, "reason": "PVCSelector cannot be changed", "UID": "fe9e0521-6180-48a0-ac21-98cb0d139b8d", "allowed": false} 2023-07-11T13:17:08.760Z ERROR controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:857 Failed to add label and finalizer to drpc {"DRPC": "busybox-1/busybox-1-placement-1-drpc", "rid": "70fb387e-e0aa-4267-bb8d-bffd818f20b4", "error": "admission webhook \"vdrplacementcontrol.kb.io\" denied the request: PVCSelector cannot be changed"} github.com/ramendr/ramen/controllers.DRPlacementControlReconciler.addLabelsAndFinalizers /remote-source/app/controllers/drplacementcontrol_controller.go:857 github.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).createDRPCInstance /remote-source/app/controllers/drplacementcontrol_controller.go:705 github.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).Reconcile /remote-source/app/controllers/drplacementcontrol_controller.go:643 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:122 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:323 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:235 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1455 Updating DRPC status {"DRPC": "busybox-1/busybox-1-placement-1-drpc", "rid": "70fb387e-e0aa-4267-bb8d-bffd818f20b4"} 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1713 Retrieved ApplicationSets {"count": 0} 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1732 Placement busybox-1-placement-1 does not belong to any ApplicationSet. Defaulting the dest namespace to busybox-1 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1588 Found ClusterDecision {"ClsDedicision": [{"clusterName":"sagrawal-nc1","reason":""}]} 2023-07-11T13:17:08.760Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-07-11 12:41:47 +0000 UTC GetResourceFailed failed to get resource with err: volumereplicationgroups.ramendr.openshift.io "busybox-1-placement-1-drpc" not found}] {"resourceName": "busybox-1-placement-1-drpc"} 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1504 Failed to get VRG from managed cluster {"DRPC": "busybox-1/busybox-1-placement-1-drpc", "rid": "70fb387e-e0aa-4267-bb8d-bffd818f20b4", "errMsg": "getManagedClusterResource results: \"requested resource not found in ManagedCluster\" not found", "errMsgVerbose": " \"requested resource not found in ManagedCluster\" not found\ngetManagedClusterResource results\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.GetResource\n\t/remote-source/app/controllers/util/mcv_util.go:278\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.getManagedClusterResource\n\t/remote-source/app/controllers/util/mcv_util.go:225\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.GetVRGFromManagedCluster\n\t/remote-source/app/controllers/util/mcv_util.go:81\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).updateResourceCondition\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:1501\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).updateDRPCStatus\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:1464\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).Reconcile\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:645\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1594"} 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1474 No need to update DRPC Status {"DRPC": "busybox-1/busybox-1-placement-1-drpc", "rid": "70fb387e-e0aa-4267-bb8d-bffd818f20b4"} 2023-07-11T13:17:08.760Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:647 Exiting reconcile loop {"DRPC": "busybox-1/busybox-1-placement-1-drpc", "rid": "70fb387e-e0aa-4267-bb8d-bffd818f20b4"} 2023-07-11T13:17:08.760Z ERROR controller/controller.go:329 Reconciler error {"controller": "drplacementcontrol", "controllerGroup": "ramendr.openshift.io", "controllerKind": "DRPlacementControl", "DRPlacementControl": {"name":"busybox-1-placement-1-drpc","namespace":"busybox-1"}, "namespace": "busybox-1", "name": "busybox-1-placement-1-drpc", "reconcileID": "e9aa7e51-bcca-42c2-a448-b9c73b230ff0", "error": "failed to create DRPC instance (admission webhook \"vdrplacementcontrol.kb.io\" denied the request: PVCSelector cannot be changed) and (<nil>)"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:329 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:274 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.5/pkg/internal/controller/controller.go:235