Description of problem (please be detailed as possible and provide log
snippests):


Version of all relevant components (if applicable):
OCP 4.13.0-0.nightly-2023-07-27-073427
subctl version: v0.15.1 (with globalnet enabled)
ACM 2.8 GA'ed
ODF 4.13.0-219.snaptrim
ceph version 17.2.6-70.0.TEST.bz2119217.el9cp (6d74fefa15d1216867d1d112b47bb83c4913d28f) quincy (stable)



Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. On a RDR setup, deployed rbd based DR workloads on C1 (both subscription and appset based workloads) and ran IOs for a few days (8-10 days) and deleted the workloads via ACM UI by selecting option to delete all related resources (and ensuring mirroring status is healthy and subctl verify check is passing on both the managed clusters before workload deletion was trigerred).
2. It was found that the deployment (pods were deleted from C1) which is expected but s3 store still has the pv/pvc resources for the app workloads which shouldn't be there and was expected to be deleted as the deletion request was sent to VRG (deletiontimestamp could be seen in the VRG yaml).
3. 


Actual results: VRG Deletion Pending: VRG Resources Awaiting cleanup from S3 Store, but Encountering Error 504.

Expected results: No pv/pvc resources for the app should exist on the s3 store.


Additional info: