Bug 2231102

Summary:	[RDR][Tracker][Noobaa] VRG Deletion Pending: VRG Resources Awaiting cleanup from S3 Store, but Encountering Error 504.
Product:	[Red Hat Storage] Red Hat OpenShift Data Foundation	Reporter:	Aman Agrawal <amagrawa>
Component:	odf-dr	Assignee:	Benamar Mekhissi <bmekhiss>
odf-dr sub component:	ramen	QA Contact:	Aman Agrawal <amagrawa>
Status:	NEW ---	Docs Contact:
Severity:	urgent
Priority:	unspecified	CC:	belimele, kramdoss, kseeger, muagarwa, nbecker, rtalur
Version:	4.13	Flags:	bmekhiss: needinfo? (nbecker) belimele: needinfo? (amagrawa)
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:		Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Aman Agrawal 2023-08-10 14:56:03 UTC

Description of problem (please be detailed as possible and provide log
snippests):


Version of all relevant components (if applicable):
OCP 4.13.0-0.nightly-2023-07-27-073427
subctl version: v0.15.1 (with globalnet enabled)
ACM 2.8 GA'ed
ODF 4.13.0-219.snaptrim
ceph version 17.2.6-70.0.TEST.bz2119217.el9cp (6d74fefa15d1216867d1d112b47bb83c4913d28f) quincy (stable)



Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. On a RDR setup, deployed rbd based DR workloads on C1 (both subscription and appset based workloads) and ran IOs for a few days (8-10 days) and deleted the workloads via ACM UI by selecting option to delete all related resources (and ensuring mirroring status is healthy and subctl verify check is passing on both the managed clusters before workload deletion was trigerred).
2. It was found that the deployment (pods were deleted from C1) which is expected but s3 store still has the pv/pvc resources for the app workloads which shouldn't be there and was expected to be deleted as the deletion request was sent to VRG (deletiontimestamp could be seen in the VRG yaml).
3. 


Actual results: VRG Deletion Pending: VRG Resources Awaiting cleanup from S3 Store, but Encountering Error 504.

Expected results: No pv/pvc resources for the app should exist on the s3 store.


Additional info:

Comment 10 Aman Agrawal 2023-08-11 13:41:11 UTC

Thanks Benamar! Updated the bug description accordingly.