Bug 2252880

+++ This bug was initially created as a clone of Bug #2252756 +++

Description of problem (please be detailed as possible and provide log
snippests):


Version of all relevant components (if applicable):
OCP 4.14.0-0.nightly-2023-11-30-174049
ACM 2.9.0 GA'ed (from OperatorHub)
ODF 4.14.1-15
ceph version 17.2.6-161.el9cp (7037a43d8f7fa86659a0575b566ec10080df0d71) quincy (stable)
Submariner 0.16.2
VolSync 0.8.0


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Deployed multiple rbd and cephfs backed workloads of both appset and subscription types.
2. Failedover and relocated them in such a way that they are finally running on the primary managed cluster (which is expected to host all the workloads and can go under disaster).
3. Ensure that we have all the workloads in distict states like deployed, failedover, relocated etc.
4. Let the latest backups be taken at least 1 or 2 (at each 1 hr) for all the different states of the workloads (when progression is completed and no action is going on any of the workloads). Also ensure sync for all the workloads when on active hub is working fine and cluster is healthy. Note drpc -o wide, lastGroupSyncTime etc.
5. Bring active hub completely down, move to passive hub. Restore backps, ensure velero backup reports successful restoration. Make sure both the managed clusters are successfully imported, drpolicy gets validated.
6. Wait for drpc to be restored, check if all the workloads are in their last backedup state or not.
They seem to have retained their last state which was backedup. So everything is fine so far. Label cluster-monitoring on the hub cluster so that VolumeSync.DelayAlert is fired if data sync is affected for any workload.
7. Let IOs continue and check lastGroupSyncTime and VolumeSync.DelayAlert alert. sync for rbd based workloads were progressing just fine along with other cephfs backed workloads except appset-cephfs-busybox9-placement-drpc in NS busybox-workloads-9.
8. Upon further validation, it was found that the dst pods and PVCs were lost from the secondary managed cluster for this workload.

(Older hub remains down forever and is completely unreachable).


Actual results: dst pods and PVCs are lost from the secondary managed cluster for appset-cephfs-busybox9-placement-drpc in NS busybox-workloads-9 

From C2-

amagrawa:~$ oc get pods,vrg,vr,pvc -o wide -n busybox-workloads-9
NAME                                                                                DESIREDSTATE   CURRENTSTATE
volumereplicationgroup.ramendr.openshift.io/appset-cephfs-busybox9-placement-drpc   secondary      


From passive hub-

drpc-
openshift-gitops       appset-cephfs-busybox9-placement-drpc    23h   amagrawa-1-1d                                       Deployed       Completed                                           True


We also found a bunch of NonCompliant policies post hub recovery however all the policies were Compliant on the older hub.

amagrawa:~$ oc get policy -A | grep NonCompliant
amagrawa-1-1d                    busybox-workloads-12.vs-secret-9b0006378bc4b3dde4a82c04dd7dd3f6                        NonCompliant       23h
amagrawa-1-1d                    openshift-gitops.vs-secret-e00de88ac33afae138ce0bc1dc989ce1                            NonCompliant       23h
amagrawa-2-1d                    busybox-workloads-10.vs-secret-674962f86373128a3005a9a581929a8c                        NonCompliant       23h
amagrawa-2-1d                    busybox-workloads-11.vs-secret-991de86951c29d86a7b4f21afcc222a0                        NonCompliant       23h
amagrawa-2-1d                    busybox-workloads-16.vs-secret-875a04e3829a1b8e35315f7c4a6e0c66                        NonCompliant       23h
amagrawa-2-1d                    openshift-gitops.vs-secret-1a2537d31f7cda5a558e40664f973bd4                            NonCompliant       23h
amagrawa-2-1d                    openshift-gitops.vs-secret-7e65a08e63102201af8f9ada3062686b                            NonCompliant       23h
amagrawa-2-1d                    openshift-gitops.vs-secret-e00de88ac33afae138ce0bc1dc989ce1                            NonCompliant       23h
amagrawa-2-1d                    openshift-gitops.vs-secret-e3b95572f8ad4080e8aec77f9b19d4e4                            NonCompliant       23h
busybox-workloads-10             vs-secret-674962f86373128a3005a9a581929a8c                                             NonCompliant       23h
busybox-workloads-11             vs-secret-991de86951c29d86a7b4f21afcc222a0                                             NonCompliant       23h
busybox-workloads-12             vs-secret-9b0006378bc4b3dde4a82c04dd7dd3f6                                             NonCompliant       23h
busybox-workloads-16             vs-secret-875a04e3829a1b8e35315f7c4a6e0c66                                             NonCompliant       23h
local-cluster                    open-cluster-management-backup.backup-restore-enabled             inform               NonCompliant       28h
open-cluster-management-backup   backup-restore-enabled                                            inform               NonCompliant       28h
openshift-gitops                 vs-secret-1a2537d31f7cda5a558e40664f973bd4                                             NonCompliant       23h
openshift-gitops                 vs-secret-7e65a08e63102201af8f9ada3062686b                                             NonCompliant       23h
openshift-gitops                 vs-secret-e00de88ac33afae138ce0bc1dc989ce1                                             NonCompliant       23h
openshift-gitops                 vs-secret-e3b95572f8ad4080e8aec77f9b19d4e4                                             NonCompliant       23h


Due to missing volumes on the secondary site, data sync isn't progressing for this workload and if in case primary site goes down or there is a need to perform relocate on this workload which we still can, it can lead to complete loss of data (assuming workload pods won't come up on the secondary cluster). 


Logs collected some time after moving to passive hub could be downloaded from https://drive.google.com/file/d/16aUyq1tbkKpumnE6Bmzx1PzBbvuBMwbI/view?usp=drive_link

Pls note it can not unzipped as it's in the Gdrive and not in QE server (which is currently down).

Expected results: volumes shouldn't be lost on the secondary site and data sync should continue just fine for all the DR protected cephfs backed workloads


Additional info:

--- Additional comment from RHEL Program Management on 2023-12-04 14:26:19 UTC ---

This bug having no release flag set previously, is now set with release flag 'odf‑4.15.0' to '?', and so is being proposed to be fixed at the ODF 4.15.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.

--- Additional comment from RHEL Program Management on 2023-12-04 14:26:19 UTC ---

Since this bug has severity set to 'urgent', it is being proposed as a blocker for the currently set release flag. Please resolve ASAP.

--- Additional comment from Benamar Mekhissi on 2023-12-04 15:58:49 UTC ---

Here are the preliminary issues:

1. VRG on cluster2 got deleted, most likely due to the eviction routine. Why did this happen?
2. It is possible that the secondary VRG will always be deleted. Busybox-workloads-12 got deleted as well, but it recovered because it didn't experience the next issue.
3. The VRG busybox-workloads-9 got stuck because the policy became noncompliant, resulting in the secret not getting delivered.
4. The policy for busybox-workloads-9 was restored before the secret. We were told (by ACM backup team) that secrets would be restored before policies.


For issue 4, the secret is being restored after the policy. This caused the Policy to be NonCompliant and therefore, the secret didn't get propagated by the policy to the managed clusters.

The policy is created "2023-12-03T13:13:51Z"
The secret is created "2023-12-03T13:14:22Z"

The velero logs shows that the policy was restored from backup
```
time="2023-12-03T13:13:10Z" level=info msg="the managed fields for openshift-gitops/vs-secret-e3b95572f8ad4080e8aec77f9b19d4e4 is patched" logSource="/remote-source/velero/app/pkg/restore/restore.go:1522" restore=open-cluster-management-backup/restore-acm-acm-resources-schedule-20231203130337
time="2023-12-03T13:13:12Z" level=info msg="the managed fields for openshift-gitops/vs-secret-e3b95572f8ad4080e8aec77f9b19d4e4-decision-1 is patched" logSource="/remote-source/velero/app/pkg/restore/restore.go:1522" restore=open-cluster-management-backup/restore-acm-acm-resources-schedule-20231203130337
time="2023-12-03T13:13:13Z" level=info msg="the managed fields for openshift-gitops/vs-secret-e3b95572f8ad4080e8aec77f9b19d4e4 is patched" logSource="/remote-source/velero/app/pkg/restore/restore.go:1522" restore=open-cluster-management-backup/restore-acm-acm-resources-schedule-20231203130337
time="2023-12-03T13:13:16Z" level=info msg="the managed fields for openshift-gitops/vs-secret-e3b95572f8ad4080e8aec77f9b19d4e4 is patched" logSource="/remote-source/velero/app/pkg/restore/restore.go:1522" restore=open-cluster-management-backup/restore-acm-acm-resources-schedule-20231203130337
```

and for secret it is here:

```
time="2023-12-03T13:14:22Z" level=info msg="[secret-restore] Secret: appset-cephfs-busybox9-placement-drpc-vs-secret-hub" cmd=/plugins/velero-plugins logSource="/remote-source/src/github.com/konveyor/openshift-velero-plugin/velero-plugins/secret/restore.go:34" pluginName=velero-plugins restore=open-cluster-management-backup/restore-acm-acm-credentials-schedule-20231203130337
time="2023-12-03T13:14:22Z" level=info msg="Attempting to restore Secret: appset-cephfs-busybox9-placement-drpc-vs-secret-hub" logSource="/remote-source/velero/app/pkg/restore/restore.go:1337" restore=open-cluster-management-backup/restore-acm-acm-credentials-schedule-20231203130337
time="2023-12-03T13:14:22Z" level=info msg="the managed fields for openshift-gitops/appset-cephfs-busybox9-placement-drpc-vs-secret-hub is patched" logSource="/remote-source/velero/app/pkg/restore/restore.go:1522" restore=open-cluster-management-backup/restore-acm-acm-credentials-schedule-20231203130337
time="2023-12-03T13:14:22Z" level=info msg="Restored 16 items out of an estimated total of 20 (estimate will change throughout the restore)" logSource="/remote-source/velero/app/pkg/restore/restore.go:669" name=appset-cephfs-busybox9-placement-drpc-vs-secret-hub namespace=openshift-gitops progress= resource=secrets restore=open-cluster-management-backup/restore-acm-acm-credentials-schedule-20231203130337
```

I have engaged Valentina and we are looking at the issue as we speak.

More updates later...

--- Additional comment from Benamar Mekhissi on 2023-12-04 16:19:20 UTC ---

Issue 3 and 4 above are tracked by this jira: https://issues.redhat.com/browse/ACM-8857

--- Additional comment from Benamar Mekhissi on 2023-12-04 18:13:24 UTC ---

The second part of the issue has been identified. The VRG AppliedManifestWork was evicted by ACM for failure in recreating the secondary manifest work for cephfs workloads. It impacts workloads that are in the deployed phase only. We'll fix it.

--- Additional comment from Benamar Mekhissi on 2023-12-04 20:33:05 UTC ---

PR for the Ramen fix: https://github.com/RamenDR/ramen/pull/1157