Description of problem (please be detailed as possible and provide log snippests): These 2 rbd based workloads were failedover to C2 which were in deployed state earlier. amagrawa:acm$ drpc NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 2d23h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True openshift-gitops busybox-workloads-1-placement-drpc 2d23h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True Version of all relevant components (if applicable): ODF 4.14.0-132.stable OCP 4.14.0-0.nightly-2023-09-02-132842 ACM 2.9.0-DOWNSTREAM-2023-08-24-09-30-12 subctl version: v0.16.0 ceph version 17.2.6-138.el9cp (b488c8dad42b2ecffcd96f3d76eeeecce48b8590) quincy (stable) Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Deploy appset and subscription based DR protected rbd based workloads on a RDR setup and run IOs for some time (4-5 days in this case) 2. Perform submariner connectivity check, ensure mirroring is working, lastGroupSyncTime is within desired range, rbd image status is healthy, etc. 3. Bring master nodes of primary cluster down 4. Perform failover of both appset and subscription based workloads when cluster is marked unavailable on the ACM UI 5. Make sure to check drpc -o wide status in a loop before failover is triggered and check the CURRENTSTATE which reports FailedOver and PROGRESSION which says Completed the moment failover is trigerred and stays like that for a few mins and then the PROGRESSION status changes further to WaitForStorageMaintenanceActivation, WaitingForResourceRestore, Cleaning Up etc. Actual results: DRPC directly reports progression as Completed the moment failover is triggered. Must gather logs are kept here- http://rhsqe-repo.lab.eng.blr.redhat.com/OCS/ocs-qe-bugs/bz-aman/19sept23-1/ Expected results: DRPC progression directly reporting as Completed the moment failover is triggered is misleading and appropriate progression status should be shown instead. Additional info:
I am marking this as a Regression bcz- 1. It wasn't the case with 4.13 as drpc used to show the correct status. 2. We are experiencing increased timeout in kicking maintenance mode being activated on the failover cluster as seen from the drpc output. It can now take upto 3-4mins to kick in mmode. amagrawa:~$ while true; do oc get drpc -o wide -A; sleep 5; echo "********************************"; done NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c2 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:17:45Z 44m1.2498824s True openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailedOver Completed 2023-09-18T19:18:58Z 48m18.757117217s True busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitingForResourceRestore 2023-09-20T12:13:32Z False openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitForStorageMaintenanceActivation 2023-09-20T12:13:43Z False busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitingForResourceRestore 2023-09-20T12:13:32Z False openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitForStorageMaintenanceActivation 2023-09-20T12:13:43Z False busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitingForResourceRestore 2023-09-20T12:13:32Z False openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 busybox-workloads-2-placement-1-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitForStorageMaintenanceActivation 2023-09-20T12:13:43Z False busybox-workloads-5 busybox-workloads-5-placement-1-drpc 6d3h amagrawa-c2 Deployed Completed True openshift-gitops busybox-workloads-1-placement-drpc 4d4h amagrawa-c1 amagrawa-c1 Failover FailingOver WaitingForResourceRestore 2023-09-20T12:13:32Z False openshift-gitops busybox-workloads-3-placement-drpc 8d amagrawa-c1 Deployed EnsuringVolSyncSetup True ******************************** From failover cluster- amagrawa:c1-acm$ mm -o yaml apiVersion: v1 items: - apiVersion: ramendr.openshift.io/v1alpha1 kind: MaintenanceMode metadata: creationTimestamp: "2023-09-20T12:15:33Z" finalizers: - maintenance.multicluster.odf.openshift.io generation: 1 name: e3f84faf9e8077c06e0a94ca290c447e37d5cf3 ownerReferences: - apiVersion: work.open-cluster-management.io/v1 kind: AppliedManifestWork name: f9e723bc6ad94f8af2fac0602a31f48e84899168c4a8dd245cf24ac09430ec13-e3f84faf9e8077c06e0a94ca290c447e37d5cf3-mmode-mw uid: 625c837e-7279-49ff-b1b5-09216c3c7747 resourceVersion: "18537721" uid: 16e74788-1130-4377-854d-bd0155ada4e0 spec: modes: - Failover storageProvisioner: openshift-storage.rbd.csi.ceph.com targetID: e3f84faf9e8077c06e0a94ca290c447e37d5cf3 status: conditions: - lastTransitionTime: "2023-09-20T12:15:33Z" message: Maintenance(mode=Failover) of cluster has completed successfully observedGeneration: 1 reason: MaintenanceCompleted status: "True" type: FailoverActivated observedGeneration: 1 state: Completed kind: List metadata: resourceVersion: ""
It's a regression hence 4.14 blocker, and was already discussed in triage call. Hence providing qa_ack.
Please provide an ETA
The issue couldn't be reproduced and seems intermittent. Closing this bug for now and will re-open if needed.
Looking in the current ramen-hub log bz2239590-logs/logs/acm/must-gather.local.816384060008061347/quay-io-acm-d-acm-must-gather-rhel8-sha256-90468b98c401ffbd14c589de7472dc747a988e7e7ebaa9a95fbda20dec33f80e/namespaces/openshift-operators/pods/ramen-hub-operator-67657d944-dt82j/manager/manager/logs/current.log we see lot of errors accessing the (unavailable?) cluster at 172.30.0.1:443. ``` $ egrep 'rbd-sub-busybox-2.+timeout' current.log 2023-11-23T12:05:26.587128807Z 2023-11-23T12:05:26.587Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:26.587159632Z 2023-11-23T12:05:26.587Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "a52b640c-7c68-44e7-b34c-a471e92fa8f0"} 2023-11-23T12:05:26.621367562Z 2023-11-23T12:05:26.621Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:26.621399000Z 2023-11-23T12:05:26.621Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "70b7919e-0626-47f0-9444-65f1d82fcf2c"} 2023-11-23T12:05:26.656461330Z 2023-11-23T12:05:26.656Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:26.656499398Z 2023-11-23T12:05:26.656Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "52edf625-d44e-495e-9ea8-4bc8e1328a4d"} 2023-11-23T12:05:26.749522375Z 2023-11-23T12:05:26.749Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:26.749559670Z 2023-11-23T12:05:26.749Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "23797389-44ce-45e8-9b4b-200067300c52"} 2023-11-23T12:05:26.867013012Z 2023-11-23T12:05:26.866Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:26.867054002Z 2023-11-23T12:05:26.867Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "5022c405-f73c-496a-b661-6e2dd65aa06d"} 2023-11-23T12:05:26.997690894Z 2023-11-23T12:05:26.997Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:26.997719350Z 2023-11-23T12:05:26.997Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "3fc54317-5c03-436d-98df-309fe5235a3e"} 2023-11-23T12:05:27.183058208Z 2023-11-23T12:05:27.183Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:27.183087527Z 2023-11-23T12:05:27.183Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "fd36813f-9cce-4383-a22e-f9860f394ad7"} 2023-11-23T12:05:27.527246286Z 2023-11-23T12:05:27.527Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:27.527275069Z 2023-11-23T12:05:27.527Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "10f9d486-a1af-4eef-8a3a-a44f61ebe688"} 2023-11-23T12:05:28.191640726Z 2023-11-23T12:05:28.191Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:28.191674814Z 2023-11-23T12:05:28.191Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "eb958de7-5790-4cb5-ab4c-f383219166c9"} 2023-11-23T12:05:29.497793599Z 2023-11-23T12:05:29.497Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:29.497824124Z 2023-11-23T12:05:29.497Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "4e2e50cf-43df-428e-b401-e32abad19f35"} 2023-11-23T12:05:32.081328379Z 2023-11-23T12:05:32.081Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:32.081357268Z 2023-11-23T12:05:32.081Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "fda4ed53-a1d8-4bc9-90a6-0a795e64c248"} 2023-11-23T12:05:37.224323981Z 2023-11-23T12:05:37.224Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:37.224365464Z 2023-11-23T12:05:37.224Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "bf56c21b-917c-4d17-829b-305cbd0ac5b5"} 2023-11-23T12:05:47.486610023Z 2023-11-23T12:05:47.486Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:05:47.486700500Z 2023-11-23T12:05:47.486Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "ed87477f-0fe0-4e36-9886-ae33616c73ed"} 2023-11-23T12:06:07.988835693Z 2023-11-23T12:06:07.988Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:06:07.988887320Z 2023-11-23T12:06:07.988Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "d0e14684-1f93-4ff7-a956-c7198e05401e"} 2023-11-23T12:06:48.976752530Z 2023-11-23T12:06:48.976Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:06:48.976781314Z 2023-11-23T12:06:48.976Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "3b3f3783-e19a-4f08-8d23-0a3dc427f49b"} 2023-11-23T12:08:10.928203488Z 2023-11-23T12:08:10.928Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:10.928218319Z 2023-11-23T12:08:10.928Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "5fb42bef-5d88-4569-8a19-4cb6924e2b27"} 2023-11-23T12:08:50.631391817Z 2023-11-23T12:08:50.631Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:50.631420342Z 2023-11-23T12:08:50.631Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "c66e03c7-2e79-4ee6-bcf1-efdd3e0824a0"} 2023-11-23T12:08:50.664229454Z 2023-11-23T12:08:50.664Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:50.664247171Z 2023-11-23T12:08:50.664Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "ba8f8b5e-45d2-4580-af76-1df13f5046c7"} 2023-11-23T12:08:50.914590463Z 2023-11-23T12:08:50.914Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:50.914613558Z 2023-11-23T12:08:50.914Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "35b903e5-4122-4daf-8434-7c27b5b6b26b"} 2023-11-23T12:08:51.059260277Z 2023-11-23T12:08:51.059Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:51.059283328Z 2023-11-23T12:08:51.059Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "feb23543-2b72-408c-8bd8-0d0f6cca12bc"} 2023-11-23T12:08:51.204916266Z 2023-11-23T12:08:51.204Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:51.204938151Z 2023-11-23T12:08:51.204Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "76d2ea6a-2049-47c6-8d84-68d3df15d16a"} 2023-11-23T12:08:51.317038792Z 2023-11-23T12:08:51.317Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:51.317065590Z 2023-11-23T12:08:51.317Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "2b10f35d-ec3f-47bc-a4f4-85d29eb99422"} 2023-11-23T12:08:51.507003172Z 2023-11-23T12:08:51.506Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:51.507026417Z 2023-11-23T12:08:51.507Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "b4340356-47d3-4164-a81b-4a88ac6af8da"} 2023-11-23T12:08:51.582281872Z 2023-11-23T12:08:51.582Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:51.582309256Z 2023-11-23T12:08:51.582Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "f109acd7-95e7-4c89-a47f-ce4cc250ae29"} 2023-11-23T12:08:51.762495309Z 2023-11-23T12:08:51.762Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:51.762516858Z 2023-11-23T12:08:51.762Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "19397217-8470-41ea-9f1b-46f59c9b892d"} 2023-11-23T12:08:52.516396410Z 2023-11-23T12:08:52.516Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:52.516415995Z 2023-11-23T12:08:52.516Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "8c3a5a50-9e5a-4b5f-a1ab-275e5d675f60"} 2023-11-23T12:08:53.935026788Z 2023-11-23T12:08:53.935Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:53.935051336Z 2023-11-23T12:08:53.935Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "567fab8b-4e3c-436b-91d9-f93c4bfd4215"} 2023-11-23T12:08:56.568919299Z 2023-11-23T12:08:56.568Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:08:56.568956200Z 2023-11-23T12:08:56.568Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "34c1ff6b-6bc5-4eaf-a1ec-1e61320aba02"} 2023-11-23T12:09:01.862412189Z 2023-11-23T12:09:01.862Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:09:01.862431506Z 2023-11-23T12:09:01.862Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "6cdaac28-d8e3-4a39-ab7d-f89d20b1556c"} 2023-11-23T12:09:12.233210808Z 2023-11-23T12:09:12.233Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:09:12.233234713Z 2023-11-23T12:09:12.233Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "2fffd40f-9d00-4c80-98e5-835784d760f0"} 2023-11-23T12:09:20.053755585Z 2023-11-23T12:09:20.053Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:09:20.053773588Z 2023-11-23T12:09:20.053Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "8ca39e7e-cfef-469b-acad-2ea71dfb4984"} 2023-11-23T12:09:20.219008756Z 2023-11-23T12:09:20.218Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:09:20.219030545Z 2023-11-23T12:09:20.219Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "b85e26ac-f65a-4a86-84d4-7b7084795599"} 2023-11-23T12:09:32.847654306Z 2023-11-23T12:09:32.847Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:09:32.847669997Z 2023-11-23T12:09:32.847Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "e399391c-b52d-4153-982c-2e789ae630ce"} 2023-11-23T12:10:33.527694889Z 2023-11-23T12:10:33.527Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:10:33.527724449Z 2023-11-23T12:10:33.527Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "7cedb496-1d04-48c6-83fd-2c276505ec76"} 2023-11-23T12:10:40.921299733Z 2023-11-23T12:10:40.921Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:10:40.921319843Z 2023-11-23T12:10:40.921Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "dad72c63-fe77-451a-a34b-efa923fbc892"} 2023-11-23T12:10:45.228511247Z 2023-11-23T12:10:45.228Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:10:45.228530182Z 2023-11-23T12:10:45.228Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "88e3ae65-5652-450d-aa54-410f857c80ae"} 2023-11-23T12:10:50.920601851Z 2023-11-23T12:10:50.920Z ERROR controllers.DRPlacementControl controllers/drplacementcontrol.go:661 Kube objects capture-to-recover-from identifier get error {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "c66e03c7-2e79-4ee6-bcf1-efdd3e0824a0", "error": "failed to download data of odrbucket-4b139affc8ef:busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc/v1alpha1.VolumeReplicationGroup/a, RequestError: send request failed\ncaused by: Get \"https://s3-openshift-storage.apps.amagrawa-22n-1st.qe.rh-ocs.com/odrbucket-4b139affc8ef/busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc/v1alpha1.VolumeReplicationGroup/a\": dial tcp 10.1.114.49:443: i/o timeout"} 2023-11-23T12:10:50.946250474Z 2023-11-23T12:10:50.946Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:10:50.946310612Z 2023-11-23T12:10:50.946Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1622 Failed to get VRG from managed cluster {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "c66e03c7-2e79-4ee6-bcf1-efdd3e0824a0", "errMsg": "getManagedClusterResource results: err: Get \"https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc\": dial tcp 172.30.0.1:443: i/o timeout", "errMsgVerbose": "err: Get \"https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc\": dial tcp 172.30.0.1:443: i/o timeout\ngetManagedClusterResource results\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.GetResource\n\t/remote-source/app/controllers/util/mcv_util.go:278\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.getManagedClusterResource\n\t/remote-source/app/controllers/util/mcv_util.go:225\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.GetVRGFromManagedCluster\n\t/remote-source/app/controllers/util/mcv_util.go:81\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).updateResourceCondition\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:1619\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).updateDRPCStatus\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:1581\ngithub.com/ramendr/ramen/controllers.(*DRPCInstance).startProcessing\n\t/remote-source/app/controllers/drplacementcontrol.go:72\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).reconcileDRPCInstance\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:854\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).Reconcile\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:670\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598"} 2023-11-23T12:10:50.956178771Z 2023-11-23T12:10:50.956Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:10:50.956178771Z 2023-11-23T12:10:50.956Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "58689178-fd3a-45d5-9ec7-3bc19d50b349"} 2023-11-23T12:10:51.017554033Z 2023-11-23T12:10:51.017Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:10:51.017577086Z 2023-11-23T12:10:51.017Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "3633c6cd-ec76-4475-91c9-867890dfbfb8"} 2023-11-23T12:12:51.206727155Z 2023-11-23T12:12:51.206Z ERROR controllers.DRPlacementControl controllers/drplacementcontrol.go:661 Kube objects capture-to-recover-from identifier get error {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "58689178-fd3a-45d5-9ec7-3bc19d50b349", "error": "failed to download data of odrbucket-4b139affc8ef:busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc/v1alpha1.VolumeReplicationGroup/a, RequestError: send request failed\ncaused by: Get \"https://s3-openshift-storage.apps.amagrawa-22n-1st.qe.rh-ocs.com/odrbucket-4b139affc8ef/busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc/v1alpha1.VolumeReplicationGroup/a\": dial tcp 10.1.114.49:443: i/o timeout"} 2023-11-23T12:12:51.259597642Z 2023-11-23T12:12:51.259Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:12:51.259673798Z 2023-11-23T12:12:51.259Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1622 Failed to get VRG from managed cluster {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "58689178-fd3a-45d5-9ec7-3bc19d50b349", "errMsg": "getManagedClusterResource results: err: Get \"https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc\": dial tcp 172.30.0.1:443: i/o timeout", "errMsgVerbose": "err: Get \"https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc\": dial tcp 172.30.0.1:443: i/o timeout\ngetManagedClusterResource results\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.GetResource\n\t/remote-source/app/controllers/util/mcv_util.go:278\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.getManagedClusterResource\n\t/remote-source/app/controllers/util/mcv_util.go:225\ngithub.com/ramendr/ramen/controllers/util.ManagedClusterViewGetterImpl.GetVRGFromManagedCluster\n\t/remote-source/app/controllers/util/mcv_util.go:81\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).updateResourceCondition\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:1619\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).updateDRPCStatus\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:1581\ngithub.com/ramendr/ramen/controllers.(*DRPCInstance).startProcessing\n\t/remote-source/app/controllers/drplacementcontrol.go:72\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).reconcileDRPCInstance\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:854\ngithub.com/ramendr/ramen/controllers.(*DRPlacementControlReconciler).Reconcile\n\t/remote-source/app/controllers/drplacementcontrol_controller.go:670\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime.6/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/lib/golang/src/runtime/asm_amd64.s:1598"} 2023-11-23T12:12:51.269483341Z 2023-11-23T12:12:51.269Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:12:51.269519427Z 2023-11-23T12:12:51.269Z INFO controllers.DRPlacementControl controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"DRPC": "busybox-workloads-2/rbd-sub-busybox-2-placement-1-drpc", "rid": "803ad3f2-5030-456f-8228-09d5af08c687"} 2023-11-23T12:12:51.338485411Z 2023-11-23T12:12:51.338Z INFO MCV util/mcv_util.go:223 MCV Conditions: [{Processing False 0 2023-11-23 12:05:26 +0000 UTC GetResourceFailed failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout}] {"resourceName": "rbd-sub-busybox-2-placement-1-drpc"} 2023-11-23T12:12:51.338510745Z 2023-11-23T12:12:51.338Z INFO controllers.DRCluster controllers/drplacementcontrol_controller.go:1471 failed to retrieve VRG from amagrawa-22n-1st. err (getManagedClusterResource results: err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-drpc": dial tcp 172.30.0.1:443: i/o timeout) {"name": "amagrawa-22n-2nd", "rid": "db6a2dec-2b45-4ef9-b313-cd08cb4cd9d9"} ``` Maybe the issue is related to failure to access the unavailable cluster cause skip to update the drpc status at the first phase (Initiating). Need deeper look, following the entire failover flow for this application (and maybe other).
Based the drpc output, the failover started on 12:10:30: Thursday 23 November 2023 12:10:27 PM UTC *********************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 rbd-sub-busybox-2-placement-1-drpc 49m amagrawa-22n-1st amagrawa-22n-2nd Failover Deployed Completed True Thursday 23 November 2023 12:10:30 PM UTC *********************** NAMESPACE NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY busybox-workloads-2 rbd-sub-busybox-2-placement-1-drpc 49m amagrawa-22n-1st amagrawa-22n-2nd Failover Deployed Completed True Aman reports that the cluster was shutdown 5-7 minutes before the failover. The timeouts come from the VRG MCV: bz2239590-logs/logs/acm/must-gather.local.816384060008061347/quay-io-acm-d-acm-must-gather-rhel8-sha256-90468b98c401ffbd14c589de7472dc747a988e7e7ebaa9a95fbda20dec33f80e/namespaces/amagrawa-22n-1st/view.open-cluster-management.io/managedclusterviews/rbd-sub-busybox-2-placement-1-drpc-busybox-workloads-2-vrg-mcv.yaml status: conditions: - lastTransitionTime: "2023-11-23T12:05:26Z" message: 'failed to get resource with err: Get "https://172.30.0.1:443/apis/ramendr.openshift.io/v1alpha1/namespaces/busybox-workloads-2/volumereplicationgroups/rbd-sub-busybox-2-placement-1-dr pc": dial tcp 172.30.0.1:443: i/o timeout' reason: GetResourceFailed status: "False" type: Processing result: apiVersion: ramendr.openshift.io/v1alpha1 kind: VolumeReplicationGroup ... status: conditions: - lastTransitionTime: "2023-11-23T11:21:22Z" message: PVCs in the VolumeReplicationGroup are ready for use observedGeneration: 1 reason: Ready status: "True" type: DataReady - lastTransitionTime: "2023-11-23T11:21:11Z" message: VolumeReplicationGroup is replicating observedGeneration: 1 reason: Replicating status: "False" type: DataProtected - lastTransitionTime: "2023-11-23T11:21:09Z" message: Restored cluster data observedGeneration: 1 reason: Restored status: "True" type: ClusterDataReady - lastTransitionTime: "2023-11-23T11:52:32Z" message: Cluster data of all PVs are protected observedGeneration: 1 reason: Uploaded status: "True" type: ClusterDataProtected Aman - it will help to get same logs from another run of this test that *did not* show this issue.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.15.0 security, enhancement, & bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:1383