2291305 – [Tracker] [RDR][Discovered Apps] drpc status is stuck in WaitForReadiness after performing failover operation

Bug 2291305 - [Tracker] [RDR][Discovered Apps] drpc status is stuck in WaitForReadiness after performing failover operation

Summary: [Tracker] [RDR][Discovered Apps] drpc status is stuck in WaitForReadiness aft...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	odf-dr
Sub Component:
Version:	4.16
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	---
Target Release:	ODF 4.16.0
Assignee:	Shyamsundar
QA Contact:	Sidhant Agrawal
Docs Contact:
URL:
Whiteboard:	verification-blocked
Duplicates (1):	2291126 (view as bug list)
Depends On:	2293634
Blocks:
TreeView+	depends on / blocked

Reported:	2024-06-11 12:28 UTC by Pratik Surve
Modified:	2024-07-17 13:24 UTC (History)
CC List:	7 users (show)
Fixed In Version:	4.16.0-127
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2024-07-17 13:24:48 UTC
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	red-hat-storage ramen pull 303	None	open	Bug 2291305: velero: backupStorageLocation requires config.checksumAlgorithm	2024-06-14 05:39:09 UTC
Red Hat Issue Tracker	OADP-4075	None	None	None	2024-06-12 12:34:36 UTC
Red Hat Issue Tracker	OADP-4274	None	None	None	2024-06-12 12:34:36 UTC
Red Hat Product Errata	RHSA-2024:4591	None	None	None	2024-07-17 13:24:49 UTC

Description Pratik Surve 2024-06-11 12:28:09 UTC

Description of problem (please be detailed as possible and provide log
snippests):

[RDR][Discovered Apps] drpc status is stuck in WaitForReadiness after performing failover operation

Version of all relevant components (if applicable):


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
OCP version:- 4.16.0-0.nightly-2024-06-03-060250
ODF version:- 4.16.0-125
CEPH version:- ceph version 18.2.1-194.el9cp (04a992766839cd3207877e518a1238cdbac3787e) reef (stable)
ACM version:- 2.11.0-102
SUBMARINER version:- v0.18.0
VOLSYNC version:- volsync-product.v0.9.1
VOLSYNC method:- destinationCopyMethod: Direct

Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy RDR cluster
2.Deploy RBD imperative app workload
3.When dr Protecting use same label present on deployment and pod.


Actual results:

Failed to restore PVs/PVCs: failed to restore PV/PVC for VolRep (failed to restore PVs and PVCs using profile list ([s3profile-prsurve-c1-ocs-storagecluster s3profile-prsurve-vm-d-ocs-storagecluster]): restoreInProgress)
    Observed Generation:   1
    Reason:                Error
    Status:                False
    Type:                  ClusterDataReady
    Last Transition Time:  2024-06-10T11:50:34Z
    Message:               Initializing VolumeReplicationGroup
    Observed Generation:   1
    Reason:                Initializing
    Status:                Unknown
    Type:                  ClusterDataProtected



"No BackupStorageLocations found, at least one is required" backup-storage-location=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0----s3profile-prsurve-c1-ocs-storagecluster controller=backup-storage-location error="no backup storage locations found" error.file="/remote-source/velero/app/internal/storage/storagelocation.go:90" error.function=github.com/vmware-tanzu/velero/internal/storage.ListBackupStorageLocations logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:92"
time="2024-06-10T13:49:29Z" level=error msg="backup failed" backuprequest=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0----s3profile-prsurve-c1-ocs-storagecluster controller=backup error="backup already exists in object storage" error.file="/remote-source/velero/app/pkg/controller/backup_controller.go:652" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupReconciler).runBackup" logSource="/remote-source/velero/app/pkg/controller/backup_controller.go:292"
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-1-679794bd89-6gq8m: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-10-84b4f6c5d-gzjnp: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-11-6444959676-5mqd2: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-12-86b7f56dbb-2crwr: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-13-57fd76f76d-vl4mb: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-14-c6b76f474-77d24: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-15-6d58dc648d-wrtzr: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-16-6795f74b86-trzhg: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-17-75d87bc964-4f7tl: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-18-54fbb778-4xcks: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-19-6cb7556d8d-kxw8k: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-2-6bff6c45b6-rkzgf: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-20-78b899674d-lsz4j: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-3-bb976f794-khz9n: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-4-548f9bdf8b-tmssn: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-5-dfcfb8b4b-dchrj: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-6-5f98777b86-wgsvg: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-7-6479489cbb-fz5cl: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-8-68f555cf58-qnz2n: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0
time="2024-06-10T13:49:49Z" level=error msg="Namespace app-rbd-1-imp, resource restore error: error preparing pods/app-rbd-1-imp/busybox-9-79667f7c54-4r4rd: rpc error: code = Unknown desc = resource name may not be empty" logSource="/remote-source/velero/app/pkg/controller/restore_controller.go:560" restore=openshift-adp/openshift-dr-ops--app-rbd-1-imp--0


pods -n app-rbd-1-imp --show-labels
NAME                          READY   STATUS    RESTARTS   AGE   LABELS
busybox-1-679794bd89-6gq8m    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=679794bd89,workload=busybox-1
busybox-10-84b4f6c5d-gzjnp    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=84b4f6c5d,workload=busybox-10
busybox-11-6444959676-5mqd2   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=6444959676,workload=busybox-11
busybox-12-86b7f56dbb-2crwr   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=86b7f56dbb,workload=busybox-12
busybox-13-57fd76f76d-vl4mb   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=57fd76f76d,workload=busybox-13
busybox-14-c6b76f474-77d24    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=c6b76f474,workload=busybox-14
busybox-15-6d58dc648d-wrtzr   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=6d58dc648d,workload=busybox-15
busybox-16-6795f74b86-trzhg   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=6795f74b86,workload=busybox-16
busybox-17-75d87bc964-4f7tl   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=75d87bc964,workload=busybox-17
busybox-18-54fbb778-4xcks     1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=54fbb778,workload=busybox-18
busybox-19-6cb7556d8d-kxw8k   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=6cb7556d8d,workload=busybox-19
busybox-2-6bff6c45b6-rkzgf    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=6bff6c45b6,workload=busybox-2
busybox-20-78b899674d-lsz4j   1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=78b899674d,workload=busybox-20
busybox-3-bb976f794-khz9n     1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=bb976f794,workload=busybox-3
busybox-4-548f9bdf8b-tmssn    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=548f9bdf8b,workload=busybox-4
busybox-5-dfcfb8b4b-dchrj     1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=dfcfb8b4b,workload=busybox-5
busybox-6-5f98777b86-wgsvg    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=5f98777b86,workload=busybox-6
busybox-7-6479489cbb-fz5cl    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=6479489cbb,workload=busybox-7
busybox-8-68f555cf58-qnz2n    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=68f555cf58,workload=busybox-8
busybox-9-79667f7c54-4r4rd    1/1     Running   0          25h   app-name=app-busybox-rbd-1,pod-template-hash=79667f7c54,workload=busybox-9




oc get deploy --show-labels -n app-rbd-1-imp
NAME         READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
busybox-1    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-1,workloadpattern=simple_io
busybox-10   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-10,workloadpattern=simple_io
busybox-11   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-11,workloadpattern=simple_io
busybox-12   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-12,workloadpattern=simple_io
busybox-13   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-13,workloadpattern=simple_io
busybox-14   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-14,workloadpattern=simple_io
busybox-15   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-15,workloadpattern=simple_io
busybox-16   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-16,workloadpattern=simple_io
busybox-17   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-17,workloadpattern=simple_io
busybox-18   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-18,workloadpattern=simple_io
busybox-19   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-19,workloadpattern=simple_io
busybox-2    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-2,workloadpattern=simple_io
busybox-20   1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-20,workloadpattern=simple_io
busybox-3    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-3,workloadpattern=simple_io
busybox-4    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-4,workloadpattern=simple_io
busybox-5    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-5,workloadpattern=simple_io
busybox-6    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-6,workloadpattern=simple_io
busybox-7    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-7,workloadpattern=simple_io
busybox-8    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-8,workloadpattern=simple_io
busybox-9    1/1     1            1           25h   app-name=app-busybox-rbd-1,appname=busybox_app1-9,workloadpattern=simple_io


Expected results:


Additional info:

Comment 12 wes hayutin 2024-06-12 17:37:16 UTC

Couple notes from the OADP team :) 0/
ONLY OADP-1.4 will be supported on OCP 4.16
* https://issues.redhat.com/browse/OADP-4075 was found with OADP-1.3x, and should be resolved in the latest 1.4.0-x builds
* https://issues.redhat.com/browse/OADP-4274 is ONLY found in OADP-1.4.0-x and the patch to fix is landing now https://github.com/openshift/oadp-operator/pull/1424
   * This should be in our next build of 1.4.0, I've added folks from this bug to the oadp-team email list where we publish the latest build information.

Thanks all 0/

Comment 17 Sunil Kumar Acharya 2024-06-18 06:45:26 UTC

Please update the RDT flag/text appropriately.

Comment 21 errata-xmlrpc 2024-07-17 13:24:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591

Note You need to log in before you can comment on or make changes to this bug.