Bug 2096604
| Summary: | [RDR] All the 5 DRPC shows an error message and doesn't overcome it while DR Policy does | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Aman Agrawal <amagrawa> |
| Component: | odf-dr | Assignee: | Shyamsundar <srangana> |
| odf-dr sub component: | ramen | QA Contact: | krishnaram Karthick <kramdoss> |
| Status: | CLOSED WONTFIX | Docs Contact: | |
| Severity: | low | ||
| Priority: | unspecified | CC: | bmekhiss, madam, muagarwa, ocs-bugs, odf-bz-bot |
| Version: | 4.10 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-07-18 20:33:40 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
If you Failover or a relocation now, it should succeed and the status will be corrected. |
Here is what happened: 1. Initial Deployment succeeded. DRPolicy was happy as well as DRPC 2. At `2022-06-12T23:16:34.442Z` DRPolicy started complaining about the s3Store. Specifically ``` 2022-06-12T23:16:34.442Z INFO controllers.drpolicy controllers/drpolicy_controller.go:352 condition update {"name": "odr-policy-10m", "type": "Validated", "old status": "True", "new status": "False", "old reason": "Succeeded", "new reason": "s3ListFailed", "old message": "drpolicy validated", "new message": "s3profile-amagrawa-c1-ocs-storagecluster: failed to list object s in bucket odrbucket-1224929aa43a:/odr-policy-10m, SerializationError: failed to unmarshal error message\n\tstatus code: 503, request id: , host id: \ncaused by: UnmarshalError: failed to unmarsha l error message\n\t00000000 3c 68 74 6d 6c 3e 0d 0a 20 20 3c 68 65 61 64 3e |<html>.. <head>|\n00000010 0d 0a 20 20 20 20 3c 6d 65 74 61 20 6e 61 6d 65 ``` 3. It lasted for 1 minute. Then it switched to this error: ``` 2022-06-12T23:17:36.716Z ERROR controller.drpolicy controller/controller.go:214 Reconciler error {"reconciler group": "ramendr.openshift.io", "reconciler kind": "DRPolicy", " name": "odr-policy-10m", "namespace": "", "error": "validate: s3profile-amagrawa-c1-ocs-storagecluster: failed to list objects in bucket odrbucket-1224929aa43a:/odr-policy-10m, InternalError: We en countered an internal error. Please try again.\n\tstatus code: 500, request id: l4bxfk8m-57xq87-15lr, host id: l4bxfk8m-57xq87-15lr"} ``` 4. That last error lasted for 1m14s, at which the DRPolicy became valid again: ``` 2022-06-12T23:18:50.202Z INFO controllers.drpolicy controllers/drpolicy_controller.go:348 condition update {"name": "odr-policy-10m", "type": "Validated", "old status": "False", "new status": "True", "old reason": "s3ListFailed", "new reason": "Succeeded", "old message": "s3profile-amagrawa-c2-ocs-storagecluster: failed to list objects in bucket odrbucket-1224929aa43a:/odr-policy-10m, InternalError: We encountered an internal error. Please try again.\n\tstatus code: 500, request id: l4bxh2w0-3xhxfn-qcl, host id: l4bxh2w0-3xhxfn-qcl", "new message": "drpolicy validated", "old generation": 2, "new generation": 2} ```