Bug 2068387
| Summary: | Observability - observability operator always CrashLoopBackOff in FIPS upgrading hub | ||
|---|---|---|---|
| Product: | Red Hat Advanced Cluster Management for Kubernetes | Reporter: | cqu |
| Component: | Core Services / Observability | Assignee: | Chunlin Yang <chuyang> |
| Status: | CLOSED ERRATA | QA Contact: | Xiang Yin <xiyin> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhacm-2.5 | CC: | austinpatrick711, cqu, juhsu, jwakely, miguelwatsonfv99312, phanthanhhaikdb33, smeduri, thanhbachle2766 |
| Target Milestone: | --- | Flags: | bot-tracker-sync:
rhacm-2.5+
|
| Target Release: | rhacm-2.5 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-09 02:10:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
G2Bsync 1079797397 comment marcolan018 Sun, 27 Mar 2022 00:01:36 UTC G2Bsync there is a problem to handle the crd resource comparison before, that didn't consider different version for the same kind, fixed it. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:4956 This comment was flagged a spam, view the edit history to see the original text if required. This comment was flagged a spam, view the edit history to see the original text if required. This comment was flagged a spam, view the edit history to see the original text if required. This comment was flagged a spam, view the edit history to see the original text if required. Thanks for the information.. https://www.mybkexperience.one/ Bug 2068387 is a specific issue related to the observability operator in the context of FIPS (Federal Information Processing Standards) upgrading hub. The problem described in the bug report is that the observability operator consistently goes into a CrashLoopBackOff state during the FIPS upgrade process. A CrashLoopBackOff state indicates that the pod running the observability operator keeps crashing and restarting in a loop, unable to successfully run and stabilize. This issue is specifically observed when upgrading the FIPS (a set of security standards) on the hub. https://www-prepaidgiftbalance.com/ The bug report suggests that the observability operator is not functioning as expected in the FIPS upgrade scenario, leading to the continuous crash and restart cycle. The details of the specific cause or resolution for this bug are not provided in the short description. Such a fascinating article on a lovely day like today; thank you for your efforts in creating such a beneficial article. https://www.survey.onl/sakfcsurvey/ |
Description of the problem: On one FIPS upgrading hub, observability operator pods always CrashLoopBackOff, this issue only be found on this hub. On others hub with the same DS image without this issue. THe operator pod contains error: ``` goroutine 1783 [running]: github.com/stolostron/multicluster-observability-operator/operators/pkg/util.compareCRD({0x20a1e28, 0xc002fc2a00}, {0x20a1ea0, 0xc001235600}) /remote-source/app/operators/pkg/util/obj_compare.go:211 +0x245 github.com/stolostron/multicluster-observability-operator/operators/pkg/util.CompareObject({{0xc002eea000, 0xe1d, 0xe1d}, {0x0, 0x0}}, {{0x0, 0x0, 0x0}, {0x20a1ea0, 0xc001235600}}) /remote-source/app/operators/pkg/util/obj_compare.go:80 +0x362 github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.createManifestwork({0x20f6b28, 0xc0000a5630}, 0xc001d9d080) /remote-source/app/operators/multiclusterobservability/controllers/placementrule/manifestwork.go:170 +0x94f github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.createManifestWorks({0x20f6b28, 0xc0000a5630}, {0xc0002ffa80, 0x1e34e0a}, {0xc000e94314, 0xc0023e5c20}, {0xc000e94314, 0x6}, 0xc00279c000, {0xc001509680, ...}, ...) /remote-source/app/operators/multiclusterobservability/controllers/placementrule/manifestwork.go:359 +0x1209 github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.createManagedClusterRes({0x20f6b28, 0xc0000a5630}, {0x20e49a0, 0xc000336640}, 0x1, {0xc000e94314, 0xa}, {0xc000e94314, 0xa}, {0xc001509680, ...}, ...) /remote-source/app/operators/multiclusterobservability/controllers/placementrule/placementrule_controller.go:430 +0x16f github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.createAllRelatedRes({0x20f6b28, 0xc0000a5630}, {0x20e49a0, 0xc000336640}, {{{0x0, 0x2117fc8}, {0x1dec384, 0xc000116210}}}, 0xc001944490, 0xc003250c40, ...) /remote-source/app/operators/multiclusterobservability/controllers/placementrule/placementrule_controller.go:342 +0x959 github.com/stolostron/multicluster-observability-operator/operators/multiclusterobservability/controllers/placementrule.(*PlacementRuleReconciler).Reconcile(0xc00184db40, {0xc001334e10, 0x1b112a0}, {{{0x0, 0x1c73cc0}, {0x1dec384, 0xc00039dd80}}}) /remote-source/app/operators/multiclusterobservability/controllers/placementrule/placementrule_controller.go:157 +0x74c sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc000b31b80, {0x20bd958, 0xc001334e10}, {{{0x0, 0x1c73cc0}, {0x1dec384, 0xc002776fc0}}}) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114 +0x222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b31b80, {0x20bd8b0, 0xc000762000}, {0x1bc7400, 0xc001a81720}) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311 +0x2f2 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b31b80, {0x20bd8b0, 0xc000762000}) /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x205 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x85 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x354 ``` Release version: 2.5.0-DOWNSTREAM-2022-03-23-01-06-05 Operator snapshot version: OCP version: AWS FIPs enabled OCP 4.9.23 Browser Info: Steps to reproduce: 1. Deployed ACM/Observability 2.4 2. Upgrade to 2.5 with 2.5.0-DOWNSTREAM-2022-03-23-01-06-05 3. Observability operator is CrashLoopBackOff 4. hub env info - https://docs.google.com/spreadsheets/d/1yg75xNpeO1i_K39D43FZmMauQe2d2OuJ1UjunX7jz3A/edit?pli=1&skip_itp2_check=true#gid=1907824342&range=B6 Actual results: Expected results: Additional info: