Bug 2112394
| Summary: | NodeHasIntegrityFailure alert doesn't set namespace label after upgrade file-integrity-operator from v0.1.24>v0.1.30 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | xiyuan |
| Component: | File Integrity Operator | Assignee: | Matt Rogers <mrogers> |
| Status: | CLOSED ERRATA | QA Contact: | xiyuan |
| Severity: | medium | Docs Contact: | Jeana Routh <jrouth> |
| Priority: | medium | ||
| Version: | 4.11 | CC: | jhrozek, lbragsta, wenshen |
| Target Milestone: | --- | ||
| Target Release: | 4.12.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
* Previously, the File Integrity Operator did not properly handle modifying alerts during an upgrade. As a result, alerts did not include the namespace in which the Operator was installed. With this release, the Operator includes the namespace it was installed into in the alert, making it easier to narrow down what component needs attention.
(link:https://bugzilla.redhat.com/show_bug.cgi?id=2112394[*2112394*])
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-11-09 22:27:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
xiyuan
2022-07-29 15:10:38 UTC
The monitoring Rule isn't updated if it has been created by the operator previously: https://github.com/openshift/file-integrity-operator/blob/master/cmd/manager/operator.go#L295 So in this situation, the namespace addition from 0.1.27 is not applied. Deleting the Rule and restarting the operator pod would be a workaround. verification pass with file-integrity-operator.v0.1.31 + 4.12.0-0.nightly-2022-10-20-104328
Tried to verify with below two scenarios:
1. install File Integrity Operator v0.1.24
2. Create fileintegrity and make NodeHasIntegrityFailure fire from a node
3. Upgrade to File-integrity-operator.v0.1.31, retrigger a NodeHasIntegrityFailure, and inspect the alert lables. NodeHasIntegrityFailure has namespace label.
$ oc get csv
NAME DISPLAY VERSION REPLACES PHASE
file-integrity-operator.v0.1.31 File Integrity Operator 0.1.31 file-integrity-operator.v0.1.30 Succeeded
$ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring -o jsonpath='{@.spec.host}')
$ curl -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)" https://$ALERT_MANAGER/api/v1/alerts | jq -r
{
"status": "success",
"data": [
...
{
"labels": {
"alertname": "NodeHasIntegrityFailure",
"namespace": "openshift-file-integrity",
"node": "xiyuan21-1-6xq7z-master-1.c.openshift-qe.internal",
"openshift_io_alert_source": "platform",
"prometheus": "openshift-monitoring/k8s",
"severity": "warning"
},
"annotations": {
"description": "Node xiyuan21-1-6xq7z-master-1.c.openshift-qe.internal has an integrity check status of Failed for more than 1 second.",
"summary": "Node xiyuan21-1-6xq7z-master-1.c.openshift-qe.internal has a file integrity failure"
},
"startsAt": "2022-10-21T09:14:27.768Z",
"endsAt": "2022-10-21T11:24:27.768Z",
"generatorURL": "https:/console-openshift-console.apps.xiyuan21-1.qe.gcp.devcluster.openshift.com/monitoring/graph?g0.expr=file_integrity_operator_node_failed%7Bnode%3D~%22.%2B%22%7D+%2A+on+%28node%29+kube_node_info+%3E+0&g0.tab=1",
"status": {
"state": "active",
"silencedBy": null,
"inhibitedBy": null
},
"receivers": [
"Default"
],
"fingerprint": "2c4439dd0333d79d"
},
...
Per https://bugzilla.redhat.com/show_bug.cgi?id=2112394#c3 and https://bugzilla.redhat.com/show_bug.cgi?id=2112394#c7, move it to verified. Update one typo error in https://bugzilla.redhat.com/show_bug.cgi?id=2112394#c7 in step1: should be: 1. install File Integrity Operator v0.1.30 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift File Integrity Operator bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:7095 |