Bug 2291298 - PrometheusRuleFailures alert with rule_group /etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-storage-prometheus-ceph-rules-* after change of in-transit encryption configuration
Summary: PrometheusRuleFailures alert with rule_group /etc/prometheus/rules/prometheus...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: ceph-monitoring
Version: 4.16
Hardware: Unspecified
OS: Unspecified
unspecified
low
Target Milestone: ---
: ---
Assignee: arun kumar mohan
QA Contact: Harish NV Rao
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-06-11 11:06 UTC by Filip Balák
Modified: 2024-08-30 11:25 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-08-30 11:25:31 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 2262943 0 unspecified CLOSED PrometheusRule evaluation failing for pool-quota.rules 2024-07-17 13:13:31 UTC
Red Hat Bugzilla 2266316 0 unspecified CLOSED PrometheusRuleFailures alert after installation or upgrade 2024-07-17 13:14:30 UTC

Description Filip Balák 2024-06-11 11:06:04 UTC
Description of problem (please be detailed as possible and provide log
snippests):
After following steps are executed on cluster with set in-transit encryption:
        1. Enable in-transit Encryption if not Enabled.
        2. Verify in-transit Encryption is Enable on setup.
        3. Disable Encryption
        4. Verify in-transit encryption configuration is removed.
        5. Enable encryption Again and verify it.
        6. Verify in-transit encryption config is exists.

There is a similar alert for some time:
{'labels': {'alertname': 'PrometheusRuleFailures', 'container': 'kube-rbac-proxy', 'endpoint': 'metrics', 'instance': '10.131.0.31:9092', 'job': 'prometheus-k8s', 'namespace': 'openshift-monitoring', 'pod': 'prometheus-k8s-0', 'rule_group': '/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-storage-prometheus-ceph-rules-737a4da2-7bec-4c21-a113-6e3b8ac91d83.yaml;telemeter.rules', 'service': 'prometheus-k8s', 'severity': 'warning'}, 'annotations': {'description': 'Prometheus openshift-monitoring/prometheus-k8s-0 has failed to evaluate 4 rules in the last 5m.', 'runbook_url': 'https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusRuleFailures.md', 'summary': 'Prometheus is failing rule evaluations.'}, 'state': 'pending', 'activeAt': '2024-06-01T02:34:11.604846708Z', 'value': '4.130400000000001e+00'},

Version of all relevant components (if applicable):
ODF 4.16.0-22

Can this issue reproducible?
yes

Steps to Reproduce:
1. Enable in-transit Encryption if not Enabled.
2. Verify in-transit Encryption is Enable on setup.
3. Disable Encryption
4. Verify in-transit encryption configuration is removed.
5. Enable encryption Again and verify it.
6. Verify in-transit encryption config is exists.
7. Check alerts

Actual results:
There is an alert PrometheusRuleFailures

Expected results:
There is no PrometheusRuleFailures alert

Additional info:
This is automated by test case test_intransit_encryption_enable_disable_statetransition (https://github.com/red-hat-storage/ocs-ci/blob/0cf981385c50d65b29201505965b6e31faf672a5/tests/functional/encryption/test_intransit_encryption_sanity.py#L38)


Note You need to log in before you can comment on or make changes to this bug.