Bug 1820180
| Summary: | The deleted PrometheusRule is still in the thanos-ruler UI | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Junqi Zhao <juzhao> | ||||
| Component: | Monitoring | Assignee: | Lili Cosic <lcosic> | ||||
| Status: | CLOSED UPSTREAM | QA Contact: | Junqi Zhao <juzhao> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 4.5 | CC: | alegrand, anpicker, erooth, kakkoyun, lcosic, mloibl, pkrupa, surbania | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.6.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2020-05-18 11:24:41 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Created attachment 1675728 [details] still see the deleted PrometheusRule in thanos-ruler UI Description of problem: The deleted PrometheusRule is still in the thanos-ruler UI Steps: enable techPreviewUserWorkload and do the followings to create PrometheusRule *********************************************** # oc new-project test3 # oc create -f - << EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: test3.rules spec: groups: - name: alerting rules rules: - alert: Watchdog expr: vector(1) labels: severity: none message: This is an alert meant to ensure that the entire alerting pipeline is functional. EOF *********************************************** The PrometheusRule is in rules-configmap-reloader container # oc -n openshift-user-workload-monitoring exec -c rules-configmap-reloader thanos-ruler-user-workload-0 -- cat /etc/thanos/rules/thanos-ruler-user-workload-rulefiles-0/test3-test3.rules.yaml groups: - name: alerting rules rules: - alert: Watchdog expr: vector(1) labels: namespace: test3 severity: none # oc -n openshift-user-workload-monitoring exec -c rules-configmap-reloader thanos-ruler-user-workload-0 -- ls -al /etc/thanos/rules/thanos-ruler-user-workload-rulefiles-0/test3-test3.rules.yaml lrwxrwxrwx. 1 root 1000420000 29 Apr 2 12:04 /etc/thanos/rules/thanos-ruler-user-workload-rulefiles-0/test3-test3.rules.yaml -> ..data/test3-test3.rules.yaml delete PrometheusRule in test3 # oc delete project test3 project.project.openshift.io "test3" deleted # oc -n test3 get PrometheusRule No resources found in test3 namespace. # oc -n openshift-user-workload-monitoring logs -c rules-configmap-reloader thanos-ruler-user-workload-0 2020/04/02 12:01:58 Watching directory: "/etc/thanos/rules/thanos-ruler-user-workload-rulefiles-0" 2020/04/02 12:04:22 config map updated 2020/04/02 12:04:22 successfully triggered reload 2020/04/02 12:22:24 config map updated 2020/04/02 12:22:24 successfully triggered reload The PrometheusRule is also removed from rules-configmap-reloader container # oc -n openshift-user-workload-monitoring exec -c rules-configmap-reloader thanos-ruler-user-workload-0 -- cat /etc/thanos/rules/thanos-ruler-user-workload-rulefiles-0/test3-test3.rules.yaml cat: /etc/thanos/rules/thanos-ruler-user-workload-rulefiles-0/test3-test3.rules.yaml: No such file or directory command terminated with exit code 1 Check after a few minutes,the alert still can be found in thanos-ruler UI, see the picture. Version-Release number of selected component (if applicable): 4.5.0-0.nightly-2020-04-01-232323 How reproducible: Always Steps to Reproduce: 1. See the description 2. 3. Actual results: Expected results: Additional info: