Bug 1845642 - image prunring alert breaks 4.2->4.5 upgrade test
Summary: image prunring alert breaks 4.2->4.5 upgrade test
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image Registry
Version: 4.5
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.6.0
Assignee: Oleg Bulatov
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks: 1846336
TreeView+ depends on / blocked
 
Reported: 2020-06-09 17:49 UTC by Ben Parees
Modified: 2020-10-27 16:06 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: there was an alert when the image pruner is disabled Consequence: on upgraded clusters this alert was firing Fix: remove the alert Result: no alerts after upgrades
Clone Of:
Environment:
Last Closed: 2020-10-27 16:06:02 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-image-registry-operator pull 568 0 None closed Bug 1845642: Remove alert about image pruner 2020-12-18 16:07:42 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:06:19 UTC

Description Ben Parees 2020-06-09 17:49:41 UTC
Description of problem:
See failures in https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.2-to-4.3-to-4.4-to-4.5-ci/91

The alert is unexpected:
[sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] [Suite:openshift/conformance/parallel] expand_less 	1m50s
fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1": {
            s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"ImagePruningDisabled\",\"alertstate\":\"firing\",\"endpoint\":\"60000\",\"instance\":\"10.128.0.17:60000\",\"job\":\"image-registry-operator\",\"namespace\":\"openshift-image-registry\",\"pod\":\"cluster-image-registry-operator-d4f4bbd45-2dp9x\",\"service\":\"image-registry-operator\",\"severity\":\"warning\"},\"value\":[1591601588.198,\"1\"]}]",
        },
    }
to be empty

This seems to be causing https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-informing#release-openshift-origin-installer-e2e-aws-upgrade-4.2-to-4.3-to-4.4-to-4.5-ci to be entirely red.

(there may be other failing tests, but this is one that needs to be cleaned up)

Comment 3 Wenjing Zheng 2020-06-15 02:54:40 UTC
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.3-to-4.4-to-4.5-to-4.6-ci/1272299115971088384 

[sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] [Suite:openshift/conformance/parallel] is passed in above test.

Comment 5 errata-xmlrpc 2020-10-27 16:06:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.