Bug 1845642

Summary: image prunring alert breaks 4.2->4.5 upgrade test
Product: OpenShift Container Platform Reporter: Ben Parees <bparees>
Component: Image RegistryAssignee: Oleg Bulatov <obulatov>
Status: CLOSED ERRATA QA Contact: Wenjing Zheng <wzheng>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.5CC: aos-bugs, pasik, scuppett
Target Milestone: ---Keywords: Upgrades
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: there was an alert when the image pruner is disabled Consequence: on upgraded clusters this alert was firing Fix: remove the alert Result: no alerts after upgrades
 Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-27 16:06:02 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1846336    

Description Ben Parees 2020-06-09 17:49:41 UTC 
Description of problem:
See failures in https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.2-to-4.3-to-4.4-to-4.5-ci/91

The alert is unexpected:
[sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] [Suite:openshift/conformance/parallel] expand_less 	1m50s
fail [github.com/openshift/origin/test/extended/util/prometheus/helpers.go:174]: Expected
    <map[string]error | len:1>: {
        "ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1": {
            s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\",severity!=\"info\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"ImagePruningDisabled\",\"alertstate\":\"firing\",\"endpoint\":\"60000\",\"instance\":\"10.128.0.17:60000\",\"job\":\"image-registry-operator\",\"namespace\":\"openshift-image-registry\",\"pod\":\"cluster-image-registry-operator-d4f4bbd45-2dp9x\",\"service\":\"image-registry-operator\",\"severity\":\"warning\"},\"value\":[1591601588.198,\"1\"]}]",
        },
    }
to be empty

This seems to be causing https://testgrid.k8s.io/redhat-openshift-ocp-release-4.5-informing#release-openshift-origin-installer-e2e-aws-upgrade-4.2-to-4.3-to-4.4-to-4.5-ci to be entirely red.

(there may be other failing tests, but this is one that needs to be cleaned up)
Comment 3 Wenjing Zheng 2020-06-15 02:54:40 UTC 
https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-upgrade-4.3-to-4.4-to-4.5-to-4.6-ci/1272299115971088384 

[sig-instrumentation] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early] [Suite:openshift/conformance/parallel] is passed in above test.
Comment 5 errata-xmlrpc 2020-10-27 16:06:02 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196