Bug 1796739 - [flake] olm-operator firing prometheus alerts about etcdoperator
Summary: [flake] olm-operator firing prometheus alerts about etcdoperator
Keywords:
Status: CLOSED DUPLICATE of bug 1790825
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Evan Cordell
QA Contact: Jian Zhang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-31 05:12 UTC by Dan Williams
Modified: 2020-01-31 13:45 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-01-31 13:45:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Dan Williams 2020-01-31 05:12:50 UTC 
failed: (1m30s) 2020-01-31T04:39:21 "[Feature:Prometheus][Conformance] Prometheus when installed on the cluster [Top Level] [Feature:Prometheus][Conformance] Prometheus when installed on the cluster shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Suite:openshift/conformance/parallel/minimal]"


"ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\"} >= 1": {
            s: "promQL query: ALERTS{alertname!~\"Watchdog|AlertmanagerReceiversNotConfigured|PrometheusRemoteWriteDesiredShards\",alertstate=\"firing\"} >= 1 had reported incorrect results:\n[{\"metric\":{\"__name__\":\"ALERTS\",\"alertname\":\"FailingOperator\",\"alertstate\":\"firing\",\"endpoint\":\"https-metrics\",\"exported_namespace\":\"e2e-test-olm-23440-jtwfs\",\"instance\":\"10.130.0.7:8081\",\"job\":\"olm-operator-metrics\",\"name\":\"etcdoperator.v0.9.4\",\"namespace\":\"openshift-operator-lifecycle-manager\",\"phase\":\"Failed\",\"pod\":\"olm-operator-746cfcd9dd-lt4sq\",\"reason\":\"NoOperatorGroup\",\"service\":\"olm-operator-metrics\",\"severity\":\"info\",\"version\":\"0.9.4\"},\"value\":[1580445549.069,\"1\"]}]",

https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/83/pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn/217
Comment 1 Dan Williams 2020-01-31 05:18:54 UTC 
time="2020-01-31T04:25:37Z" level=warning msg="csv in namespace with no operatorgroups" csv=etcdoperator.v0.9.4 id=wajMR namespace=e2e-test-olm-23440-jtwfs phase=Failed
time="2020-01-31T04:25:37Z" level=info msg="operatorgroup incorrect" csv=etcdoperator.v0.9.4 error="csv in namespace with no operatorgroups" id=wajMR namespace=e2e-test-olm-23440-jtwfs phase=Failed
I0131 04:25:37.315026       1 event.go:255] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"e2e-test-olm-23440-jtwfs", Name:"etcdoperator.v0.9.4", UID:"cb28a466-b6e1-4a81-b470-4c877724090e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"64275", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups
time="2020-01-31T04:25:37Z" level=info msg="operatorgroup not found" csv=etcdoperator.v0.9.4 id=GDfPy namespace=e2e-test-olm-23440-jtwfs operatorgroup=test-operator phase=Failed
E0131 04:25:37.331452       1 queueinformer_operator.go:282] sync {"update" "e2e-test-olm-23440-jtwfs/etcdoperator.v0.9.4"} failed: csv in namespace with no operatorgroups
time="2020-01-31T04:25:37Z" level=warning msg="csv in namespace with no operatorgroups" csv=etcdoperator.v0.9.4 id=8Wz3D namespace=e2e-test-olm-23440-jtwfs phase=Failed
time="2020-01-31T04:25:37Z" level=info msg="operatorgroup incorrect" csv=etcdoperator.v0.9.4 error="csv in namespace with no operatorgroups" id=8Wz3D namespace=e2e-test-olm-23440-jtwfs phase=Failed
I0131 04:25:37.343586       1 event.go:255] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"e2e-test-olm-23440-jtwfs", Name:"etcdoperator.v0.9.4", UID:"cb28a466-b6e1-4a81-b470-4c877724090e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"64275", FieldPath:""}): type: 'Warning' reason: 'NoOperatorGroup' csv in namespace with no operatorgroups
time="2020-01-31T04:25:37Z" level=info msg="operatorgroup not found" csv=etcdoperator.v0.9.4 id=Zzefa namespace=e2e-test-olm-23440-jtwfs operatorgroup=test-operator phase=Failed
E0131 04:25:37.359453       1 queueinformer_operator.go:282] sync {"update" "e2e-test-olm-23440-jtwfs/etcdoperator.v0.9.4"} failed: error transitioning ClusterServiceVersion: csv in namespace with no operatorgroups and error updating CSV status: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "etcdoperator.v0.9.4": StorageError: invalid object, Code: 4, Key: /kubernetes.io/operators.coreos.com/clusterserviceversions/e2e-test-olm-23440-jtwfs/etcdoperator.v0.9.4, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: cb28a466-b6e1-4a81-b470-4c877724090e, UID in object meta: 
time="2020-01-31T04:25:37Z" level=info msg="gcing children" csv=etcdoperator.v0.9.4 id=wZ0aZ namespace=e2e-test-olm-23440-jtwfs phase=Failed

https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_ovn-kubernetes/83/pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn/217/artifacts/e2e-aws-ovn/pods/openshift-operator-lifecycle-manager_olm-operator-746cfcd9dd-lt4sq_olm-operator.log
Comment 2 Evan Cordell 2020-01-31 13:45:46 UTC 

*** This bug has been marked as a duplicate of bug 1790825 ***
Note You need to log in before you can comment on or make changes to this bug.