Bug 1776700 - CCOProvisioningFailed alert is found in a fresh cluster
Summary: CCOProvisioningFailed alert is found in a fresh cluster
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.3.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.3.0
Assignee: Joel Diaz
QA Contact: Xiaoli Tian
: 1783963 (view as bug list)
Depends On: 1781109 1783963
TreeView+ depends on / blocked
Reported: 2019-11-26 07:44 UTC by Junqi Zhao
Modified: 2020-01-23 11:14 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2020-01-23 11:14:34 UTC
Target Upstream Version:

Attachments (Terms of Use)
cloud-credential-operator pod logs (161.37 KB, text/plain)
2019-11-26 07:44 UTC, Junqi Zhao
no flags Details

System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 147 0 None closed Bug 1776700: pre-populate conditions with count of zero 2020-07-06 19:24:05 UTC
Red Hat Product Errata RHBA-2020:0062 0 None None None 2020-01-23 11:14:48 UTC

Description Junqi Zhao 2019-11-26 07:44:05 UTC
Created attachment 1639715 [details]
cloud-credential-operator pod logs

Description of problem:
4.3.0-0.nightly-2019-11-25-153929 fresh cluster, CCOProvisioningFailed alert is found

# oc -n openshift-monitoring get ep | grep alertmanager-main
NAME                          ENDPOINTS                                                          AGE
alertmanager-main   ,,                 6h31m

# token=`oc -n openshift-monitoring sa get-token prometheus-k8s`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-1  -- curl -k -H "Authorization: Bearer $token" '' | jq
      "labels": {
        "alertname": "CCOProvisioningFailed",
        "condition": "CredentialsProvisionFailure",
        "endpoint": "cco-metrics",
        "instance": "",
        "job": "cco-metrics",
        "namespace": "openshift-cloud-credential-operator",
        "pod": "cloud-credential-operator-7b4fd65dc5-z5z5q",
        "prometheus": "openshift-monitoring/k8s",
        "service": "cco-metrics",
        "severity": "warning"
      "annotations": {
        "summary": "CredentialsRequest(s) unable to be fulfilled"
      "startsAt": "2019-11-26T01:13:42.851606264Z",
      "endsAt": "2019-11-26T07:40:42.851606264Z",
      "generatorURL": "https://prometheus-k8s-openshift-monitoring.apps.juzhao-11-26.qe.devcluster.openshift.com/graph?g0.expr=cco_credentials_requests_conditions%7Bcondition%3D%22CredentialsProvisionFailure%22%7D+%3E+0&g0.tab=1",
      "status": {
        "state": "active",
        "silencedBy": [],
        "inhibitedBy": []
      "receivers": [
      "fingerprint": "554807430686d598"

CCOProvisioningFailed detail
alert: CCOProvisioningFailed
expr: cco_credentials_requests_conditions{condition="CredentialsProvisionFailure"}
  > 0
for: 5m
  severity: warning
  summary: CredentialsRequest(s) unable to be fulfilled
cco_credentials_requests_conditions{condition="CredentialsProvisionFailure"} > 0
Element	Value
cco_credentials_requests_conditions{condition="CredentialsProvisionFailure",endpoint="cco-metrics",instance="",job="cco-metrics",namespace="openshift-cloud-credential-operator",pod="cloud-credential-operator-7b4fd65dc5-z5z5q",service="cco-metrics"}	1

logs see the attached file
Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. See the description

Actual results:
CCOProvisioningFailed alert is found in a fresh cluster

Expected results:
no such alert

Additional info:

Comment 1 Scott Dodson 2019-12-09 19:53:29 UTC
This is the same as Bug 1781109, setting up dependency on that one as the 4.4 bug.

Comment 3 Joel Diaz 2019-12-10 15:57:06 UTC
The issue would be intermittent. Fundamentally what is happening is that once an alert fires (which wouldn't happen on every installation), the alert would never clear.

You can force an alert by adding a CredentialsRequest object that points to a namespace that doesn't exist.

apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
  name: my-cred-request
  namespace: openshift-cloud-credential-operator
    name: my-cred-request-secret
    namespace: namespace-does-not-exist
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    - effect: Allow
      - s3:CreateBucket
      - s3:DeleteBucket
      resource: "*"

After a few minutes you should see the alert. Now you can either create the namespace to clear the alert, or delete the CredentialsRequest so there is no longer a CredentialsRequest in a bad state, and you would expect the alert to clear, but it never does (at least it doesn't clear without the changes in the PR).

Comment 9 Vadim Rutkovsky 2019-12-16 10:51:31 UTC
*** Bug 1783963 has been marked as a duplicate of this bug. ***

Comment 12 errata-xmlrpc 2020-01-23 11:14:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.