Bug 2023832

Summary: CCO updates lastTransitionTime for non-Status changes
Product: OpenShift Container Platform Reporter: Joel Diaz <jdiaz>
Component: Cloud Credential OperatorAssignee: Nobody <nobody>
Status: CLOSED ERRATA QA Contact: Jianping SHu <jshu>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.10CC: jshu, lwan, wking
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-10 16:28:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Joel Diaz 2021-11-16 16:21:55 UTC 
Description of problem:
The ClusterOperator lastTransitionTime field for conditions is meant to be updated on status field changes. Presently, CCO updates on status or reason or message changes.

Version-Release number of selected component (if applicable):


How reproducible: 100%


Steps to Reproduce:
1. Put CCO into a degraded state. For exmaple make a CredentialsRequest that specifies a .spec.secretRef.namespace that doesn't exist.
2. Wait for Degraded=True, and notice the lastTransitionTime
3. Create a second CredentailsRequest that also references a non-existent namespace.

Actual results: the Degraded condition's lastTransitionTime will be updated when the message changes from "1 of X CredentailsRequests failing" to "2 of X CredentailsRequests failing"


Expected results: the lastTransitionTime should only change when Degraded status changes from true-to-false or false-to-true.


Additional info:
Comment 2 Jianping SHu 2021-11-17 05:48:28 UTC 
Verified with 4.10.0-0.nightly-2021-11-16-212854.

1. Check cloud-credential status.conditions
jianpingshu@jshu-mac mycluster % oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2021-11-17T00:28:18Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2021-11-17T00:28:22Z",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2021-11-17T00:36:15Z",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2021-11-17T00:28:18Z",
    "status": "True",
    "type": "Upgradeable"
  }
]

2.Create 1st CredentialsRequest that specifies a .spec.secretRef.namespace that doesn't exist (as below template)
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: my-cred-request1
  namespace: openshift-cloud-credential-operator
spec:
  secretRef:
    name: my-cred-request1-secret
    namespace: namespace-does-not-exist
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - s3:DeleteBucket
      resource: "*"

3.Check cloud-credential status.conditions again
jianpingshu@jshu-mac mycluster % oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2021-11-17T00:28:18Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2021-11-17T02:35:20Z",
    "message": "1 of 6 credentials requests are failing to sync.",
    "reason": "CredentialsFailing",
    "status": "True",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2021-11-17T02:35:18Z",
    "message": "5 of 6 credentials requests provisioned, 1 reporting errors.",
    "reason": "Reconciling",
    "status": "True",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2021-11-17T00:28:18Z",
    "status": "True",
    "type": "Upgradeable"
  }
]
4.Wait for minutes, create 2nd CredentialsRequest that specifies a .spec.secretRef.namespace that doesn't exist

5.Check cloud-credential status.conditions again. The lastTransitionTime of Degraded/Progressing kept no change and message updated.
jianpingshu@jshu-mac mycluster % oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2021-11-17T00:28:18Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2021-11-17T02:35:20Z",
    "message": "2 of 7 credentials requests are failing to sync.",
    "reason": "CredentialsFailing",
    "status": "True",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2021-11-17T02:35:18Z",
    "message": "5 of 7 credentials requests provisioned, 2 reporting errors.",
    "reason": "Reconciling",
    "status": "True",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2021-11-17T00:28:18Z",
    "status": "True",
    "type": "Upgradeable"
  }
]
Comment 3 Jianping SHu 2021-11-17 05:53:14 UTC 
Reproduced with 4.10.0-0.nightly-2021-11-15-034648.

1. Check cloud-credential status.conditions
jianpingshu@jshu-mac ~ % oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "False",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2021-11-16T23:53:45Z",
    "status": "False",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "True",
    "type": "Upgradeable"
  }
]

2.Create 1st CredentialsRequest that specifies a .spec.secretRef.namespace that doesn't exist (as below template)
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
metadata:
  name: my-cred-request1
  namespace: openshift-cloud-credential-operator
spec:
  secretRef:
    name: my-cred-request1-secret
    namespace: namespace-does-not-exist
  providerSpec:
    apiVersion: cloudcredential.openshift.io/v1
    kind: AWSProviderSpec
    statementEntries:
    - effect: Allow
      action:
      - s3:DeleteBucket
      resource: "*"

3.Check cloud-credential status.conditions again
jianpingshu@jshu-mac ~ % oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2021-11-17T02:23:44Z",
    "message": "1 of 6 credentials requests are failing to sync.",
    "reason": "CredentialsFailing",
    "status": "True",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2021-11-17T02:23:44Z",
    "message": "5 of 6 credentials requests provisioned, 1 reporting errors.",
    "reason": "Reconciling",
    "status": "True",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "True",
    "type": "Upgradeable"
  }
]

4.Wait for minutes, create 2nd CredentialsRequest that specifies a .spec.secretRef.namespace that doesn't exist

5.Check cloud-credential status.conditions again. The lastTransitionTime of Degraded/Progressing changed agin and message updated.
jianpingshu@jshu-mac ~ % oc get co cloud-credential -o json | jq -r ".status.conditions"
[
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "True",
    "type": "Available"
  },
  {
    "lastTransitionTime": "2021-11-17T02:30:41Z",
    "message": "2 of 7 credentials requests are failing to sync.",
    "reason": "CredentialsFailing",
    "status": "True",
    "type": "Degraded"
  },
  {
    "lastTransitionTime": "2021-11-17T02:30:41Z",
    "message": "5 of 7 credentials requests provisioned, 2 reporting errors.",
    "reason": "Reconciling",
    "status": "True",
    "type": "Progressing"
  },
  {
    "lastTransitionTime": "2021-11-16T23:47:07Z",
    "status": "True",
    "type": "Upgradeable"
  }
]
Comment 7 errata-xmlrpc 2022-03-10 16:28:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056