Bug 1712775
| Summary: | Cloud-credentials clusteroperator have different version for 4.1.0-0.nightly-2019-05-22-050858 | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | weiwei jiang <wjiang> | ||||
| Component: | Cloud Credential Operator | Assignee: | Devan Goodwin <dgoodwin> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | Oleg Nesterov <olnester> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | medium | ||||||
| Version: | 4.1.0 | CC: | aos-bugs, ccoleman, jokerman, mmccomas, wking, wsun | ||||
| Target Milestone: | --- | Keywords: | Regression | ||||
| Target Release: | 4.1.z | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-06-27 15:01:25 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
No this issue from 4.1.0-0.nightly-2019-05-18-050636 -> 4.1.0-0.nightly-2019-05-21-060354 $ oc get clusteroperator NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.1.0-0.nightly-2019-05-21-060354 True False False 31h cloud-credential 4.1.0-0.nightly-2019-05-21-060354 True False False 31h cluster-autoscaler 4.1.0-0.nightly-2019-05-21-060354 True False False 31h console 4.1.0-0.nightly-2019-05-21-060354 True False False 31h dns 4.1.0-0.nightly-2019-05-21-060354 True False False 31h image-registry 4.1.0-0.nightly-2019-05-21-060354 True False False 3h22m ingress 4.1.0-0.nightly-2019-05-21-060354 True False False 27h kube-apiserver 4.1.0-0.nightly-2019-05-21-060354 True False False 31h kube-controller-manager 4.1.0-0.nightly-2019-05-21-060354 True False False 31h kube-scheduler 4.1.0-0.nightly-2019-05-21-060354 True False False 31h machine-api 4.1.0-0.nightly-2019-05-21-060354 True False False 31h machine-config 4.1.0-0.nightly-2019-05-21-060354 True False False 11h marketplace 4.1.0-0.nightly-2019-05-21-060354 True False False 71m monitoring 4.1.0-0.nightly-2019-05-21-060354 True False False 4h network 4.1.0-0.nightly-2019-05-21-060354 True False False 31h node-tuning 4.1.0-0.nightly-2019-05-21-060354 True False False 4h15m openshift-apiserver 4.1.0-0.nightly-2019-05-21-060354 True False False 4h1m openshift-controller-manager 4.1.0-0.nightly-2019-05-21-060354 True False False 31h openshift-samples 4.1.0-0.nightly-2019-05-21-060354 True False False 4h15m operator-lifecycle-manager 4.1.0-0.nightly-2019-05-21-060354 True False False 31h operator-lifecycle-manager-catalog 4.1.0-0.nightly-2019-05-21-060354 True False False 31h service-ca 4.1.0-0.nightly-2019-05-21-060354 True False False 31h service-catalog-apiserver 4.1.0-0.nightly-2019-05-21-060354 True False False 4h4m service-catalog-controller-manager 4.1.0-0.nightly-2019-05-21-060354 True False False 30h storage 4.1.0-0.nightly-2019-05-21-060354 True False False 4h15m Probably going to need logs from the cluster-version and cred operators for this. Probably going to need logs from the cluster-version and cred operators for this. This can be fixed in 4.1.1 because this is a bug in the operator during upgrade. Checked again, and now the version is same with others.
➜ ~ date -u
Thu 23 May 2019 02:36:31 AM UTC
➜ ~ oc get clusteroperator cloud-credential -o yaml
apiVersion: config.openshift.io/v1
kind: ClusterOperator
metadata:
creationTimestamp: "2019-05-22T02:32:19Z"
generation: 1
name: cloud-credential
resourceVersion: "607758"
selfLink: /apis/config.openshift.io/v1/clusteroperators/cloud-credential
uid: d2777e35-7c39-11e9-89d7-801844eef6b8
spec: {}
status:
conditions:
- lastTransitionTime: "2019-05-22T02:32:19Z"
message: No credentials requests reporting errors.
reason: NoCredentialsFailing
status: "False"
type: Degraded
- lastTransitionTime: "2019-05-22T17:21:48Z"
message: 4 of 4 credentials requests provisioned and reconciled.
reason: ReconcilingComplete
status: "False"
type: Progressing
- lastTransitionTime: "2019-05-22T02:32:19Z"
status: "True"
type: Available
extension: null
versions:
- name: operator
version: 4.1.0-0.nightly-2019-05-22-050858
➜ ~ oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.1.0-0.nightly-2019-05-22-050858 True False False 23h
cloud-credential 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
cluster-autoscaler 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
console 4.1.0-0.nightly-2019-05-22-050858 True False False 23h
dns 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
image-registry 4.1.0-0.nightly-2019-05-22-050858 True False False 23h
ingress 4.1.0-0.nightly-2019-05-22-050858 True False False 23h
kube-apiserver 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
kube-controller-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
kube-scheduler 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
machine-api 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
machine-config 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
marketplace 4.1.0-0.nightly-2019-05-22-050858 True False False 19h
monitoring 4.1.0-0.nightly-2019-05-22-050858 True False False 22h
network 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
node-tuning 4.1.0-0.nightly-2019-05-22-050858 True False False 19h
openshift-apiserver 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
openshift-controller-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
openshift-samples 4.1.0-0.nightly-2019-05-22-050858 True False False 19h
operator-lifecycle-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
operator-lifecycle-manager-catalog 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
service-ca 4.1.0-0.nightly-2019-05-22-050858 True False False 24h
service-catalog-apiserver 4.1.0-0.nightly-2019-05-22-050858 True False False 20h
service-catalog-controller-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 20h
storage 4.1.0-0.nightly-2019-05-22-050858 True False False 19h
Created attachment 1572274 [details]
cloud_credential_operator_pod_log
Add cloud credential operator pod log here for further debuging
I reproduced this issue on upi-on-vpshere and did not reproduce it on ipi-on-aws Cred operator has literally no upgrade code, we just report the image we're running via the RELEASE_VERSION env var: https://github.com/openshift/cloud-credential-operator/blob/master/pkg/controller/credentialsrequest/status.go#L116 However that code to update status executes after every CredentialsRequest reconcile occurs, as long as there are CredentialsRequests in the system, it should execute and update the version. The describe deployment update above shows the correct value: RELEASE_VERSION: 4.1.0-0.nightly-2019-05-22-050858 I am curious if the cred operator pod was successfully restarted, based on this: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 72m deployment-controller Scaled up replica set cloud-credential-operator-7d9dfc6789 to 1 Normal ScalingReplicaSet 72m deployment-controller Scaled down replica set cloud-credential-operator-6c5fc856b8 to 0 It would appear that it was. In the logs we see: time="2019-05-22T07:21:48Z" level=debug msg="syncing cluster operator status" time="2019-05-22T07:21:48Z" level=debug msg="4 cred requests" time="2019-05-22T07:21:48Z" level=debug msg="set ClusterOperator condition" message="No credentials requests reporting errors." reason=NoCredentialsFailing status=False type=Degraded time="2019-05-22T07:21:48Z" level=debug msg="set ClusterOperator condition" message="4 of 4 credentials requests provisioned and reconciled." reason=ReconcilingComplete status=False type=Progressing time="2019-05-22T07:21:48Z" level=debug msg="set ClusterOperator condition" message= reason= status=True type=Available time="2019-05-22T07:21:48Z" level=info msg="version has changed, updating progressing condition lastTransitionTime" new="[{operator 4.1.0-0.nightly-2019-05-22-050858}]" old="[{operator 4.1.0-0.nightly-2019-05-21-060354}]" time="2019-05-22T07:21:48Z" level=debug msg="cluster operator status updated" So clearly the cred operator sees the new version. This block appears three times, other times when we sync status we do not see this line, which almost seems like the operator versions is somehow getting reset to it's old value by something external to this pod, as the log file never shows it setting it back. If anyone reproduces could we get output from oc admin must-gather for that cluster. Also grab cluster-version operator logs, since must-gather may not collect those yet. Due to the resource limit, the original cluster is already flushed. I have to collect all the logs once I met this again. We are unable to reproduce this so if anyone sees it please keep the cluster us and get in touch, want to see exactly what's running in the openshift-cloud-credential-operator namespace when it occurs. Closing as a duplicate for another issue with more logging. *** This bug has been marked as a duplicate of bug 1707827 *** |
Description of problem: After I upgrade cluster to 4.1.0-0.nightly-2019-05-22-050858, check the clusteroperators list, and found cloud-credentials have different version than others # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.1.0-0.nightly-2019-05-22-050858 True False False 5h42m cloud-credential 4.1.0-0.nightly-2019-05-21-060354 True False False 5h58m cluster-autoscaler 4.1.0-0.nightly-2019-05-22-050858 True False False 5h58m console 4.1.0-0.nightly-2019-05-22-050858 True False False 5h49m dns 4.1.0-0.nightly-2019-05-22-050858 True False False 5h57m image-registry 4.1.0-0.nightly-2019-05-22-050858 True False False 5h52m ingress 4.1.0-0.nightly-2019-05-22-050858 True False False 5h52m kube-apiserver 4.1.0-0.nightly-2019-05-22-050858 True False False 5h56m kube-controller-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 5h56m kube-scheduler 4.1.0-0.nightly-2019-05-22-050858 True False False 5h56m machine-api 4.1.0-0.nightly-2019-05-22-050858 True False False 5h58m machine-config 4.1.0-0.nightly-2019-05-22-050858 True False False 5h57m marketplace 4.1.0-0.nightly-2019-05-22-050858 True False False 68m monitoring 4.1.0-0.nightly-2019-05-22-050858 True False False 4h5m network 4.1.0-0.nightly-2019-05-22-050858 True False False 5h58m node-tuning 4.1.0-0.nightly-2019-05-22-050858 True False False 69m openshift-apiserver 4.1.0-0.nightly-2019-05-22-050858 True False False 5h55m openshift-controller-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 5h57m openshift-samples 4.1.0-0.nightly-2019-05-22-050858 True False False 68m operator-lifecycle-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 5h57m operator-lifecycle-manager-catalog 4.1.0-0.nightly-2019-05-22-050858 True False False 5h57m service-ca 4.1.0-0.nightly-2019-05-22-050858 True False False 5h58m service-catalog-apiserver 4.1.0-0.nightly-2019-05-22-050858 True False False 164m service-catalog-controller-manager 4.1.0-0.nightly-2019-05-22-050858 True False False 164m storage 4.1.0-0.nightly-2019-05-22-050858 True False False 68m Version-Release number of selected component (if applicable): 4.1.0-0.nightly-2019-05-21-060354 -> 4.1.0-0.nightly-2019-05-22-050858 How reproducible: Always Steps to Reproduce: 1. Setup a 4.1.0-0.nightly-2019-05-21-060354 cluster 2. Upgrade to 4.1.0-0.nightly-2019-05-22-050858 3. Check clusteroperators list Actual results: cloud-credentials have different version than others Expected results: Should be consistent Additional info: # oc describe deployment.apps/cloud-credential-operator -n openshift-cloud-credential-operator Name: cloud-credential-operator Namespace: openshift-cloud-credential-operator CreationTimestamp: Wed, 22 May 2019 10:31:13 +0800 Labels: control-plane=controller-manager controller-tools.k8s.io=1.0 Annotations: deployment.kubernetes.io/revision: 2 Selector: control-plane=controller-manager,controller-tools.k8s.io=1.0 Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 25% max unavailable, 25% max surge Pod Template: Labels: control-plane=controller-manager controller-tools.k8s.io=1.0 Containers: manager: Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba803653c4af0089feca91b065d1d96ad4660ae061cf40793894d24585891f3c Port: 9876/TCP Host Port: 0/TCP Command: /root/manager --log-level debug Limits: memory: 500Mi Requests: cpu: 10m memory: 150Mi Environment: RELEASE_VERSION: 4.1.0-0.nightly-2019-05-22-050858 Mounts: <none> Volumes: <none> Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: <none> NewReplicaSet: cloud-credential-operator-7d9dfc6789 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 72m deployment-controller Scaled up replica set cloud-credential-operator-7d9dfc6789 to 1 Normal ScalingReplicaSet 72m deployment-controller Scaled down replica set cloud-credential-operator-6c5fc856b8 to 0