Bug 1880053
| Summary: | On downgrade from 4.6.0-fc.6 to 4.5.10 storage operator never progresses | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Scott Dodson <sdodson> |
| Component: | Storage | Assignee: | aos-storage-staff <aos-storage-staff> |
| Status: | CLOSED DUPLICATE | QA Contact: | Qin Ping <piqin> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 4.5 | CC: | aos-bugs |
| Target Milestone: | --- | Keywords: | Upgrades |
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-17 15:44:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** This bug has been marked as a duplicate of bug 1877316 *** |
Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Install 4.5.10 2. Upgrade to 4.6.0-fc.6 3. Downgrade to 4.5.10 Actual results: Downgrade hangs forever waiting on storage operator which appears hung attempting to acquire the lease, the deployment has been downgraded but it's not making any progress. Expected results: storage operator downgrades to 4.5.10 $ oc get co storage -o yaml apiVersion: config.openshift.io/v1 kind: ClusterOperator metadata: creationTimestamp: "2020-09-17T12:56:47Z" generation: 1 managedFields: - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:spec: {} f:status: .: {} f:extension: {} manager: cluster-version-operator operation: Update time: "2020-09-17T12:56:47Z" - apiVersion: config.openshift.io/v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: {} f:relatedObjects: {} f:versions: {} manager: cluster-storage-operator operation: Update time: "2020-09-17T14:41:12Z" name: storage resourceVersion: "92660" selfLink: /apis/config.openshift.io/v1/clusteroperators/storage uid: 6e573e02-45e0-4e1b-ad12-c8e5e9f7a924 spec: {} status: conditions: - lastTransitionTime: "2020-09-17T14:37:02Z" reason: AsExpected status: "True" type: Available - lastTransitionTime: "2020-09-17T13:58:37Z" reason: AsExpected status: "False" type: Degraded - lastTransitionTime: "2020-09-17T13:58:39Z" reason: AsExpected status: "True" type: Upgradeable - lastTransitionTime: "2020-09-17T14:41:12Z" reason: AsExpected status: "False" type: Progressing extension: null relatedObjects: - group: "" name: openshift-cluster-storage-operator resource: namespaces - group: "" name: openshift-cluster-csi-drivers resource: namespaces - group: "" name: openshift-manila-csi-driver resource: namespaces - group: operator.openshift.io name: cluster resource: storages - group: operator.openshift.io name: ebs.csi.aws.com resource: clustercsidrivers - group: operator.openshift.io name: csi.ovirt.org resource: clustercsidrivers - group: operator.openshift.io name: manila.csi.openstack.org resource: clustercsidrivers versions: - name: operator version: 4.6.0-fc.6 - name: AWSEBSCSIDriverOperator version: 4.6.0-fc.6 $ oc logs -n openshift-cluster-storage-operator pod/cluster-storage-operator-7c88bc6f57-jl2ww -f {"level":"info","ts":1600354307.0386753,"logger":"cmd","msg":"Go Version: go1.13.4"} {"level":"info","ts":1600354307.0389068,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1600354307.0389543,"logger":"cmd","msg":"Version of operator-sdk: v0.4.0"} {"level":"info","ts":1600354307.0402992,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1600354307.5411584,"logger":"leader","msg":"Not the leader. Waiting."} {"level":"info","ts":1600354308.6798072,"logger":"leader","msg":"Not the leader. Waiting."} ... {"level":"info","ts":1600356082.2428927,"logger":"leader","msg":"Not the leader. Waiting."} {"level":"info","ts":1600356100.5972443,"logger":"leader","msg":"Not the leader. Waiting."} {"level":"info","ts":1600356119.281552,"logger":"leader","msg":"Not the leader. Waiting."} $ oc describe -n openshift-cluster-storage-operator pod/cluster-storage-operator-7c88bc6f57-jl2ww Name: cluster-storage-operator-7c88bc6f57-jl2ww Namespace: openshift-cluster-storage-operator Priority: 2000000000 Priority Class Name: system-cluster-critical Node: ip-10-0-163-136.us-west-1.compute.internal/10.0.163.136 Start Time: Thu, 17 Sep 2020 10:51:36 -0400 Labels: name=cluster-storage-operator pod-template-hash=7c88bc6f57 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "", "interface": "eth0", "ips": [ "10.128.0.32" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "", "interface": "eth0", "ips": [ "10.128.0.32" ], "default": true, "dns": {} }] openshift.io/scc: anyuid Status: Running IP: 10.128.0.32 IPs: IP: 10.128.0.32 Controlled By: ReplicaSet/cluster-storage-operator-7c88bc6f57 Containers: cluster-storage-operator: Container ID: cri-o://6ff5d37294f25f63b68979f4222480746f083373912d4c071b0fdb9900ec6512 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd1483fddffd6dc92958ec82fd8bd1d29bf4fad155bd760a05b55c57cacf9f40 Image ID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd1483fddffd6dc92958ec82fd8bd1d29bf4fad155bd760a05b55c57cacf9f40 Port: 60000/TCP Host Port: 0/TCP Command: cluster-storage-operator State: Running Started: Thu, 17 Sep 2020 10:51:47 -0400 Ready: True Restart Count: 0 Requests: cpu: 10m memory: 20Mi Environment: RELEASE_VERSION: 4.5.10 WATCH_NAMESPACE: openshift-cluster-storage-operator (v1:metadata.namespace) POD_NAME: cluster-storage-operator-7c88bc6f57-jl2ww (v1:metadata.name) OPERATOR_NAME: storage Mounts: /var/run/secrets/kubernetes.io/serviceaccount from cluster-storage-operator-token-svtf6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: cluster-storage-operator-token-svtf6: Type: Secret (a volume populated by a Secret) SecretName: cluster-storage-operator-token-svtf6 Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute for 120s node.kubernetes.io/unreachable:NoExecute for 120s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned openshift-cluster-storage-operator/cluster-storage-operator-7c88bc6f57-jl2ww to ip-10-0-163-136.us-west-1.compute.internal Normal AddedInterface 31m multus Add eth0 [10.128.0.32/23] Normal Pulling 30m kubelet, ip-10-0-163-136.us-west-1.compute.internal Pulling image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd1483fddffd6dc92958ec82fd8bd1d29bf4fad155bd760a05b55c57cacf9f40" Normal Pulled 30m kubelet, ip-10-0-163-136.us-west-1.compute.internal Successfully pulled image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd1483fddffd6dc92958ec82fd8bd1d29bf4fad155bd760a05b55c57cacf9f40" in 5.803690059s Normal Created 30m kubelet, ip-10-0-163-136.us-west-1.compute.internal Created container cluster-storage-operator Normal Started 30m kubelet, ip-10-0-163-136.us-west-1.compute.internal Started container cluster-storage-operator