Bug 1994730
Summary: | upgrade from 4.6 to 4.7 to 4.8 with mcp worker "paused=true", crio report "panic: close of closed channel" which lead to a master Node go into Restart loop | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
Component: | Node | Assignee: | Peter Hunt <pehunt> |
Node sub component: | CRI-O | QA Contact: | Sunil Choudhary <schoudha> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | high | ||
Priority: | high | CC: | aos-bugs |
Version: | 4.7 | ||
Target Milestone: | --- | ||
Target Release: | 4.6.z | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-09-09 01:52:54 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1994729 | ||
Bug Blocks: |
Description
OpenShift BugZilla Robot
2021-08-17 19:07:08 UTC
PR merged Verified on 4.6.0-0.nightly-2021-08-31-113011 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-0.nightly-2021-08-31-113011 True False 4h21m Cluster version is 4.6.0-0.nightly-2021-08-31-113011 $ oc get machineconfigpool worker -o yaml apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfigPool metadata: creationTimestamp: "2021-09-02T11:43:45Z" generation: 4 labels: machineconfiguration.openshift.io/mco-built-in: "" pools.operator.machineconfiguration.openshift.io/worker: "" name: worker resourceVersion: "134023" selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker uid: 4718361d-f20d-43a5-b0f0-12bc5eefed89 spec: configuration: name: rendered-worker-1cf2954d719a18381144898ac5856c45 source: - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 00-worker - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-container-runtime - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-kubelet - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-generated-registries - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-ssh machineConfigSelector: matchLabels: machineconfiguration.openshift.io/role: worker nodeSelector: matchLabels: node-role.kubernetes.io/worker: "" paused: true status: conditions: - lastTransitionTime: "2021-09-02T11:45:14Z" message: "" reason: "" status: "False" type: RenderDegraded - lastTransitionTime: "2021-09-02T11:45:19Z" message: "" reason: "" status: "False" type: NodeDegraded - lastTransitionTime: "2021-09-02T11:45:19Z" message: "" reason: "" status: "False" type: Degraded - lastTransitionTime: "2021-09-02T11:55:34Z" message: "" reason: "" status: "False" type: Updating - lastTransitionTime: "2021-09-02T17:40:18Z" message: "" reason: "" status: "False" type: Updated configuration: name: rendered-worker-c0a999b199adc5ba3f226cfcc999547d source: - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 00-worker - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-container-runtime - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 01-worker-kubelet - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-generated-registries - apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig name: 99-worker-ssh degradedMachineCount: 0 machineCount: 3 observedGeneration: 4 readyMachineCount: 0 unavailableMachineCount: 0 updatedMachineCount: 0 $ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-09-02-093257 --force --allow-explicit-upgrade=true ... Updating to release image registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-09-02-093257 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-09-02-093257 True False 3m42s Cluster version is 4.7.0-0.nightly-2021-09-02-093257 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-134-64.ap-south-1.compute.internal Ready worker 6h16m v1.19.0+4c3480d 10.0.134.64 <none> Red Hat Enterprise Linux CoreOS 46.82.202108251457-0 (Ootpa) 4.18.0-193.60.2.el8_2.x86_64 cri-o://1.19.3-11.rhaos4.6.git66a69b8.el8 ip-10-0-151-39.ap-south-1.compute.internal Ready master 6h25m v1.20.0+9689d22 10.0.151.39 <none> Red Hat Enterprise Linux CoreOS 47.84.202109010857-0 (Ootpa) 4.18.0-305.12.1.el8_4.x86_64 cri-o://1.20.4-14.rhaos4.7.gitbce257b.el8 ip-10-0-167-213.ap-south-1.compute.internal Ready worker 6h13m v1.19.0+4c3480d 10.0.167.213 <none> Red Hat Enterprise Linux CoreOS 46.82.202108251457-0 (Ootpa) 4.18.0-193.60.2.el8_2.x86_64 cri-o://1.19.3-11.rhaos4.6.git66a69b8.el8 ip-10-0-191-32.ap-south-1.compute.internal Ready master 6h20m v1.20.0+9689d22 10.0.191.32 <none> Red Hat Enterprise Linux CoreOS 47.84.202109010857-0 (Ootpa) 4.18.0-305.12.1.el8_4.x86_64 cri-o://1.20.4-14.rhaos4.7.gitbce257b.el8 ip-10-0-196-226.ap-south-1.compute.internal Ready worker 6h15m v1.19.0+4c3480d 10.0.196.226 <none> Red Hat Enterprise Linux CoreOS 46.82.202108251457-0 (Ootpa) 4.18.0-193.60.2.el8_2.x86_64 cri-o://1.19.3-11.rhaos4.6.git66a69b8.el8 ip-10-0-198-117.ap-south-1.compute.internal Ready master 6h25m v1.20.0+9689d22 10.0.198.117 <none> Red Hat Enterprise Linux CoreOS 47.84.202109010857-0 (Ootpa) 4.18.0-305.12.1.el8_4.x86_64 cri-o://1.20.4-14.rhaos4.7.gitbce257b.el8 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.44 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3395 |