Bug 1994730
| Summary: | upgrade from 4.6 to 4.7 to 4.8 with mcp worker "paused=true", crio report "panic: close of closed channel" which lead to a master Node go into Restart loop | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | OpenShift BugZilla Robot <openshift-bugzilla-robot> |
| Component: | Node | Assignee: | Peter Hunt <pehunt> |
| Node sub component: | CRI-O | QA Contact: | Sunil Choudhary <schoudha> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | aos-bugs |
| Version: | 4.7 | ||
| Target Milestone: | --- | ||
| Target Release: | 4.6.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2021-09-09 01:52:54 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1994729 | ||
| Bug Blocks: | |||
|
Description
OpenShift BugZilla Robot
2021-08-17 19:07:08 UTC
PR merged Verified on 4.6.0-0.nightly-2021-08-31-113011
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.6.0-0.nightly-2021-08-31-113011 True False 4h21m Cluster version is 4.6.0-0.nightly-2021-08-31-113011
$ oc get machineconfigpool worker -o yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
creationTimestamp: "2021-09-02T11:43:45Z"
generation: 4
labels:
machineconfiguration.openshift.io/mco-built-in: ""
pools.operator.machineconfiguration.openshift.io/worker: ""
name: worker
resourceVersion: "134023"
selfLink: /apis/machineconfiguration.openshift.io/v1/machineconfigpools/worker
uid: 4718361d-f20d-43a5-b0f0-12bc5eefed89
spec:
configuration:
name: rendered-worker-1cf2954d719a18381144898ac5856c45
source:
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 00-worker
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-container-runtime
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-kubelet
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 99-worker-generated-registries
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 99-worker-ssh
machineConfigSelector:
matchLabels:
machineconfiguration.openshift.io/role: worker
nodeSelector:
matchLabels:
node-role.kubernetes.io/worker: ""
paused: true
status:
conditions:
- lastTransitionTime: "2021-09-02T11:45:14Z"
message: ""
reason: ""
status: "False"
type: RenderDegraded
- lastTransitionTime: "2021-09-02T11:45:19Z"
message: ""
reason: ""
status: "False"
type: NodeDegraded
- lastTransitionTime: "2021-09-02T11:45:19Z"
message: ""
reason: ""
status: "False"
type: Degraded
- lastTransitionTime: "2021-09-02T11:55:34Z"
message: ""
reason: ""
status: "False"
type: Updating
- lastTransitionTime: "2021-09-02T17:40:18Z"
message: ""
reason: ""
status: "False"
type: Updated
configuration:
name: rendered-worker-c0a999b199adc5ba3f226cfcc999547d
source:
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 00-worker
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-container-runtime
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 01-worker-kubelet
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 99-worker-generated-registries
- apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
name: 99-worker-ssh
degradedMachineCount: 0
machineCount: 3
observedGeneration: 4
readyMachineCount: 0
unavailableMachineCount: 0
updatedMachineCount: 0
$ oc adm upgrade --to-image=registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-09-02-093257 --force --allow-explicit-upgrade=true
...
Updating to release image registry.ci.openshift.org/ocp/release:4.7.0-0.nightly-2021-09-02-093257
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.7.0-0.nightly-2021-09-02-093257 True False 3m42s Cluster version is 4.7.0-0.nightly-2021-09-02-093257
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-134-64.ap-south-1.compute.internal Ready worker 6h16m v1.19.0+4c3480d 10.0.134.64 <none> Red Hat Enterprise Linux CoreOS 46.82.202108251457-0 (Ootpa) 4.18.0-193.60.2.el8_2.x86_64 cri-o://1.19.3-11.rhaos4.6.git66a69b8.el8
ip-10-0-151-39.ap-south-1.compute.internal Ready master 6h25m v1.20.0+9689d22 10.0.151.39 <none> Red Hat Enterprise Linux CoreOS 47.84.202109010857-0 (Ootpa) 4.18.0-305.12.1.el8_4.x86_64 cri-o://1.20.4-14.rhaos4.7.gitbce257b.el8
ip-10-0-167-213.ap-south-1.compute.internal Ready worker 6h13m v1.19.0+4c3480d 10.0.167.213 <none> Red Hat Enterprise Linux CoreOS 46.82.202108251457-0 (Ootpa) 4.18.0-193.60.2.el8_2.x86_64 cri-o://1.19.3-11.rhaos4.6.git66a69b8.el8
ip-10-0-191-32.ap-south-1.compute.internal Ready master 6h20m v1.20.0+9689d22 10.0.191.32 <none> Red Hat Enterprise Linux CoreOS 47.84.202109010857-0 (Ootpa) 4.18.0-305.12.1.el8_4.x86_64 cri-o://1.20.4-14.rhaos4.7.gitbce257b.el8
ip-10-0-196-226.ap-south-1.compute.internal Ready worker 6h15m v1.19.0+4c3480d 10.0.196.226 <none> Red Hat Enterprise Linux CoreOS 46.82.202108251457-0 (Ootpa) 4.18.0-193.60.2.el8_2.x86_64 cri-o://1.19.3-11.rhaos4.6.git66a69b8.el8
ip-10-0-198-117.ap-south-1.compute.internal Ready master 6h25m v1.20.0+9689d22 10.0.198.117 <none> Red Hat Enterprise Linux CoreOS 47.84.202109010857-0 (Ootpa) 4.18.0-305.12.1.el8_4.x86_64 cri-o://1.20.4-14.rhaos4.7.gitbce257b.el8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6.44 bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:3395 |