Created attachment 1844582 [details] must-gather Description of problem: CRI-O failing to start due to panic: runtime error: invalid memory address or nil pointer dereference ClusterID: aef37418-3b8f-4c4b-bb4b-ed8503f69ff0 ClusterVersion: Updating to "4.10.0-0.nightly-2021-12-02-140313" from "4.9.0-0.nightly-2021-12-02-183002" for 5 hours: Working towards 4.10.0-0.nightly-2021-12-02-140313: 668 of 819 done (81% complete) ClusterOperators: clusteroperator/dns is progressing: DNS "default" reports Progressing=True: "Have 7 available node-resolver pods, want 8." clusteroperator/machine-config is not available (Cluster not available for [{operator 4.9.0-0.nightly-2021-12-02-183002}]) because Unable to apply 4.10.0-0.nightly-2021-12-02-140313: timed out waiting for the condition during waitForDaemonsetRollout: Daemonset machine-config-daemon is not ready. status: (desired: 8, updated: 8, ready: 7, unavailable: 1) clusteroperator/network is degraded because DaemonSet "openshift-multus/multus" rollout is not making progress - last change 2021-12-03T07:56:46Z DaemonSet "openshift-multus/multus-additional-cni-plugins" rollout is not making progress - last change 2021-12-03T07:57:32Z DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-12-03T07:57:11Z clusteroperator/storage is progressing: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverNodeServiceControllerProgressing: Waiting for DaemonSet to deploy node pods $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-12-02-183002 True True 4h49m Working towards 4.10.0-0.nightly-2021-12-02-140313: 668 of 819 done (81% complete) $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-48-118.us-east-2.compute.internal Ready worker 5h38m v1.22.1+bac83a5 10.0.48.118 <none> Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8 ip-10-0-49-163.us-east-2.compute.internal Ready worker 5h3m v1.22.3+ffbb954 10.0.49.163 <none> Red Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-348.2.1.el8_5.x86_64 cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8 ip-10-0-55-112.us-east-2.compute.internal NotReady,SchedulingDisabled worker 5h3m v1.22.3+ffbb954 10.0.55.112 <none> Red Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-348.2.1.el8_5.x86_64 cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8 ip-10-0-55-99.us-east-2.compute.internal Ready master 5h44m v1.22.1+bac83a5 10.0.55.99 <none> Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8 ip-10-0-66-193.us-east-2.compute.internal Ready worker 5h38m v1.22.3+ffbb954 10.0.66.193 <none> Red Hat Enterprise Linux CoreOS 49.84.202111292103-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8 ip-10-0-69-232.us-east-2.compute.internal Ready master 5h45m v1.22.1+bac83a5 10.0.69.232 <none> Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8 ip-10-0-78-138.us-east-2.compute.internal Ready master 5h44m v1.22.1+bac83a5 10.0.78.138 <none> Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8 ip-10-0-78-63.us-east-2.compute.internal Ready worker 5h38m v1.22.3+ffbb954 10.0.78.63 <none> Red Hat Enterprise Linux CoreOS 49.84.202111292103-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8 # rpm -q cri-o cri-o-1.22.1-7.rhaos4.9.git8ed8e79.el8.x86_64 From crio journal logs: ... Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: time="2021-12-03 07:44:12.536773980Z" level=info msg="Node configuration value for systemd CollectMode is true" Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: time="2021-12-03 07:44:12.544140811Z" level=info msg="Node configuration value for systemd AllowedCPUs is true" Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: time="2021-12-03 07:44:12.547080992Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL" Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: panic: runtime error: invalid memory address or nil pointer dereference Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x556f3a99a8f3] Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 1 [running]: Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: panic(0x556f3bb34e00, 0x556f3cdf0570) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/panic.go:1065 +0x565 fp=0xc0006997a8 sp=0xc0006996e0 pc=0x556f39a94ac5 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.panicmem() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/panic.go:212 +0x5d fp=0xc0006997c8 sp=0xc0006997a8 pc=0x556f39a92abd Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.sigpanic() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/signal_unix.go:734 +0x185 fp=0xc000699800 sp=0xc0006997c8 pc=0x556f39aac925 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*Resources).ValidateDefaults(...) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/workloads.go:99 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*WorkloadConfig).Validate(0xc000129d10, 0xc0001d1365, 0x11, 0x1e, 0xc0006998d0) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/workloads.go:48 +0x33 fp=0xc000699858 sp=0xc000699800 pc=0x556f3a99a8f3 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.Workloads.Validate(0xc000129ce0, 0x9, 0x9) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/workloads.go:37 +0xb6 fp=0xc0006998f0 sp=0xc000699858 pc=0x556f3a99a856 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*RuntimeConfig).Validate(0xc0005c3738, 0xc0005f8e00, 0x1, 0x0, 0x7) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/config.go:932 +0x21f fp=0xc000699a18 sp=0xc0006998f0 pc=0x556f3a98f97f Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*Config).Validate(0xc0005c3600, 0xc000475b01, 0x556f3b4708db, 0x6) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/config.go:775 +0xb9 fp=0xc000699a60 sp=0xc000699a18 pc=0x556f3a98edb9 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: main.main.func2(0xc000388b40, 0x0, 0x0) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/cmd/crio/main.go:204 +0x17f fp=0xc000699d90 sp=0xc000699a60 pc=0x556f3b45e53f Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2.(*App).RunContext(0xc0001f8000, 0x556f3be7e570, 0xc000054040, 0xc000050210, 0x1, 0x1, 0x0, 0x0) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2/app.go:322 +0x6fe fp=0xc000699eb8 sp=0xc000699d90 pc=0x556f3a9b895e Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2.(*App).Run(...) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2/app.go:224 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: main.main() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/cmd/crio/main.go:353 +0x510 fp=0xc000699f88 sp=0xc000699eb8 pc=0x556f3b45b610 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.main() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:225 +0x263 fp=0xc000699fe0 sp=0xc000699f88 pc=0x556f39a97803 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000699fe8 sp=0xc000699fe0 pc=0x556f39acfca1 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 2 [force gc (idle)]: Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3ceb7ea0, 0x1411, 0x1) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006cfb0 sp=0xc00006cf90 pc=0x556f39a97c46 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:342 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.forcegchelper() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:276 +0xc5 fp=0xc00006cfe0 sp=0xc00006cfb0 pc=0x556f39a97aa5 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x556f39acfca1 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.init.7 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:264 +0x37 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 3 [GC sweep wait]: Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3cec0ea0, 0x140c, 0x1) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006d7a8 sp=0xc00006d788 pc=0x556f39a97c46 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:342 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.bgsweep(0xc00007a000) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mgcsweep.go:182 +0x145 fp=0xc00006d7d8 sp=0xc00006d7a8 pc=0x556f39a81a45 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006d7e0 sp=0xc00006d7d8 pc=0x556f39acfca1 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.gcenable Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mgc.go:217 +0x5e Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 4 [sleep]: Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3cec1980, 0x1313, 0x2) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006df20 sp=0xc00006df00 pc=0x556f39a97c46 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:342 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.scavengeSleep(0x2e122f, 0x10624e) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mgcscavenge.go:241 +0xc5 fp=0xc00006df78 sp=0xc00006df20 pc=0x556f39a7f8e5 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.bgscavenge(0xc00007a000) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mgcscavenge.go:366 +0x1ed fp=0xc00006dfd8 sp=0xc00006df78 pc=0x556f39a7fb0d Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006dfe0 sp=0xc00006dfd8 pc=0x556f39acfca1 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.gcenable Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mgc.go:218 +0x85 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 5 [finalizer wait]: Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3cef3ab8, 0xffff1410, 0x1) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006c758 sp=0xc00006c738 pc=0x556f39a97c46 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...) Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/proc.go:342 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.runfinq() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mfinal.go:175 +0xab fp=0xc00006c7e0 sp=0xc00006c758 pc=0x556f39a76a2b Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit() Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006c7e8 sp=0xc00006c7e0 pc=0x556f39acfca1 Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.createfing Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: /usr/lib/golang/src/runtime/mfinal.go:156 +0x66 Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
oop that is bad
I'm going for a two tiered fix here to cover bases: we can fix upgrades for all 4.9->4.10 by adding https://github.com/openshift/machine-config-operator/pull/2857, but we also can prevent the need for that table to be specified by a code change in 4.9: https://github.com/cri-o/cri-o/pull/5486. I don't see harm in both
Verified on 4.10.0-0.ci-2021-12-05-113922 $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.ci-2021-12-05-113922 True False 20m Cluster version is 4.10.0-0.ci-2021-12-05-113922 $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-53-253.us-east-2.compute.internal Ready worker 97m v1.22.3+ffbb954 10.0.53.253 <none> Red Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-348.2.1.el8_5.x86_64 cri-o://1.22.1-8.rhaos4.9.gite059965.el8 ip-10-0-59-18.us-east-2.compute.internal Ready master 146m v1.22.1+6859754 10.0.59.18 <none> Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-89.rhaos4.10.git367232b.el8 ip-10-0-61-145.us-east-2.compute.internal Ready worker 97m v1.22.3+ffbb954 10.0.61.145 <none> Red Hat Enterprise Linux 8.4 (Ootpa) 4.18.0-348.2.1.el8_5.x86_64 cri-o://1.22.1-8.rhaos4.9.gite059965.el8 ip-10-0-62-167.us-east-2.compute.internal Ready worker 137m v1.22.1+6859754 10.0.62.167 <none> Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-89.rhaos4.10.git367232b.el8 ip-10-0-67-101.us-east-2.compute.internal Ready worker 137m v1.22.1+6859754 10.0.67.101 <none> Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-89.rhaos4.10.git367232b.el8 ip-10-0-69-204.us-east-2.compute.internal Ready worker 137m v1.22.1+6859754 10.0.69.204 <none> Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-89.rhaos4.10.git367232b.el8 ip-10-0-74-12.us-east-2.compute.internal Ready master 146m v1.22.1+6859754 10.0.74.12 <none> Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-89.rhaos4.10.git367232b.el8 ip-10-0-77-27.us-east-2.compute.internal Ready master 145m v1.22.1+6859754 10.0.77.27 <none> Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa) 4.18.0-305.28.1.el8_4.x86_64 cri-o://1.23.0-89.rhaos4.10.git367232b.el8 $ oc describe clusterversion Name: version Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: ClusterVersion Metadata: Creation Timestamp: 2021-12-06T06:23:48Z Generation: 3 Managed Fields: API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: .: f:channel: f:clusterID: Manager: cluster-bootstrap Operation: Update Time: 2021-12-06T06:23:48Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:availableUpdates: f:conditions: f:desired: .: f:image: f:version: f:history: f:observedGeneration: f:versionHash: Manager: cluster-version-operator Operation: Update Subresource: status Time: 2021-12-06T07:25:39Z API Version: config.openshift.io/v1 Fields Type: FieldsV1 fieldsV1: f:spec: f:desiredUpdate: .: f:force: f:image: f:version: Manager: oc Operation: Update Time: 2021-12-06T07:25:39Z Resource Version: 81138 UID: 79a6dafb-4bb0-43fd-9ee3-e08c2da40f99 Spec: Channel: stable-4.9 Cluster ID: 7cde9dc2-05ef-4ea7-b1c0-d6b417bcc9e5 Desired Update: Force: true Image: registry.ci.openshift.org/ocp/release:4.10.0-0.ci-2021-12-05-113922 Version: Status: Available Updates: <nil> Conditions: Last Transition Time: 2021-12-06T06:48:28Z Message: Done applying 4.10.0-0.ci-2021-12-05-113922 Status: True Type: Available Last Transition Time: 2021-12-06T07:55:28Z Status: False Type: Failing Last Transition Time: 2021-12-06T08:32:10Z Message: Cluster version is 4.10.0-0.ci-2021-12-05-113922 Status: False Type: Progressing Last Transition Time: 2021-12-06T06:23:48Z Message: Unable to retrieve available updates: currently reconciling cluster version 4.10.0-0.ci-2021-12-05-113922 not found in the "stable-4.9" channel Reason: VersionNotFound Status: False Type: RetrievedUpdates Desired: Image: registry.ci.openshift.org/ocp/release:4.10.0-0.ci-2021-12-05-113922 Version: 4.10.0-0.ci-2021-12-05-113922 History: Completion Time: 2021-12-06T08:32:10Z Image: registry.ci.openshift.org/ocp/release:4.10.0-0.ci-2021-12-05-113922 Started Time: 2021-12-06T07:25:39Z State: Completed Verified: false Version: 4.10.0-0.ci-2021-12-05-113922 Completion Time: 2021-12-06T06:48:28Z Image: quay.io/openshift-release-dev/ocp-release@sha256:e1853d68d8ff093ec353ca7078b6b6df1533729688bb016b8208263ee7423f66 Started Time: 2021-12-06T06:23:48Z State: Completed Verified: false Version: 4.9.10 Observed Generation: 3 Version Hash: YivEXFVUVMA= Events: <none> sh-4.4# systemctl status crio ● crio.service - Container Runtime Interface for OCI (CRI-O) Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system/crio.service.d └─10-mco-default-madv.conf, 10-mco-profile-unix-socket.conf Active: active (running) since Mon 2021-12-06 08:18:17 UTC; 36min ago Docs: https://github.com/cri-o/cri-o Main PID: 1429 (crio) Tasks: 24 Memory: 790.6M CPU: 2min 24.244s CGroup: /system.slice/crio.service └─1429 /usr/bin/crio sh-4.4# journalctl -u crio | grep -i panic sh-4.4#
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056