Bug 2028802 - CRI-O panic due to invalid memory address or nil pointer dereference
Summary: CRI-O panic due to invalid memory address or nil pointer dereference
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.10
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.10.0
Assignee: Peter Hunt
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-03 11:57 UTC by Sunil Choudhary
Modified: 2022-03-10 16:31 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-10 16:31:36 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
must-gather (10.19 MB, application/x-xz)
2021-12-03 11:57 UTC, Sunil Choudhary
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2857 0 None open Bug 2028802: crio: fix a segfault on 4.9->4.10 upgrade 2021-12-03 14:55:47 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-10 16:31:53 UTC

Description Sunil Choudhary 2021-12-03 11:57:08 UTC
Created attachment 1844582 [details]
must-gather

Description of problem:

CRI-O failing to start due to panic: runtime error: invalid memory address or nil pointer dereference

ClusterID: aef37418-3b8f-4c4b-bb4b-ed8503f69ff0
ClusterVersion: Updating to "4.10.0-0.nightly-2021-12-02-140313" from "4.9.0-0.nightly-2021-12-02-183002" for 5 hours: Working towards 4.10.0-0.nightly-2021-12-02-140313: 668 of 819 done (81% complete)
ClusterOperators:
	clusteroperator/dns is progressing: DNS "default" reports Progressing=True: "Have 7 available node-resolver pods, want 8."
	clusteroperator/machine-config is not available (Cluster not available for [{operator 4.9.0-0.nightly-2021-12-02-183002}]) because Unable to apply 4.10.0-0.nightly-2021-12-02-140313: timed out waiting for the condition during waitForDaemonsetRollout: Daemonset machine-config-daemon is not ready. status: (desired: 8, updated: 8, ready: 7, unavailable: 1)
	clusteroperator/network is degraded because DaemonSet "openshift-multus/multus" rollout is not making progress - last change 2021-12-03T07:56:46Z
DaemonSet "openshift-multus/multus-additional-cni-plugins" rollout is not making progress - last change 2021-12-03T07:57:32Z
DaemonSet "openshift-sdn/sdn" rollout is not making progress - last change 2021-12-03T07:57:11Z
	clusteroperator/storage is progressing: AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverNodeServiceControllerProgressing: Waiting for DaemonSet to deploy node pods


$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.nightly-2021-12-02-183002   True        True          4h49m   Working towards 4.10.0-0.nightly-2021-12-02-140313: 668 of 819 done (81% complete)

$ oc get nodes -o wide
NAME                                        STATUS                        ROLES    AGE     VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-48-118.us-east-2.compute.internal   Ready                         worker   5h38m   v1.22.1+bac83a5   10.0.48.118   <none>        Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ip-10-0-49-163.us-east-2.compute.internal   Ready                         worker   5h3m    v1.22.3+ffbb954   10.0.49.163   <none>        Red Hat Enterprise Linux 8.4 (Ootpa)                            4.18.0-348.2.1.el8_5.x86_64    cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8
ip-10-0-55-112.us-east-2.compute.internal   NotReady,SchedulingDisabled   worker   5h3m    v1.22.3+ffbb954   10.0.55.112   <none>        Red Hat Enterprise Linux 8.4 (Ootpa)                            4.18.0-348.2.1.el8_5.x86_64    cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8
ip-10-0-55-99.us-east-2.compute.internal    Ready                         master   5h44m   v1.22.1+bac83a5   10.0.55.99    <none>        Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ip-10-0-66-193.us-east-2.compute.internal   Ready                         worker   5h38m   v1.22.3+ffbb954   10.0.66.193   <none>        Red Hat Enterprise Linux CoreOS 49.84.202111292103-0 (Ootpa)    4.18.0-305.28.1.el8_4.x86_64   cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8
ip-10-0-69-232.us-east-2.compute.internal   Ready                         master   5h45m   v1.22.1+bac83a5   10.0.69.232   <none>        Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ip-10-0-78-138.us-east-2.compute.internal   Ready                         master   5h44m   v1.22.1+bac83a5   10.0.78.138   <none>        Red Hat Enterprise Linux CoreOS 410.84.202112012203-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-87.rhaos4.10.git1e2f62a.el8
ip-10-0-78-63.us-east-2.compute.internal    Ready                         worker   5h38m   v1.22.3+ffbb954   10.0.78.63    <none>        Red Hat Enterprise Linux CoreOS 49.84.202111292103-0 (Ootpa)    4.18.0-305.28.1.el8_4.x86_64   cri-o://1.22.1-7.rhaos4.9.git8ed8e79.el8


# rpm -q cri-o
cri-o-1.22.1-7.rhaos4.9.git8ed8e79.el8.x86_64

From crio journal logs:
...
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: time="2021-12-03 07:44:12.536773980Z" level=info msg="Node configuration value for systemd CollectMode is true"
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: time="2021-12-03 07:44:12.544140811Z" level=info msg="Node configuration value for systemd AllowedCPUs is true"
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: time="2021-12-03 07:44:12.547080992Z" level=info msg="Using default capabilities: CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FSETID, CAP_FOWNER, CAP_SETGID, CAP_SETUID, CAP_SETPCAP, CAP_NET_BIND_SERVICE, CAP_KILL"
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x556f3a99a8f3]
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 1 [running]:
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: panic(0x556f3bb34e00, 0x556f3cdf0570)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/panic.go:1065 +0x565 fp=0xc0006997a8 sp=0xc0006996e0 pc=0x556f39a94ac5
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.panicmem()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/panic.go:212 +0x5d fp=0xc0006997c8 sp=0xc0006997a8 pc=0x556f39a92abd
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.sigpanic()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/signal_unix.go:734 +0x185 fp=0xc000699800 sp=0xc0006997c8 pc=0x556f39aac925
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*Resources).ValidateDefaults(...)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/workloads.go:99
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*WorkloadConfig).Validate(0xc000129d10, 0xc0001d1365, 0x11, 0x1e, 0xc0006998d0)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/workloads.go:48 +0x33 fp=0xc000699858 sp=0xc000699800 pc=0x556f3a99a8f3
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.Workloads.Validate(0xc000129ce0, 0x9, 0x9)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/workloads.go:37 +0xb6 fp=0xc0006998f0 sp=0xc000699858 pc=0x556f3a99a856
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*RuntimeConfig).Validate(0xc0005c3738, 0xc0005f8e00, 0x1, 0x0, 0x7)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/config.go:932 +0x21f fp=0xc000699a18 sp=0xc0006998f0 pc=0x556f3a98f97f
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/pkg/config.(*Config).Validate(0xc0005c3600, 0xc000475b01, 0x556f3b4708db, 0x6)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/pkg/config/config.go:775 +0xb9 fp=0xc000699a60 sp=0xc000699a18 pc=0x556f3a98edb9
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: main.main.func2(0xc000388b40, 0x0, 0x0)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/cmd/crio/main.go:204 +0x17f fp=0xc000699d90 sp=0xc000699a60 pc=0x556f3b45e53f
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2.(*App).RunContext(0xc0001f8000, 0x556f3be7e570, 0xc000054040, 0xc000050210, 0x1, 0x1, 0x0, 0x0)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2/app.go:322 +0x6fe fp=0xc000699eb8 sp=0xc000699d90 pc=0x556f3a9b895e
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2.(*App).Run(...)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/vendor/github.com/urfave/cli/v2/app.go:224
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: main.main()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /builddir/build/BUILD/cri-o-8ed8e7995c7a5f890b84b34b121d9e691a4b1e49/_output/src/github.com/cri-o/cri-o/cmd/crio/main.go:353 +0x510 fp=0xc000699f88 sp=0xc000699eb8 pc=0x556f3b45b610
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.main()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:225 +0x263 fp=0xc000699fe0 sp=0xc000699f88 pc=0x556f39a97803
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000699fe8 sp=0xc000699fe0 pc=0x556f39acfca1
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 2 [force gc (idle)]:
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3ceb7ea0, 0x1411, 0x1)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006cfb0 sp=0xc00006cf90 pc=0x556f39a97c46
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:342
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.forcegchelper()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:276 +0xc5 fp=0xc00006cfe0 sp=0xc00006cfb0 pc=0x556f39a97aa5
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006cfe8 sp=0xc00006cfe0 pc=0x556f39acfca1
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.init.7
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:264 +0x37
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 3 [GC sweep wait]:
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3cec0ea0, 0x140c, 0x1)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006d7a8 sp=0xc00006d788 pc=0x556f39a97c46
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:342
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.bgsweep(0xc00007a000)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mgcsweep.go:182 +0x145 fp=0xc00006d7d8 sp=0xc00006d7a8 pc=0x556f39a81a45
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006d7e0 sp=0xc00006d7d8 pc=0x556f39acfca1
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.gcenable
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mgc.go:217 +0x5e
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 4 [sleep]:
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3cec1980, 0x1313, 0x2)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006df20 sp=0xc00006df00 pc=0x556f39a97c46
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:342
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.scavengeSleep(0x2e122f, 0x10624e)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mgcscavenge.go:241 +0xc5 fp=0xc00006df78 sp=0xc00006df20 pc=0x556f39a7f8e5
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.bgscavenge(0xc00007a000)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mgcscavenge.go:366 +0x1ed fp=0xc00006dfd8 sp=0xc00006df78 pc=0x556f39a7fb0d
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006dfe0 sp=0xc00006dfd8 pc=0x556f39acfca1
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.gcenable
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mgc.go:218 +0x85
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: goroutine 5 [finalizer wait]:
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.gopark(0x556f3be00218, 0x556f3cef3ab8, 0xffff1410, 0x1)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00006c758 sp=0xc00006c738 pc=0x556f39a97c46
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goparkunlock(...)
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/proc.go:342
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.runfinq()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mfinal.go:175 +0xab fp=0xc00006c7e0 sp=0xc00006c758 pc=0x556f39a76a2b
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: runtime.goexit()
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00006c7e8 sp=0xc00006c7e0 pc=0x556f39acfca1
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]: created by runtime.createfing
Dec 03 07:44:12 ip-10-0-55-112.us-east-2.compute.internal crio[1870]:         /usr/lib/golang/src/runtime/mfinal.go:156 +0x66

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Peter Hunt 2021-12-03 14:55:31 UTC
oop that is bad

Comment 2 Peter Hunt 2021-12-03 14:58:06 UTC
I'm  going for a two tiered fix here to cover bases: we can fix upgrades for all 4.9->4.10 by adding https://github.com/openshift/machine-config-operator/pull/2857, but we also can prevent the need for that table to be specified by a code change in 4.9: https://github.com/cri-o/cri-o/pull/5486. I don't see harm in both

Comment 4 Sunil Choudhary 2021-12-06 09:01:07 UTC
Verified on 4.10.0-0.ci-2021-12-05-113922

$ oc get clusterversion
NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.ci-2021-12-05-113922   True        False         20m     Cluster version is 4.10.0-0.ci-2021-12-05-113922

$ oc get nodes -o wide
NAME                                        STATUS   ROLES    AGE    VERSION           INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
ip-10-0-53-253.us-east-2.compute.internal   Ready    worker   97m    v1.22.3+ffbb954   10.0.53.253   <none>        Red Hat Enterprise Linux 8.4 (Ootpa)                            4.18.0-348.2.1.el8_5.x86_64    cri-o://1.22.1-8.rhaos4.9.gite059965.el8
ip-10-0-59-18.us-east-2.compute.internal    Ready    master   146m   v1.22.1+6859754   10.0.59.18    <none>        Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-89.rhaos4.10.git367232b.el8
ip-10-0-61-145.us-east-2.compute.internal   Ready    worker   97m    v1.22.3+ffbb954   10.0.61.145   <none>        Red Hat Enterprise Linux 8.4 (Ootpa)                            4.18.0-348.2.1.el8_5.x86_64    cri-o://1.22.1-8.rhaos4.9.gite059965.el8
ip-10-0-62-167.us-east-2.compute.internal   Ready    worker   137m   v1.22.1+6859754   10.0.62.167   <none>        Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-89.rhaos4.10.git367232b.el8
ip-10-0-67-101.us-east-2.compute.internal   Ready    worker   137m   v1.22.1+6859754   10.0.67.101   <none>        Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-89.rhaos4.10.git367232b.el8
ip-10-0-69-204.us-east-2.compute.internal   Ready    worker   137m   v1.22.1+6859754   10.0.69.204   <none>        Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-89.rhaos4.10.git367232b.el8
ip-10-0-74-12.us-east-2.compute.internal    Ready    master   146m   v1.22.1+6859754   10.0.74.12    <none>        Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-89.rhaos4.10.git367232b.el8
ip-10-0-77-27.us-east-2.compute.internal    Ready    master   145m   v1.22.1+6859754   10.0.77.27    <none>        Red Hat Enterprise Linux CoreOS 410.84.202112040202-0 (Ootpa)   4.18.0-305.28.1.el8_4.x86_64   cri-o://1.23.0-89.rhaos4.10.git367232b.el8

$ oc describe clusterversion
Name:         version
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  config.openshift.io/v1
Kind:         ClusterVersion
Metadata:
  Creation Timestamp:  2021-12-06T06:23:48Z
  Generation:          3
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:channel:
        f:clusterID:
    Manager:      cluster-bootstrap
    Operation:    Update
    Time:         2021-12-06T06:23:48Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:availableUpdates:
        f:conditions:
        f:desired:
          .:
          f:image:
          f:version:
        f:history:
        f:observedGeneration:
        f:versionHash:
    Manager:      cluster-version-operator
    Operation:    Update
    Subresource:  status
    Time:         2021-12-06T07:25:39Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:desiredUpdate:
          .:
          f:force:
          f:image:
          f:version:
    Manager:         oc
    Operation:       Update
    Time:            2021-12-06T07:25:39Z
  Resource Version:  81138
  UID:               79a6dafb-4bb0-43fd-9ee3-e08c2da40f99
Spec:
  Channel:     stable-4.9
  Cluster ID:  7cde9dc2-05ef-4ea7-b1c0-d6b417bcc9e5
  Desired Update:
    Force:    true
    Image:    registry.ci.openshift.org/ocp/release:4.10.0-0.ci-2021-12-05-113922
    Version:  
Status:
  Available Updates:  <nil>
  Conditions:
    Last Transition Time:  2021-12-06T06:48:28Z
    Message:               Done applying 4.10.0-0.ci-2021-12-05-113922
    Status:                True
    Type:                  Available
    Last Transition Time:  2021-12-06T07:55:28Z
    Status:                False
    Type:                  Failing
    Last Transition Time:  2021-12-06T08:32:10Z
    Message:               Cluster version is 4.10.0-0.ci-2021-12-05-113922
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2021-12-06T06:23:48Z
    Message:               Unable to retrieve available updates: currently reconciling cluster version 4.10.0-0.ci-2021-12-05-113922 not found in the "stable-4.9" channel
    Reason:                VersionNotFound
    Status:                False
    Type:                  RetrievedUpdates
  Desired:
    Image:    registry.ci.openshift.org/ocp/release:4.10.0-0.ci-2021-12-05-113922
    Version:  4.10.0-0.ci-2021-12-05-113922
  History:
    Completion Time:    2021-12-06T08:32:10Z
    Image:              registry.ci.openshift.org/ocp/release:4.10.0-0.ci-2021-12-05-113922
    Started Time:       2021-12-06T07:25:39Z
    State:              Completed
    Verified:           false
    Version:            4.10.0-0.ci-2021-12-05-113922
    Completion Time:    2021-12-06T06:48:28Z
    Image:              quay.io/openshift-release-dev/ocp-release@sha256:e1853d68d8ff093ec353ca7078b6b6df1533729688bb016b8208263ee7423f66
    Started Time:       2021-12-06T06:23:48Z
    State:              Completed
    Verified:           false
    Version:            4.9.10
  Observed Generation:  3
  Version Hash:         YivEXFVUVMA=
Events:                 <none>


sh-4.4# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/crio.service.d
           └─10-mco-default-madv.conf, 10-mco-profile-unix-socket.conf
   Active: active (running) since Mon 2021-12-06 08:18:17 UTC; 36min ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 1429 (crio)
    Tasks: 24
   Memory: 790.6M
      CPU: 2min 24.244s
   CGroup: /system.slice/crio.service
           └─1429 /usr/bin/crio

sh-4.4# journalctl -u crio | grep -i panic
sh-4.4#

Comment 7 errata-xmlrpc 2022-03-10 16:31:36 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.