Bug 1992927

Summary:	4.8 to 4.8 upgrade caused crio error with 'Error loading cached network config: net work \"multus-cni-network\" not found in CNI cache'
Product:	OpenShift Container Platform	Reporter:	MinLi <minmli>
Component:	Node	Assignee:	Peter Hunt <pehunt>
Node sub component:	CRI-O	QA Contact:	Weinan Liu <weinliu>
Status:	CLOSED DUPLICATE	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	aos-bugs
Version:	4.8
Target Milestone:	---
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-10-13 19:26:58 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description MinLi 2021-08-12 02:29:56 UTC

Description of problem:
when upgrade from 4.8.4-x86_64 to 4.8.0-0.nightly-2021-08-09-135211, one rhel node get NotReady, and crio log show: Error loading cached network config: net
work \"multus-cni-network\" not found in CNI cache

Version-Release number of selected component (if applicable):
4.8.4-x86_64

How reproducible:
hit once

Steps to Reproduce:
1.upgrade a cluster from 4.8.4-x86_64 to 4.8.0-0.nightly-2021-08-09-135211
2.check node status


Actual results:
2.one rhel node "yinzhou11065626-b692q-w-a-l-rhel-0" gets NotReady. I can ping internal IP successfully from master node, but can't ssh to it.

the node show relatively high cpu load from 20%-60%, whereas the other rhel node just show from 5%-8%. 

Then I reboot it from gcp platform, the crio and kubelet stop for some reason.

Expected results:
2 all node get Ready

Additional info:
the cluster's profile: 21_Disconnected IPI on GCP with RHCOS & RHEL7.9 & FIPS on & http_proxy & Etcd Encryption on

one rhcos node get NotReady when trying with the same profile earlier

Comment 1 MinLi 2021-08-12 02:43:20 UTC

crio log: 

Aug 11 02:06:48 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:39.851282742Z" level=info msg="Starting container: b91f49be2d0466f491764
e4ea1da3bb344a5e19697700419c9e8a6cebb2058af" id=4772fee7-ca9e-41d1-8591-2bda430957cb name=/runtime.v1alpha2.RuntimeService/StartContainer
Aug 11 02:06:51 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:40.999021279Z" level=info msg="Started container b91f49be2d0466f491764e4
ea1da3bb344a5e19697700419c9e8a6cebb2058af: openshift-monitoring/thanos-querier-84fd9469f9-nzx2n/oauth-proxy" id=4772fee7-ca9e-41d1-8591-2bda430957cb name=/run
time.v1alpha2.RuntimeService/StartContainer
Aug 11 02:06:52 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:43.196552534Z" level=info msg="NetworkStart: stopping network for sandbo
x 2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbo
x
Aug 11 02:06:52 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:44.842659455Z" level=info msg="Got pod network &{Name:network-check-targ
et-5tj5v Namespace:openshift-network-diagnostics ID:2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc UID:b4e76153-2c90-4f2b-8b17-8cd93e28085c 
NetNS:/var/run/netns/3320b9dc-2307-436b-8ea8-8e5eefa01185 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:
[]}] Aliases:map[]}"
Aug 11 02:06:52 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:44.842747999Z" level=error msg="Error loading cached network config: net
work \"multus-cni-network\" not found in CNI cache"
Aug 11 02:06:52 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:44.842771525Z" level=warning msg="Falling back to loading from existing 
plugins on disk"
Aug 11 02:06:52 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:06:46.162923007Z" level=info msg="Deleting pod openshift-network-diagnostic
s_network-check-target-5tj5v from CNI network \"multus-cni-network\" (type=multus)"
Aug 11 02:07:56 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:07:55.170575932Z" level=error msg="Error stopping network on cleanup: faile
d to destroy network for pod sandbox k8s_network-check-target-5tj5v_openshift-network-diagnostics_b4e76153-2c90-4f2b-8b17-8cd93e28085c_0(2ccce980f1b410e29b154
5ec97da862725f0cbf3019e50d81f6138f4788981bc): error removing pod openshift-network-diagnostics_network-check-target-5tj5v from CNI network \"multus-cni-networ
k\": netplugin failed: \"2021-08-11T02:07:47Z [error] SetNetworkStatus: failed to query the pod network-check-target-5tj5v in out of cluster comm: pods \\\"ne
twork-check-target-5tj5v\\\" not found\\n2021-08-11T02:07:49Z [error] Multus: error unsetting the networks status: SetNetworkStatus: failed to query the pod n
etwork-check-target-5tj5v in out of cluster comm: pods \\\"network-check-target-5tj5v\\\" not found\\n2021-08-11T02:07:49Z [verbose] Del: openshift-network-di
agnostics:network-check-target-5tj5v:unknownUID:openshift-sdn:eth0 {\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"name\\\":\\\"openshift-sdn\\\",\\\"type\\\":\\\"opens
hift-sdn\\\"}\\n\"" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:05 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:03.827456520Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b41
0e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:09 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:09.410653159Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b41
0e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:10 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:10.761450998Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b41
0e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:38 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:20.284952561Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:41 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:20.285027937Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:41 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:20.289548578Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:50 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:38.990255239Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:08:50 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:08:38.990318465Z" level=info msg="RunSandbox: deleting pod ID 2ccce980f1b410e29b1545ec97da862725f0cbf3019e50d81f6138f4788981bc from idIndex" id=5ef19f40-92e3-4ba3-961e-bad6569beb07 name=/runtime.v1alpha2.RuntimeService/RunPodSandbox
Aug 11 02:14:24 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:14:02.051887614Z" level=info msg="Checking image status: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cecb7839b484569a1c2826af500ff1107f811725296c58d13e0f5df6a10e882b" id=f58ece1b-9bf1-4811-837b-c5561baff141 name=/runtime.v1alpha2.ImageService/ImageStatus
Aug 11 02:17:44 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1898]: time="2021-08-11 02:17:38.796728205Z" level=info msg="Image status: &{0xc0001c4070 map[]}" id=f58ece1b-9bf1-4811-837b-c5561baff141 name=/runtime.v1alpha2.ImageService/ImageStatus
-- Reboot --
Aug 11 07:24:18 yinzhou11065626-b692q-w-a-l-rhel-0 systemd[1]: Starting Open Container Initiative Daemon...
Aug 11 07:24:18 yinzhou11065626-b692q-w-a-l-rhel-0 crio[1802]: time="2021-08-11 07:24:18.683788455Z" level=info msg="Starting CRI-O, version: 1.21.2-9.rhaos4.8.git3b87110.el7, git: ()"

Comment 2 MinLi 2021-08-12 02:44:33 UTC

kubelet log:

Aug 11 02:11:13 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:04.544669    1994 log.go:184] http: TLS handshake error from 10.0.128.2:56678: rite tcp 10.0.128.6:10250->10.0.128.2:56678: write: broken pipe
Aug 11 02:11:39 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:39.182047    1994 fsHandler.go:128] fs: disk usage and inodes count on followin dirs took 1m58.129296598s: [/var/lib/containers/storage/overlay/ef3d08b36858929b28ff0058ba079f064291b47cbdcb555ae4ca47ec57289796/diff /var/log/pods/openshiftmonitoring_alertmanager-main-2_ff0c75e9-941f-45ab-8ea7-131549030d40/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 5s
Aug 11 02:11:45 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.964143    1994 log.go:184] http: TLS handshake error from 10.0.128.3:59886: OF
Aug 11 02:11:45 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.964701    1994 log.go:184] http: TLS handshake error from 10.0.128.3:59364: rite tcp 10.0.128.6:10250->10.0.128.3:59364: write: broken pipe
Aug 11 02:11:45 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.965129    1994 log.go:184] http: TLS handshake error from 10.0.128.3:59426: rite tcp 10.0.128.6:10250->10.0.128.3:59426: write: broken pipe
Aug 11 02:11:45 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.965675    1994 log.go:184] http: TLS handshake error from 10.0.128.2:58984: OF
Aug 11 02:11:47 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.966114    1994 log.go:184] http: TLS handshake error from 10.0.128.3:59432: ug 11 02:11:47 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.966597    1994 log.go:184] http: TLS handshake error from 10.0.128.2:58436: write tcp 10.0.128.6:10250->10.0.128.2:58436: write: broken pipe
Aug 11 02:11:49 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.967026    1994 log.go:184] http: TLS handshake error from 10.0.128.2:58512: EOF
Aug 11 02:11:49 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:45.967477    1994 log.go:184] http: TLS handshake error from 10.0.128.2:59458: EOF
Aug 11 02:11:49 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:47.990365    1994 log.go:184] http: TLS handshake error from 10.0.128.2:58124: write tcp 10.0.128.6:10250->10.0.128.2:58124: write: broken pipe
Aug 11 02:11:51 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:47.990726    1994 log.go:184] http: TLS handshake error from 10.0.128.3:58956: write tcp 10.0.128.6:10250->10.0.128.3:58956: write: broken pipe
Aug 11 02:11:51 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:47.991062    1994 log.go:184] http: TLS handshake error from 10.0.128.3:58964: write tcp 10.0.128.6:10250->10.0.128.3:58964: write: broken pipe
Aug 11 02:11:51 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:47.991366    1994 log.go:184] http: TLS handshake error from 10.0.128.2:57990: write tcp 10.0.128.6:10250->10.0.128.2:57990: write: broken pipe
Aug 11 02:11:51 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:47.991662    1994 log.go:184] http: TLS handshake error from 10.0.128.2:58070: write tcp 10.0.128.6:10250->10.0.128.2:58070: write: broken pipe
Aug 11 02:11:51 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: E0811 02:11:49.832223    1994 remote_runtime.go:515] "Status from runtime service failed" rr="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Aug 11 02:11:51 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:51.782123    1994 log.go:184] http: TLS handshake error from 10.0.128.3:60792: EOF
Aug 11 02:11:53 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:11:53.385196    1994 log.go:184] http: TLS handshake error from 10.0.128.3:59892: EOF
Aug 11 02:13:21 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:14.090918    1994 log.go:184] http: TLS handshake error from 10.0.128.2:60346: OF
monitoring_alertmanager-main-2_ff0c75e9-941f-45ab-8ea7-131549030d40/alertmanager-proxy/0.log]; will not log again for this container unless duration exceeds 6s
Aug 11 02:13:25 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:25.034535    1994 trace.go:205] Trace[1382194230]: "iptables ChainExists" (11-Aug-2021 02:07:52.795) (total time: 166243ms):
Aug 11 02:13:25 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: Trace[1382194230]: [2m46.243974541s] [2m46.243974541s] END
Aug 11 02:13:29 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:27.999964    1994 log.go:184] http: TLS handshake error from 10.0.128.3:58896: write tcp 10.0.128.6:10250->10.0.128.3:58896: write: broken pipe
Aug 11 02:13:31 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:12:25.313750    1994 trace.go:205] Trace[830151287]: "iptables ChainExists" (11-Aug-2021 02:07:39.715) (total time: 175682ms):
Aug 11 02:13:31 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: Trace[830151287]: [2m55.682604064s] [2m55.682604064s] END
Aug 11 02:13:31 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:28.000247    1994 fsHandler.go:128] fs: disk usage and inodes count on followig dirs took 1m2.686483586s: [/var/lib/containers/storage/overlay/031de67180988c6d13dda2f9d17106524a66c9631169474b55b010072ef1fa63/diff /var/log/pods/openshiftmonitoring_alertmanager-main-2_ff0c75e9-941f-45ab-8ea7-131549030d40/config-reloader/0.log]; will not log again for this container unless duration exceeds 5s
Aug 11 02:13:49 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:29.706755    1994 log.go:184] http: TLS handshake error from 10.0.128.2:59030:write tcp 10.0.128.6:10250->10.0.128.2:59030: write: broken pipe
Aug 11 02:13:53 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:29.706901    1994 log.go:184] http: TLS handshake error from 10.0.128.3:60858:write tcp 10.0.128.6:10250->10.0.128.3:60858: write: broken pipe
Aug 11 02:13:53 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: E0811 02:13:29.706937    1994 remote_image.go:87] "Get ImageStatus from image service faild" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cecb7839b484569a1c286af500ff1107f811725296c58d13e0f5df6a10e882b"
Aug 11 02:13:53 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:13:33.715974    1994 log.go:184] http: TLS handshake error from 10.0.128.3:60864:write tcp 10.0.128.6:10250->10.0.128.3:60864: write: broken pipe
Aug 11 02:15:41 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:01.975467    1994 log.go:184] http: TLS handshake error from 10.0.128.2:33424: OF
Aug 11 02:15:43 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:01.975833    1994 log.go:184] http: TLS handshake error from 10.0.0.6:41814: EO
Aug 11 02:15:43 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:01.976159    1994 log.go:184] http: TLS handshake error from 10.0.0.6:41816: EO
Aug 11 02:15:43 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:03.597346    1994 reflector.go:225] Stopping reflector *v1.ConfigMap (0s) from bject-"openshift-machine-config-operator"/"openshift-service-ca.crt"
Aug 11 02:15:43 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: E0811 02:14:11.146694    1994 kubelet.go:2212] "Container runtime sanity check failed" err=rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Aug 11 02:15:43 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: E0811 02:14:16.777637    1994 kuberuntime_container.go:444] "ListContainers failed" err="rp error: code = DeadlineExceeded desc = context deadline exceeded"
Aug 11 02:15:45 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.200448    1994 log.go:184] http: TLS handshake error from 10.0.0.6:42180: EO
Aug 11 02:15:46 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.200888    1994 log.go:184] http: TLS handshake error from 10.0.0.6:42182: EO
Aug 11 02:15:49 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.202689    1994 log.go:184] http: TLS handshake error from 10.0.0.6:42374: EO
Aug 11 02:15:49 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.203023    1994 log.go:184] http: TLS handshake error from 10.0.0.6:42376: EO
Aug 11 02:15:53 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.203379    1994 log.go:184] http: TLS handshake error from 10.0.128.2:33968: OF
Aug 11 02:15:55 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.203785    1994 log.go:184] http: TLS handshake error from 10.0.128.2:34014: OF
Aug 11 02:15:57 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.204174    1994 log.go:184] http: TLS handshake error from 10.0.0.6:42566: EO
Aug 11 02:15:59 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[1994]: I0811 02:14:48.204484    1994 log.go:184] http: TLS handshake error from 10.0.0.6:42568: EO
-- Reboot --
Aug 11 07:24:20 yinzhou11065626-b692q-w-a-l-rhel-0 systemd[1]: Starting Kubernetes Kubelet...
Aug 11 07:24:22 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[2022]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --evictio-soft instead. Will be removed in a future version.
Aug 11 07:24:22 yinzhou11065626-b692q-w-a-l-rhel-0 hyperkube[2022]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config fle speci

Comment 3 Peter Hunt 2021-08-12 18:23:15 UTC

"Error loading cached network config" is not a fatal error, it is just a cache miss that will fallback to the CNI plugin. the problem is that the CNI plugin doesn't seem to be active. is multus running? A must-gather from the affected node may be helpful here...

Comment 4 MinLi 2021-08-13 08:16:20 UTC

@Peter
Please access the must-gather via: http://file.nay.redhat.com/~minmli/
file name: rhel-must-gather.tar.gz

Comment 5 Peter Hunt 2021-09-09 16:09:52 UTC

darn, must-gathers only gather information about masters by default, not workers. If I had to guess, I would guess this is actually a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1999645, which I believe this nightly had (it's hard to tell because I think it's been garbage collected)

@minmli are you able to rerun the test to get the must gather to confirm? alternatively, I am alright closing this as a dup/insufficient data, as the symptoms (node going not ready) match that of the potential duplicate

Comment 6 Peter Hunt 2021-10-13 19:26:58 UTC

closing this as I believe it's a dup. please reopen if it's found again

*** This bug has been marked as a duplicate of bug 1999645 ***