Bug 2027824

Summary:	ovnkube-master CrashLoopBackoff: panic: Expected slice or struct but got string
Product:	OpenShift Container Platform	Reporter:	Devan Goodwin <dgoodwin>
Component:	Networking	Assignee:	Surya Seetharaman <surya>
Networking sub component:	ovn-kubernetes	QA Contact:	Ross Brattain <rbrattai>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	unspecified
Priority:	unspecified	CC:	bpickard, rbrattai
Version:	4.10
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:31:00 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Devan Goodwin 2021-11-30 19:25:04 UTC

TRT identified a crash scenario that looks like a clear code error in ovn.

Prow job: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-azure-ovn-upgrade/1465486322222239744

Crash log: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.10-e2e-azure-ovn-upgrade/1465486322222239744/artifacts/e2e-azure-ovn-upgrade/gather-must-gather/artifacts/must-gather/namespaces/openshift-ovn-kubernetes/pods/ovnkube-master-dl7zg/ovnkube-master/ovnkube-master/logs/previous.log

Which shows:

2021-11-30T02:27:25.994211476Z E1130 02:27:25.994173       1 master.go:163] Failed to remove any legacy per node join switches
2021-11-30T02:27:25.994211476Z I1130 02:27:25.994191       1 master.go:197] Upgrading to Namespace Deny PortGroup OVN Topology
2021-11-30T02:27:26.148705248Z panic: Expected slice or struct but got string
2021-11-30T02:27:26.148705248Z 
2021-11-30T02:27:26.148705248Z goroutine 49 [running]:
2021-11-30T02:27:26.148705248Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops.onModels({0x17ea8a0, 2021-11-30T02:27:26.148910652Z 0x2c46a60}, 0xc0002ad2f8)
2021-11-30T02:27:26.148910652Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops/model.go:247 +0x379
2021-11-30T02:27:26.148910652Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops.(*ModelClient).buildOps(0xc0002ad438, 0xc0002ad368, 0x0, {0xc0002ad508, 0x1, 0x50})
2021-11-30T02:27:26.148910652Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops/model_client.go:231 +0x4dd
2021-11-30T02:27:26.148910652Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops.(*ModelClient).DeleteOps(0xc0002ad3b0, {0xc0002ad508, 2021-11-30T02:27:26.148982554Z 0x50, 0x19f1380})
2021-11-30T02:27:26.148982554Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops/model_client.go:201 +0x45
2021-11-30T02:27:26.148982554Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops.(*ModelClient).Delete(0xc0002ad438, {0xc0002ad508, 0x23, 0xc0002ad478})
2021-11-30T02:27:26.148982554Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/libovsdbops/model_client.go:185 +0x25
2021-11-30T02:27:26.149011954Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).cleanupDGP(0xc00012b900, 0x0)
2021-11-30T02:27:26.149011954Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/gateway_cleanup.go:3462021-11-30T02:27:26.149037455Z  +0x176
2021-11-30T02:27:26.149037455Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).upgradeOVNTopology(0xc000000005, 0x0)
2021-11-30T02:27:26.149037455Z 	2021-11-30T02:27:26.149063555Z /go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/master.go:202 +0x136
2021-11-30T02:27:26.149063555Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).StartClusterMaster(0xc00012b9002021-11-30T02:27:26.149089256Z , {0x7ffe4df868c9, 0x23})
2021-11-30T02:27:26.149089256Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/master.go:2742021-11-30T02:27:26.149113456Z  +0x3f4
2021-11-30T02:27:26.149113456Z github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*Controller).Start.func1({0xc000f3cbf8, 0xc000f3cbfa2021-11-30T02:27:26.149137857Z })
2021-11-30T02:27:26.149137857Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/master.go:98 +0x11a
2021-11-30T02:27:26.149160057Z created by k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run
2021-11-30T02:27:26.149183558Z 	/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:211 +0x154

Comment 3 Ross Brattain 2021-12-09 20:38:23 UTC

No "panic: Expected slice or struct but got string" in prow or recent 4.10-e2e-azure-ovn-upgrade

Verified

Comment 6 errata-xmlrpc 2022-03-10 16:31:00 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056