Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1880845

Summary:	After upgrade from OCP v4.3.29 to v4.4.10 cluster-logging-operator is in CrashLoopBackOff
Product:	OpenShift Container Platform	Reporter:	Radomir Ludva <rludva>
Component:	Logging	Assignee:	IgorKarpukhin <ikarpukh>
Status:	CLOSED ERRATA	QA Contact:	Anping Li <anli>
Severity:	medium	Docs Contact:
Priority:	high
Version:	4.4	CC:	aos-bugs, dahernan, fsoppels, ikarpukh, kpelc, mabajodu, ocasalsa
Target Milestone:	---
Target Release:	4.6.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: nil pointer dereferencing Consequence: application crashed Fix: fixed nil pointer dereferencing Result: the application works normally	Story Points:	---
Clone Of:
Clones:	1881493 (view as bug list)		Environment:
Last Closed:	2020-10-27 15:12:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1881991

Description Radomir Ludva 2020-09-20 14:48:45 UTC

Description of problem:
-----------------------
* cluster-logging-operator is not working (CrashLoopBackOff)
--
NAME                                            READY   STATUS             RESTARTS   AGE     IP            NODE                                NOMINATED
cluster-logging-operator-5bc6b46649-82x8j       0/1     CrashLoopBackOff   38         173m    10.60.14.75   example.com   <none>
--

* After upgrading a cluster from OCP v4.3.29 to v4.4.10

* It looks like there is an bug in the openshift-logging-operator according to the logs:
--
E0919 08:08:22.992860       1 runtime.go:69] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/panic.go:679
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/map_faststr.go:204
/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/openshift/cluster-logging-operator/pkg/k8shandler/visualization.go:286
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/client-go/util/retry/util.go:64
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:203
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/client-go/util/retry/util.go:63
/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/openshift/cluster-logging-operator/pkg/k8shandler/visualization.go:267
/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/openshift/cluster-logging-operator/pkg/k8shandler/visualization.go:169
/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/openshift/cluster-logging-operator/pkg/k8shandler/reconciler.go:141
/go/src/github.com/openshift/cluster-logging-operator/_output/src/github.com/openshift/cluster-logging-operator/pkg/controller/trustedcabundle/trustedcabundle_controller.go:80
/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/go/src/github.com/openshift/cluster-logging-operator/_output/src/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/openshift/cluster-logging-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/asm_amd64.s:1357
panic: assignment to entry in nil map [recovered]
	panic: assignment to entry in nil map


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
OCP v4.3.29 to v4.4.10


Actual results:
---------------
Upgrade is not processed without any issue.

Comment 2 IgorKarpukhin 2020-09-21 18:27:44 UTC

CLO 4.4 and EO 4.5+ are affected (since kibana handling moved from CLO to EO). These two PRs solve this issue: https://github.com/openshift/elasticsearch-operator/pull/492, https://github.com/openshift/cluster-logging-operator/pull/705

Comment 7 Anping Li 2020-09-25 07:04:13 UTC

No such issue when upgrade to 4.6.

Comment 8 David Hernández Fernández 2020-09-30 12:36:14 UTC

For reference, the backport to 4.4 is here:
https://bugzilla.redhat.com/show_bug.cgi?id=1881493
https://github.com/openshift/cluster-logging-operator/pull/705

Comment 12 errata-xmlrpc 2020-10-27 15:12:50 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6.1 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4198