Bug 1837323 - EO panic: interface conversion: interface {} is string, not map[string]interface {} [recovered]
Summary: EO panic: interface conversion: interface {} is string, not map[string]interf...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 4.5
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.5.0
Assignee: IgorKarpukhin
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-19 09:35 UTC by Qiaoling Tang
Modified: 2020-07-13 17:40 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Response from EO API wasn't checked for nil/NULL values Consequence: EO operator crashed Fix: Added additional checks for the EO API response Result: EO operator not crashing anymore
Clone Of:
Environment:
Last Closed: 2020-07-13 17:40:02 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift elasticsearch-operator pull 362 0 None closed Bug 1837323: Fixed EO crash 2021-02-09 16:09:18 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:40:18 UTC

Description Qiaoling Tang 2020-05-19 09:35:07 UTC
Description of problem:
after creating clusterlogging instance, the EO restarts, logs:

{"level":"info","ts":1589880039.5820286,"logger":"cmd","msg":"Go Version: go1.13.8"}
{"level":"info","ts":1589880039.582052,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1589880039.582057,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"}
{"level":"info","ts":1589880039.5837452,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1589880039.7752895,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1589880039.7834234,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1589880039.9057093,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1589880039.9062088,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1589880039.9064121,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1589880039.906656,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1589880039.9067993,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1589880039.9069505,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1589880040.0523393,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"elasticsearch-operator","Service.Namespace":"openshift-operators-redhat"}
{"level":"info","ts":1589880040.0523727,"logger":"cmd","msg":"This operator no longer honors the image specified by the custom resources so that it is able to properly coordinate the configuration with the image."}
{"level":"info","ts":1589880040.0523794,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1589880041.2526214,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibanasecret-controller"}
{"level":"info","ts":1589880041.2526338,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"proxyconfig-controller"}
{"level":"info","ts":1589880041.2526128,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trustedcabundle-controller"}
{"level":"info","ts":1589880041.2526255,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"}
{"level":"info","ts":1589880041.2526128,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"}
{"level":"info","ts":1589880041.3528018,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trustedcabundle-controller","worker count":1}
{"level":"info","ts":1589880041.352811,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1}
{"level":"info","ts":1589880041.352802,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibanasecret-controller","worker count":1}
{"level":"info","ts":1589880041.3528116,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"proxyconfig-controller","worker count":1}
{"level":"info","ts":1589880041.3528316,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1}
time="2020-05-19T09:21:59Z" level=error msg="Operator unable to read local file to get contents: open /tmp/ocp-eo/ca.crt: no such file or directory"
time="2020-05-19T09:21:59Z" level=error msg="Operator unable to read local file to get contents: open /tmp/ocp-eo/ca.crt: no such file or directory"
{"level":"error","ts":1589880119.4201286,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"kibana-controller","request":"openshift-logging/kibana","error":"Did not receive hashvalue for trusted CA value","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
time="2020-05-19T09:22:00Z" level=info msg="Updating status of Kibana"
time="2020-05-19T09:22:00Z" level=info msg="Kibana status successfully updated"
time="2020-05-19T09:22:00Z" level=info msg="Updating status of Kibana"
time="2020-05-19T09:22:00Z" level=info msg="Kibana status successfully updated"
time="2020-05-19T09:22:00Z" level=info msg="Updating status of Kibana"
time="2020-05-19T09:22:00Z" level=info msg="Kibana status successfully updated"
time="2020-05-19T09:22:02Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.27.141:9200: connect: connection refused\r\n"
time="2020-05-19T09:22:05Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.27.141:9200: connect: connection refused\r\n"
time="2020-05-19T09:22:30Z" level=info msg="Updating status of Kibana"
time="2020-05-19T09:22:30Z" level=info msg="Kibana status successfully updated"
time="2020-05-19T09:22:30Z" level=info msg="Kibana status successfully updated"
time="2020-05-19T09:22:52Z" level=warning msg="Unable to evaluate the number of replicas for index \"results\": Open Distro not initialized. cluster: elasticsearch, namespace: openshift-logging "
time="2020-05-19T09:22:52Z" level=error msg="Unable to evaluate number of replicas for index"
E0519 09:22:52.215518       1 runtime.go:69] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x14610e0), concrete:(*runtime._type)(0x1417700), asserted:(*runtime._type)(0x14677a0), missingMethod:""} (interface conversion: interface {} is string, not map[string]interface {})
/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/panic.go:679
/usr/local/go/src/runtime/iface.go:255
/go/src/github.com/openshift/elasticsearch-operator/pkg/elasticsearch/indices.go:98
/go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:156
/go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:48
/go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113
/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1357
panic: interface conversion: interface {} is string, not map[string]interface {} [recovered]
	panic: interface conversion: interface {} is string, not map[string]interface {}

goroutine 622 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
panic(0x14a1ae0, 0xc00203c840)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/openshift/elasticsearch-operator/pkg/elasticsearch.(*esClient).AddAliasForOldIndices(0xc0006ceb00, 0xc002049d70)
	/go/src/github.com/openshift/elasticsearch-operator/pkg/elasticsearch/indices.go:98 +0x62a
github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ElasticsearchRequest).CreateOrUpdateElasticsearchCluster(0xc000cbba80, 0x0, 0x0)
	/go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:156 +0xcdf
github.com/openshift/elasticsearch-operator/pkg/k8shandler.Reconcile(0xc000556240, 0x18c54e0, 0xc000a22f60, 0xc003a104e0, 0x11)
	/go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:48 +0x382
github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile(0xc0008e64a0, 0xc003a104e0, 0x11, 0xc00395df60, 0xd, 0xc000853ee8, 0xc00004cdf8, 0xc00004c000, 0xc000853e10)
	/go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 +0x20e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00030eaa0, 0x1000000010000)
	/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x20a
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1()
	/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0084922f0)
	/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0084922f0, 0x3b9aca00, 0x0, 0x1, 0xc0009480c0)
	/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0084922f0, 0x3b9aca00, 0xc0009480c0)
	/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x32e

$ oc get pod
NAME                                     READY   STATUS    RESTARTS   AGE
elasticsearch-operator-77d954ff8-7wlxf   1/1     Running   1          2m23s


Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. deploy CLO and EO
2. create clusterlogging instance with:
apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: "openshift-logging"
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    retentionPolicy: 
      application:
        maxAge: 1d
      infra:
        maxAge: 7d
    elasticsearch:
      nodeCount: 3
      redundancyPolicy: "SingleRedundancy"
      resources:
        requests:
          memory: "4Gi"
      storage:
        storageClassName: "gp2"
        size: "20Gi"
  visualization:
    type: "kibana"
    kibana:
      replicas: 1
  collection:
    logs:
      type: "fluentd"
      fluentd: {}

3. check EO

Actual results:


Expected results:


Additional info:

Comment 1 IgorKarpukhin 2020-05-27 15:41:34 UTC
@Qiaoling Tang fix has been merged: https://github.com/openshift/elasticsearch-operator/pull/362. Please test

Comment 2 Anping Li 2020-05-30 12:54:30 UTC
No such error when using csv:clusterlogging.4.5.0-202005291637 and elasticsearch-operator.4.5.0-202005291637

Comment 4 errata-xmlrpc 2020-07-13 17:40:02 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.