Bug 1837323
| Summary: | EO panic: interface conversion: interface {} is string, not map[string]interface {} [recovered] | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> |
| Component: | Logging | Assignee: | IgorKarpukhin <ikarpukh> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | high | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.5 | CC: | aos-bugs, ikarpukh, periklis |
| Target Milestone: | --- | ||
| Target Release: | 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: Response from EO API wasn't checked for nil/NULL values
Consequence: EO operator crashed
Fix: Added additional checks for the EO API response
Result: EO operator not crashing anymore
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-13 17:40:02 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
@Qiaoling Tang fix has been merged: https://github.com/openshift/elasticsearch-operator/pull/362. Please test No such error when using csv:clusterlogging.4.5.0-202005291637 and elasticsearch-operator.4.5.0-202005291637 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |
Description of problem: after creating clusterlogging instance, the EO restarts, logs: {"level":"info","ts":1589880039.5820286,"logger":"cmd","msg":"Go Version: go1.13.8"} {"level":"info","ts":1589880039.582052,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1589880039.582057,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"} {"level":"info","ts":1589880039.5837452,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1589880039.7752895,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1589880039.7834234,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1589880039.9057093,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1589880039.9062088,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="} {"level":"info","ts":1589880039.9064121,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="} {"level":"info","ts":1589880039.906656,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="} {"level":"info","ts":1589880039.9067993,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="} {"level":"info","ts":1589880039.9069505,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="} {"level":"info","ts":1589880040.0523393,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"elasticsearch-operator","Service.Namespace":"openshift-operators-redhat"} {"level":"info","ts":1589880040.0523727,"logger":"cmd","msg":"This operator no longer honors the image specified by the custom resources so that it is able to properly coordinate the configuration with the image."} {"level":"info","ts":1589880040.0523794,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1589880041.2526214,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibanasecret-controller"} {"level":"info","ts":1589880041.2526338,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"proxyconfig-controller"} {"level":"info","ts":1589880041.2526128,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trustedcabundle-controller"} {"level":"info","ts":1589880041.2526255,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"} {"level":"info","ts":1589880041.2526128,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"} {"level":"info","ts":1589880041.3528018,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trustedcabundle-controller","worker count":1} {"level":"info","ts":1589880041.352811,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1} {"level":"info","ts":1589880041.352802,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibanasecret-controller","worker count":1} {"level":"info","ts":1589880041.3528116,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"proxyconfig-controller","worker count":1} {"level":"info","ts":1589880041.3528316,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1} time="2020-05-19T09:21:59Z" level=error msg="Operator unable to read local file to get contents: open /tmp/ocp-eo/ca.crt: no such file or directory" time="2020-05-19T09:21:59Z" level=error msg="Operator unable to read local file to get contents: open /tmp/ocp-eo/ca.crt: no such file or directory" {"level":"error","ts":1589880119.4201286,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"kibana-controller","request":"openshift-logging/kibana","error":"Did not receive hashvalue for trusted CA value","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} time="2020-05-19T09:22:00Z" level=info msg="Updating status of Kibana" time="2020-05-19T09:22:00Z" level=info msg="Kibana status successfully updated" time="2020-05-19T09:22:00Z" level=info msg="Updating status of Kibana" time="2020-05-19T09:22:00Z" level=info msg="Kibana status successfully updated" time="2020-05-19T09:22:00Z" level=info msg="Updating status of Kibana" time="2020-05-19T09:22:00Z" level=info msg="Kibana status successfully updated" time="2020-05-19T09:22:02Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.27.141:9200: connect: connection refused\r\n" time="2020-05-19T09:22:05Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.27.141:9200: connect: connection refused\r\n" time="2020-05-19T09:22:30Z" level=info msg="Updating status of Kibana" time="2020-05-19T09:22:30Z" level=info msg="Kibana status successfully updated" time="2020-05-19T09:22:30Z" level=info msg="Kibana status successfully updated" time="2020-05-19T09:22:52Z" level=warning msg="Unable to evaluate the number of replicas for index \"results\": Open Distro not initialized. cluster: elasticsearch, namespace: openshift-logging " time="2020-05-19T09:22:52Z" level=error msg="Unable to evaluate number of replicas for index" E0519 09:22:52.215518 1 runtime.go:69] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x14610e0), concrete:(*runtime._type)(0x1417700), asserted:(*runtime._type)(0x14677a0), missingMethod:""} (interface conversion: interface {} is string, not map[string]interface {}) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/panic.go:679 /usr/local/go/src/runtime/iface.go:255 /go/src/github.com/openshift/elasticsearch-operator/pkg/elasticsearch/indices.go:98 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:156 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:48 /go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /usr/local/go/src/runtime/asm_amd64.s:1357 panic: interface conversion: interface {} is string, not map[string]interface {} [recovered] panic: interface conversion: interface {} is string, not map[string]interface {} goroutine 622 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105 panic(0x14a1ae0, 0xc00203c840) /usr/local/go/src/runtime/panic.go:679 +0x1b2 github.com/openshift/elasticsearch-operator/pkg/elasticsearch.(*esClient).AddAliasForOldIndices(0xc0006ceb00, 0xc002049d70) /go/src/github.com/openshift/elasticsearch-operator/pkg/elasticsearch/indices.go:98 +0x62a github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ElasticsearchRequest).CreateOrUpdateElasticsearchCluster(0xc000cbba80, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:156 +0xcdf github.com/openshift/elasticsearch-operator/pkg/k8shandler.Reconcile(0xc000556240, 0x18c54e0, 0xc000a22f60, 0xc003a104e0, 0x11) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:48 +0x382 github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile(0xc0008e64a0, 0xc003a104e0, 0x11, 0xc00395df60, 0xd, 0xc000853ee8, 0xc00004cdf8, 0xc00004c000, 0xc000853e10) /go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 +0x20e sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00030eaa0, 0x1000000010000) /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x20a sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1() /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36 k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0084922f0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0084922f0, 0x3b9aca00, 0x0, 0x1, 0xc0009480c0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8 k8s.io/apimachinery/pkg/util/wait.Until(0xc0084922f0, 0x3b9aca00, 0xc0009480c0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x32e $ oc get pod NAME READY STATUS RESTARTS AGE elasticsearch-operator-77d954ff8-7wlxf 1/1 Running 1 2m23s Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. deploy CLO and EO 2. create clusterlogging instance with: apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: "openshift-logging" spec: managementState: "Managed" logStore: type: "elasticsearch" retentionPolicy: application: maxAge: 1d infra: maxAge: 7d elasticsearch: nodeCount: 3 redundancyPolicy: "SingleRedundancy" resources: requests: memory: "4Gi" storage: storageClassName: "gp2" size: "20Gi" visualization: type: "kibana" kibana: replicas: 1 collection: logs: type: "fluentd" fluentd: {} 3. check EO Actual results: Expected results: Additional info: