Description of problem: The deployment/kibana doesn't appear after creating clusterlogging instance successfully. $ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-84bc65444b-nvcjl 1/1 Running 0 19m elasticsearch-cdm-27y76zid-1-b857858c4-vqnqz 2/2 Running 0 18m elasticsearch-delete-app-1588995000-4jnpb 0/1 Completed 0 3m6s elasticsearch-delete-infra-1588995000-rmbz7 0/1 Completed 0 3m6s elasticsearch-rollover-app-1588995000-dtxk6 0/1 Completed 0 3m6s elasticsearch-rollover-infra-1588995000-lqqst 0/1 Completed 0 3m6s fluentd-2zpkc 1/1 Running 0 18m fluentd-78wh7 1/1 Running 0 18m fluentd-b4bfh 1/1 Running 0 18m fluentd-bhxxb 1/1 Running 0 18m fluentd-j9sr5 1/1 Running 0 18m fluentd-nkzgt 1/1 Running 0 18m $ oc get kibana NAME AGE instance 20m $ oc get deploy NAME READY UP-TO-DATE AVAILABLE AGE cluster-logging-operator 1/1 1 1 21m elasticsearch-cdm-27y76zid-1 1/1 1 1 20m The elasticsearch-operator restarted after creating clusterlogging instance, logs in the EO: $ oc logs elasticsearch-operator-bdff87c9f-cpxgd -f {"level":"info","ts":1588994060.720107,"logger":"cmd","msg":"Go Version: go1.13.8"} {"level":"info","ts":1588994060.720128,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1588994060.7201316,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"} {"level":"info","ts":1588994060.7210255,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1588994060.8891356,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1588994060.8956196,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1588994061.1049066,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1588994061.1058147,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1060693,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1064591,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1066422,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1068423,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.2882035,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"elasticsearch-operator","Service.Namespace":"openshift-operators-redhat"} {"level":"info","ts":1588994061.2882326,"logger":"cmd","msg":"This operator no longer honors the image specified by the custom resources so that it is able to properly coordinate the configuration with the image."} {"level":"info","ts":1588994061.2882385,"logger":"cmd","msg":"Starting the Cmd."} W0509 03:14:21.370516 1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1.Kibana ended with: too old resource version: 95197 (95198) {"level":"info","ts":1588994062.0884562,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trustedcabundle-controller"} {"level":"info","ts":1588994062.0884998,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"} {"level":"info","ts":1588994062.0884726,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"proxyconfig-controller"} {"level":"info","ts":1588994062.088465,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"} {"level":"info","ts":1588994062.0884757,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibanasecret-controller"} {"level":"info","ts":1588994062.1886232,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibanasecret-controller","worker count":1} {"level":"info","ts":1588994062.1886475,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"proxyconfig-controller","worker count":1} {"level":"info","ts":1588994062.1886153,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1} {"level":"info","ts":1588994062.1886415,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trustedcabundle-controller","worker count":1} {"level":"info","ts":1588994062.188637,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1} time="2020-05-09T03:14:34Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.36.45:9200: connect: connection refused\r\n" E0509 03:15:41.909097 1 runtime.go:69] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x1460ae0), concrete:(*runtime._type)(0x1417300), asserted:(*runtime._type)(0x14671a0), missingMethod:""} (interface conversion: interface {} is string, not map[string]interface {}) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/panic.go:679 /usr/local/go/src/runtime/iface.go:255 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/elasticsearch.go:478 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:159 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:45 /go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /usr/local/go/src/runtime/asm_amd64.s:1357 panic: interface conversion: interface {} is string, not map[string]interface {} [recovered] panic: interface conversion: interface {} is string, not map[string]interface {} goroutine 679 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105 panic(0x14a14e0, 0xc005f82e10) /usr/local/go/src/runtime/panic.go:679 +0x1b2 github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ElasticsearchRequest).AddAliasForOldIndices(0xc005e3ba88, 0xc005ecdc50) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/elasticsearch.go:478 +0x632 github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ElasticsearchRequest).CreateOrUpdateElasticsearchCluster(0xc005e3ba88, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:159 +0xcea github.com/openshift/elasticsearch-operator/pkg/k8shandler.Reconcile(0xc0004c8240, 0x18c4960, 0xc00010e720, 0xc005e2fa80, 0x11) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:45 +0x2a4 github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile(0xc0003256a0, 0xc005e2fa80, 0x11, 0xc005e4a4e0, 0xd, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 +0x20e sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000255220, 0x44dc00) /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x20a sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1() /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36 k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc003adb2b0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003adb2b0, 0x3b9aca00, 0x0, 0x1, 0xc0003369c0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8 k8s.io/apimachinery/pkg/util/wait.Until(0xc003adb2b0, 0x3b9aca00, 0xc0003369c0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x32e $ oc get pod -n openshift-operators-redhat NAME READY STATUS RESTARTS AGE elasticsearch-operator-bdff87c9f-cpxgd 1/1 Running 1 25m Logs after the pod restarted: $ oc logs elasticsearch-operator-bdff87c9f-cpxgd -f {"level":"info","ts":1588994143.0450966,"logger":"cmd","msg":"Go Version: go1.13.8"} {"level":"info","ts":1588994143.0451303,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1588994143.0451362,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"} {"level":"info","ts":1588994143.0458772,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1588994143.280513,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."} {"level":"info","ts":1588994143.280539,"logger":"leader","msg":"Continuing as the leader."} {"level":"info","ts":1588994143.4577036,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1588994143.4586358,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.4590085,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.4592705,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.459453,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.4596634,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.6850328,"logger":"cmd","msg":"This operator no longer honors the image specified by the custom resources so that it is able to properly coordinate the configuration with the image."} {"level":"info","ts":1588994143.68506,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1588994144.485263,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trustedcabundle-controller"} {"level":"info","ts":1588994144.4853323,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"proxyconfig-controller"} {"level":"info","ts":1588994144.485337,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"} {"level":"info","ts":1588994144.4853513,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibanasecret-controller"} {"level":"info","ts":1588994144.4853578,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"} {"level":"info","ts":1588994144.585412,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"proxyconfig-controller","worker count":1} {"level":"info","ts":1588994144.5854142,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trustedcabundle-controller","worker count":1} {"level":"info","ts":1588994144.585468,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1} {"level":"info","ts":1588994144.5854774,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1} {"level":"info","ts":1588994144.585535,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibanasecret-controller","worker count":1} time="2020-05-09T03:15:53Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:16:23Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:16:23Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:16:29Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:16:59Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:16:59Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:17:03Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:17:33Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:17:33Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:17:38Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:18:08Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:18:08Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:18:12Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:18:42Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:18:42Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:18:47Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:19:17Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:19:17Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:19:21Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:19:51Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:19:51Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:19:56Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" Version-Release number of selected component (if applicable): Logging images are from 4.5.0-0.ci-2020-05-08-221527, manifests are copied from the master branch with the latest code. How reproducible: Always Steps to Reproduce: 1. deploy CLO and EO 2. create clusterlogging instance 3. check pods in the openshift-logging and openshift-operators-redhat namespaces Actual results: Expected results: Additional info: I didn't hit this issue yesterday, the manifests are updated with the master branch.
Anping Li it's not working because we recently renamed Kibana CR from "instance" to "kibana". Changes for EO were merged, but the CLO's changes are not merged yet: https://github.com/openshift/cluster-logging-operator/pull/508
OK, got it. But it blocks our testing, we can't test the cases relate to Kibana.
Should be resolved by https://bugzilla.redhat.com/show_bug.cgi?id=1833142
Verified with images from 4.5.0-0.ci-2020-05-11-212141.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409