Bug 1833611
| Summary: | Kibana deployment doesn't appear after creating clusterlogging instance successfully | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Qiaoling Tang <qitang> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED ERRATA | QA Contact: | Qiaoling Tang <qitang> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 4.5 | CC: | aos-bugs, ikarpukh |
| Target Milestone: | --- | Keywords: | Regression, TestBlocker |
| Target Release: | 4.5.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-07-13 17:36:32 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 1833142 | ||
| Bug Blocks: | |||
Anping Li it's not working because we recently renamed Kibana CR from "instance" to "kibana". Changes for EO were merged, but the CLO's changes are not merged yet: https://github.com/openshift/cluster-logging-operator/pull/508 OK, got it. But it blocks our testing, we can't test the cases relate to Kibana. Should be resolved by https://bugzilla.redhat.com/show_bug.cgi?id=1833142 Verified with images from 4.5.0-0.ci-2020-05-11-212141. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |
Description of problem: The deployment/kibana doesn't appear after creating clusterlogging instance successfully. $ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-84bc65444b-nvcjl 1/1 Running 0 19m elasticsearch-cdm-27y76zid-1-b857858c4-vqnqz 2/2 Running 0 18m elasticsearch-delete-app-1588995000-4jnpb 0/1 Completed 0 3m6s elasticsearch-delete-infra-1588995000-rmbz7 0/1 Completed 0 3m6s elasticsearch-rollover-app-1588995000-dtxk6 0/1 Completed 0 3m6s elasticsearch-rollover-infra-1588995000-lqqst 0/1 Completed 0 3m6s fluentd-2zpkc 1/1 Running 0 18m fluentd-78wh7 1/1 Running 0 18m fluentd-b4bfh 1/1 Running 0 18m fluentd-bhxxb 1/1 Running 0 18m fluentd-j9sr5 1/1 Running 0 18m fluentd-nkzgt 1/1 Running 0 18m $ oc get kibana NAME AGE instance 20m $ oc get deploy NAME READY UP-TO-DATE AVAILABLE AGE cluster-logging-operator 1/1 1 1 21m elasticsearch-cdm-27y76zid-1 1/1 1 1 20m The elasticsearch-operator restarted after creating clusterlogging instance, logs in the EO: $ oc logs elasticsearch-operator-bdff87c9f-cpxgd -f {"level":"info","ts":1588994060.720107,"logger":"cmd","msg":"Go Version: go1.13.8"} {"level":"info","ts":1588994060.720128,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1588994060.7201316,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"} {"level":"info","ts":1588994060.7210255,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1588994060.8891356,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1588994060.8956196,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1588994061.1049066,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1588994061.1058147,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1060693,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1064591,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1066422,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.1068423,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994061.2882035,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"elasticsearch-operator","Service.Namespace":"openshift-operators-redhat"} {"level":"info","ts":1588994061.2882326,"logger":"cmd","msg":"This operator no longer honors the image specified by the custom resources so that it is able to properly coordinate the configuration with the image."} {"level":"info","ts":1588994061.2882385,"logger":"cmd","msg":"Starting the Cmd."} W0509 03:14:21.370516 1 reflector.go:270] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:126: watch of *v1.Kibana ended with: too old resource version: 95197 (95198) {"level":"info","ts":1588994062.0884562,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trustedcabundle-controller"} {"level":"info","ts":1588994062.0884998,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"} {"level":"info","ts":1588994062.0884726,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"proxyconfig-controller"} {"level":"info","ts":1588994062.088465,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"} {"level":"info","ts":1588994062.0884757,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibanasecret-controller"} {"level":"info","ts":1588994062.1886232,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibanasecret-controller","worker count":1} {"level":"info","ts":1588994062.1886475,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"proxyconfig-controller","worker count":1} {"level":"info","ts":1588994062.1886153,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1} {"level":"info","ts":1588994062.1886415,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trustedcabundle-controller","worker count":1} {"level":"info","ts":1588994062.188637,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1} time="2020-05-09T03:14:34Z" level=warning msg="unable to get cluster node count. E: Get https://elasticsearch.openshift-logging.svc:9200/_cluster/health: dial tcp 172.30.36.45:9200: connect: connection refused\r\n" E0509 03:15:41.909097 1 runtime.go:69] Observed a panic: &runtime.TypeAssertionError{_interface:(*runtime._type)(0x1460ae0), concrete:(*runtime._type)(0x1417300), asserted:(*runtime._type)(0x14671a0), missingMethod:""} (interface conversion: interface {} is string, not map[string]interface {}) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/panic.go:679 /usr/local/go/src/runtime/iface.go:255 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/elasticsearch.go:478 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:159 /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:45 /go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /usr/local/go/src/runtime/asm_amd64.s:1357 panic: interface conversion: interface {} is string, not map[string]interface {} [recovered] panic: interface conversion: interface {} is string, not map[string]interface {} goroutine 679 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105 panic(0x14a14e0, 0xc005f82e10) /usr/local/go/src/runtime/panic.go:679 +0x1b2 github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ElasticsearchRequest).AddAliasForOldIndices(0xc005e3ba88, 0xc005ecdc50) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/elasticsearch.go:478 +0x632 github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ElasticsearchRequest).CreateOrUpdateElasticsearchCluster(0xc005e3ba88, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:159 +0xcea github.com/openshift/elasticsearch-operator/pkg/k8shandler.Reconcile(0xc0004c8240, 0x18c4960, 0xc00010e720, 0xc005e2fa80, 0x11) /go/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/reconciler.go:45 +0x2a4 github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile(0xc0003256a0, 0xc005e2fa80, 0x11, 0xc005e4a4e0, 0xd, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/pkg/controller/elasticsearch/controller.go:113 +0x20e sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000255220, 0x44dc00) /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215 +0x20a sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1() /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36 k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc003adb2b0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x5e k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc003adb2b0, 0x3b9aca00, 0x0, 0x1, 0xc0003369c0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8 k8s.io/apimachinery/pkg/util/wait.Until(0xc003adb2b0, 0x3b9aca00, 0xc0003369c0) /go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start /go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x32e $ oc get pod -n openshift-operators-redhat NAME READY STATUS RESTARTS AGE elasticsearch-operator-bdff87c9f-cpxgd 1/1 Running 1 25m Logs after the pod restarted: $ oc logs elasticsearch-operator-bdff87c9f-cpxgd -f {"level":"info","ts":1588994143.0450966,"logger":"cmd","msg":"Go Version: go1.13.8"} {"level":"info","ts":1588994143.0451303,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1588994143.0451362,"logger":"cmd","msg":"Version of operator-sdk: v0.8.2"} {"level":"info","ts":1588994143.0458772,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1588994143.280513,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."} {"level":"info","ts":1588994143.280539,"logger":"leader","msg":"Continuing as the leader."} {"level":"info","ts":1588994143.4577036,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1588994143.4586358,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibana-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.4590085,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"elasticsearch-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.4592705,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"proxyconfig-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.459453,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"kibanasecret-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.4596634,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"trustedcabundle-controller","source":"kind source: /, Kind="} {"level":"info","ts":1588994143.6850328,"logger":"cmd","msg":"This operator no longer honors the image specified by the custom resources so that it is able to properly coordinate the configuration with the image."} {"level":"info","ts":1588994143.68506,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1588994144.485263,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trustedcabundle-controller"} {"level":"info","ts":1588994144.4853323,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"proxyconfig-controller"} {"level":"info","ts":1588994144.485337,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"} {"level":"info","ts":1588994144.4853513,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibanasecret-controller"} {"level":"info","ts":1588994144.4853578,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"} {"level":"info","ts":1588994144.585412,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"proxyconfig-controller","worker count":1} {"level":"info","ts":1588994144.5854142,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trustedcabundle-controller","worker count":1} {"level":"info","ts":1588994144.585468,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1} {"level":"info","ts":1588994144.5854774,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1} {"level":"info","ts":1588994144.585535,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibanasecret-controller","worker count":1} time="2020-05-09T03:15:53Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:16:23Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:16:23Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:16:29Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:16:59Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:16:59Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:17:03Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:17:33Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:17:33Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:17:38Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:18:08Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:18:08Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:18:12Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:18:42Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:18:42Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:18:47Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:19:17Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:19:17Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:19:21Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" time="2020-05-09T03:19:51Z" level=info msg="Timed out waiting for node elasticsearch-cdm-27y76zid-1 to rollout" time="2020-05-09T03:19:51Z" level=warning msg="Error occurred while updating node elasticsearch-cdm-27y76zid-1: timed out waiting for the condition" time="2020-05-09T03:19:56Z" level=warning msg="Unable to perform synchronized flush: Failed to flush 1 shards in preparation for cluster restart" Version-Release number of selected component (if applicable): Logging images are from 4.5.0-0.ci-2020-05-08-221527, manifests are copied from the master branch with the latest code. How reproducible: Always Steps to Reproduce: 1. deploy CLO and EO 2. create clusterlogging instance 3. check pods in the openshift-logging and openshift-operators-redhat namespaces Actual results: Expected results: Additional info: I didn't hit this issue yesterday, the manifests are updated with the master branch.