Hide Forgot
Description of problem: Deploy logging with 3 es nodes in the cluster, after all pods in running status, execute `oc set env deployment/elasticsearch-clientdatamaster-0-1 REPLICA_SHARDS=1`, the check pod in openshift-logging namespace, the elasticsearch-operator pod is in "Error" status, and can't start anymore. The elasticsearch-operator pod logs show: panic: runtime error: index out of range [recovered] panic: runtime error: index out of range $ oc set env deployment/elasticsearch-clientdatamaster-0-1 REPLICA_SHARDS=1 deployment.extensions/elasticsearch-clientdatamaster-0-1 updated $ oc get pod NAME READY STATUS RESTARTS AGE cluster-logging-operator-8866ff9c8-nq68f 1/1 Running 0 9m elasticsearch-clientdatamaster-0-1-84d764899d-mn4zq 1/1 Running 0 7m elasticsearch-clientdatamaster-0-2-56984bb76c-q2p25 1/1 Running 0 7m elasticsearch-clientdatamaster-0-3-7cd67f75dd-j5fgc 1/1 Running 0 7m elasticsearch-operator-86599f8849-ptstf 0/1 Error 1 9m $ oc logs elasticsearch-operator-86599f8849-ptstf time="2018-12-26T08:11:40Z" level=info msg="Go Version: go1.10.3" time="2018-12-26T08:11:40Z" level=info msg="Go OS/Arch: linux/amd64" time="2018-12-26T08:11:40Z" level=info msg="operator-sdk Version: 0.0.7" time="2018-12-26T08:11:40Z" level=info msg="Metrics service elasticsearch-operator created" time="2018-12-26T08:11:40Z" level=info msg="Watching logging.openshift.io/v1alpha1, Elasticsearch, openshift-logging, 5000000000" E1226 08:11:41.889292 1 runtime.go:66] Observed a panic: "index out of range" (runtime error: index out of range) /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72 /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65 /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51 /usr/local/go/src/runtime/asm_amd64.s:573 /usr/local/go/src/runtime/panic.go:502 /usr/local/go/src/runtime/panic.go:28 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/deployment.go:61 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/desirednodestate.go:312 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:169 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:50 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/stub/handler.go:68 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/stub/handler.go:29 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer-sync.go:88 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer-sync.go:52 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer-sync.go:36 /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:98 /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134 /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88 /usr/local/go/src/runtime/asm_amd64.s:2361 panic: runtime error: index out of range [recovered] panic: runtime error: index out of range goroutine 179 [running]: k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107 panic(0x10eee80, 0x1a075e0) /usr/local/go/src/runtime/panic.go:502 +0x229 github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*deploymentNode).isDifferent(0xc42033ec00, 0xc420588b40, 0x0, 0x5, 0x0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/deployment.go:61 +0xd7d github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*desiredNodeState).IsUpdateNeeded(0xc420588b40, 0x0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/desirednodestate.go:312 +0x79 github.com/openshift/elasticsearch-operator/pkg/k8shandler.(*ClusterState).getRequiredAction(0xc420744f80, 0xc42068a000, 0x0, 0x1252867, 0xd, 0xc420735fc0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:169 +0x125 github.com/openshift/elasticsearch-operator/pkg/k8shandler.CreateOrUpdateElasticsearchCluster(0xc42068a000, 0x1252867, 0xd, 0x1252867, 0xd, 0x8, 0xc4203fcc30) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/k8shandler/cluster.go:50 +0x1cd github.com/openshift/elasticsearch-operator/pkg/stub.Reconcile(0xc42068a000, 0xc4203fc480, 0xc42000e540) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/stub/handler.go:68 +0x447 github.com/openshift/elasticsearch-operator/pkg/stub.(*Handler).Handle(0x1a3b8f8, 0x1361820, 0xc4200ba018, 0x134cc80, 0xc42068a000, 0x42ac00, 0x0, 0x0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/openshift/elasticsearch-operator/pkg/stub/handler.go:29 +0x6d github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).sync(0xc4201164d0, 0xc420660120, 0x1f, 0x10916e0, 0xc420374cf0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer-sync.go:88 +0x12d github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).processNextItem(0xc4201164d0, 0xc420300900) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer-sync.go:52 +0xd2 github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).runWorker(0xc4201164d0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer-sync.go:36 +0x2b github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).(github.com/operator-framework/operator-sdk/pkg/sdk.runWorker)-fm() /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:98 +0x2a k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc42026db50) /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc42026db50, 0x3b9aca00, 0x0, 0x1, 0x0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd k8s.io/apimachinery/pkg/util/wait.Until(0xc42026db50, 0x3b9aca00, 0x0) /go/src/github.com/openshift/elasticsearch-operator/_output/src/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by github.com/operator-framework/operator-sdk/pkg/sdk.(*informer).Run /go/src/github.com/openshift/elasticsearch-operator/_output/src/github.com/operator-framework/operator-sdk/pkg/sdk/informer.go:98 +0x209 Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.alpha-2018-12-25-225820 True False 6h Cluster version is 4.0.0-0.alpha-2018-12-25-225820 $ ./bin/openshift-install version ./bin/openshift-install v0.8.0-master-2-g5e7b36d6351c9cc773f1dadc64abf9d7041151b1 How reproducible: Always Steps to Reproduce: 1. Deploy logging with 3 es nodes in the cluster 2. execute `oc set env deployment/elasticsearch-clientdatamaster-0-1 REPLICA_SHARDS=1` 3. get pod in openshift-logging namespace Actual results: elasticsearch-operator in "Error" status and can't become "Running" anymore. Expected results: Additional info:
This is indeed a change we should gracefully handle, but the proper way to modify is via the CR as documented here: https://github.com/openshift/cluster-logging-operator/pull/64/files#diff-76e731333fb756df3bff5ddb3b731c46R82 . The CR to support this will be available after the merge of https://github.com/openshift/cluster-logging-operator/pull/70
Verified in docker.io/openshift/origin-elasticsearch-operator@sha256:461d03c54f87ddc28846edbe78c8f3026a4508e3ab6c45c06be1ef9492f0be6e
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758