Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1891019

Summary: Elasticsearch in crashloopback with error: " ElasticsearchException[searchguard.ssl.transport.keystore_filepath or searchguard.ssl.transport.pemkey_filepath must be set if transport ssl is reqested"
Product: OpenShift Container Platform Reporter: Oscar Casal Sanchez <ocasalsa>
Component: LoggingAssignee: ewolinet
Status: CLOSED ERRATA QA Contact: Anping Li <anli>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.5CC: aos-bugs, braander, dtarabor, ewolinet, periklis
Target Milestone: ---Flags: ewolinet: needinfo-
Target Release: 4.5.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: logging-exploration
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-11-17 16:04:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1892005    
Bug Blocks:    

Description Oscar Casal Sanchez 2020-10-23 14:45:48 UTC
[Description of problem]
After upgrading the cluster and the logging stack to 4.5. The elasticsearch pods are in crashloopback mode with the next error:

~~~
2020-10-23T13:44:16.681587427Z [2020-10-23T13:44:16,681][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
2020-10-23T13:44:16.755981999Z [2020-10-23T13:44:16,750][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-cdm-56udt08p-1] uncaught exception in thread [main]
2020-10-23T13:44:16.755981999Z org.elasticsearch.bootstrap.StartupException: ElasticsearchException[Failed to load plugin class [io.fabric8.elasticsearch.plugin.OpenShiftElasticSearchPlugin]]; nested: InvocationTargetException; nested: ElasticsearchException[searchguard.ssl.transport.keystore_filepath or searchguard.ssl.transport.pemkey_filepath must be set if transport ssl is reqested.];
~~~


Version-Release number of selected component (if applicable):

OCP 4.5

[How reproducible]
Always in customer suffering this issue, but not able to see the RCA

Comment 17 Oscar Casal Sanchez 2020-11-04 07:27:35 UTC
Hello,

The issue with kibana was different and it's fixed now. I had observed this error in the logs:

~~~
cluster-logging/eo/elasticsearch-operator.logs:{"level":"error","ts":1604322904.3834665,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"kibanasecret-controller","request":"openshift-logging/kibana","error":"failed re-indexing `.kibana` into `.kibana-6`: failed to set index \".kibana\" to read only: failed to update index settings for \".kibana\". Error code: true, map[results:Cannot authenticate user because admin user is not permitted to login via HTTP]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/openshift/elasticsearch-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
~~~

As you can see, it's the indice .kibana and it's saying "failed re-indexing `.kibana` into `.kibana-6`". After agreeing with customer to delete the .kibana indice it was recreated and kibana CR and pods appear.


@ewolinet, can be something done from engineering side with the errors that we have done? We have seen that a partial upgrade was done were the ES images was for ES5, but the configmap and other things were for ES6 as expected ( as you commented in c#6). Also, something happened with the Kibana indice where it was .kibana and it was not migrated.

In case that from engineering side is not needed to do to verify the previously commented, I believe that we could close the Bugzilla since in the customer side everything is working now.

Comment 20 Anping Li 2020-11-13 01:46:49 UTC
No regression issue(I couldn't reproduce this problem).move to verified.

Comment 22 errata-xmlrpc 2020-11-17 16:04:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.19 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:5053

Comment 23 Red Hat Bugzilla 2023-09-18 00:23:08 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days