On a 3.5 install from the OpenShift Dedicated team, we see the following stack trace which is preventing operation of Elasticsearch: [2017-06-30 17:38:57,598][WARN ][rest.suppressed ] path: /_cluster/health, params: {pretty=true} java.lang.ClassCastException: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:87) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:88) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:853) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:873) at org.elasticsearch.rest.action.admin.cluster.health.RestClusterHealthAction.handleRequest(RestClusterHealthAction.java:62) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54) at org.elasticsearch.rest.RestController.executeHandler(RestController.java:198) at org.elasticsearch.rest.RestController$RestHandlerFilter.process(RestController.java:280) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:261) at io.fabric8.elasticsearch.plugin.KibanaUserReindexFilter.process(KibanaUserReindexFilter.java:83) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at com.floragunn.searchguard.filter.SearchGuardRestFilter.process(SearchGuardRestFilter.java:65) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at com.floragunn.searchguard.filter.SearchGuardRestFilter.process(SearchGuardRestFilter.java:65) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.process(DynamicACLFilter.java:158) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:161) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451) at com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport.dispatchRequest(SearchGuardSSLNettyHttpServerTransport.java:162) at com.floragunn.searchguard.http.SearchGuardHttpServerTransport.dispatchRequest(SearchGuardHttpServerTransport.java:70) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
*** Bug 1466963 has been marked as a duplicate of this bug. ***
Also, the sgadmin.sh script is failing with: Cannot retrieve cluster state due to: class_cast_exception: null. This is not an error, will keep on trying ... * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates) * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
> On a 3.5 install from the OpenShift Dedicated team Am I correct to assume this means they are using downstream images from brew? If yes, then I think we could inspect our images and synchronize upstream and downstream. > java.lang.ClassCastException: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User When casting an instance of a class to apparently same class throws an exception, it could mean that this class was loaded by two different classloaders. And inspecting the ES downstream image, it looks like we are installing the searchguard plugin twice which could explain this exception. 1) separately here: http://pkgs.devel.redhat.com/cgit/rpms/logging-elasticsearch-docker/tree/install.sh.prod?h=rhaos-3.5-rhel-7#n27 2) as a part of openshift-elasticsearch-plugin: http://pkgs.devel.redhat.com/cgit/rpms/logging-elasticsearch-docker/tree/install.sh.prod?h=rhaos-3.5-rhel-7#n36 The searchguard became part of openshift-elasticsearch-plugin a few months ago. https://github.com/fabric8io/openshift-elasticsearch-plugin/issues/78
Closing this one as a duplicate since https://bugzilla.redhat.com/show_bug.cgi?id=1467572#c5 identifies the latest images work properly. My suspicion is that tested images were some mix of the originals that deployed SG as a plugin and the change to the openshift-elasticsearch-plugin that embed SG *** This bug has been marked as a duplicate of bug 1467572 ***