Bug 1466962
| Summary: | Elasticsearch logging dead with exception: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Peter Portante <pportant> |
| Component: | Logging | Assignee: | Jeff Cantrill <jcantril> |
| Status: | CLOSED DUPLICATE | QA Contact: | Xia Zhao <xiazhao> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.5.1 | CC: | aos-bugs, jgoulding, jwozniak, rmeggins, wgordon |
| Target Milestone: | --- | Keywords: | OpsBlocker |
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-07-05 11:51:09 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
*** Bug 1466963 has been marked as a duplicate of this bug. *** Also, the sgadmin.sh script is failing with: Cannot retrieve cluster state due to: class_cast_exception: null. This is not an error, will keep on trying ... * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates) * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file) > On a 3.5 install from the OpenShift Dedicated team Am I correct to assume this means they are using downstream images from brew? If yes, then I think we could inspect our images and synchronize upstream and downstream. > java.lang.ClassCastException: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User When casting an instance of a class to apparently same class throws an exception, it could mean that this class was loaded by two different classloaders. And inspecting the ES downstream image, it looks like we are installing the searchguard plugin twice which could explain this exception. 1) separately here: http://pkgs.devel.redhat.com/cgit/rpms/logging-elasticsearch-docker/tree/install.sh.prod?h=rhaos-3.5-rhel-7#n27 2) as a part of openshift-elasticsearch-plugin: http://pkgs.devel.redhat.com/cgit/rpms/logging-elasticsearch-docker/tree/install.sh.prod?h=rhaos-3.5-rhel-7#n36 The searchguard became part of openshift-elasticsearch-plugin a few months ago. https://github.com/fabric8io/openshift-elasticsearch-plugin/issues/78 Closing this one as a duplicate since https://bugzilla.redhat.com/show_bug.cgi?id=1467572#c5 identifies the latest images work properly. My suspicion is that tested images were some mix of the originals that deployed SG as a plugin and the change to the openshift-elasticsearch-plugin that embed SG *** This bug has been marked as a duplicate of bug 1467572 *** |
On a 3.5 install from the OpenShift Dedicated team, we see the following stack trace which is preventing operation of Elasticsearch: [2017-06-30 17:38:57,598][WARN ][rest.suppressed ] path: /_cluster/health, params: {pretty=true} java.lang.ClassCastException: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:87) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52) at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:88) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:853) at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:873) at org.elasticsearch.rest.action.admin.cluster.health.RestClusterHealthAction.handleRequest(RestClusterHealthAction.java:62) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54) at org.elasticsearch.rest.RestController.executeHandler(RestController.java:198) at org.elasticsearch.rest.RestController$RestHandlerFilter.process(RestController.java:280) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:261) at io.fabric8.elasticsearch.plugin.KibanaUserReindexFilter.process(KibanaUserReindexFilter.java:83) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at com.floragunn.searchguard.filter.SearchGuardRestFilter.process(SearchGuardRestFilter.java:65) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at com.floragunn.searchguard.filter.SearchGuardRestFilter.process(SearchGuardRestFilter.java:65) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.process(DynamicACLFilter.java:158) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:161) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451) at com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport.dispatchRequest(SearchGuardSSLNettyHttpServerTransport.java:162) at com.floragunn.searchguard.http.SearchGuardHttpServerTransport.dispatchRequest(SearchGuardHttpServerTransport.java:70) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)