Bug 1466962 - Elasticsearch logging dead with exception: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User
Elasticsearch logging dead with exception: com.floragunn.searchguard.user.Use...
Status: CLOSED DUPLICATE of bug 1467572
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging (Show other bugs)
3.5.1
All All
unspecified Severity high
: ---
: ---
Assigned To: Jeff Cantrill
Xia Zhao
: OpsBlocker
: 1466963 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-30 19:11 EDT by Peter Portante
Modified: 2017-07-05 07:51 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-07-05 07:51:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Peter Portante 2017-06-30 19:11:25 EDT
On a 3.5 install from the OpenShift Dedicated team, we see the following stack trace which is preventing operation of Elasticsearch:

[2017-06-30 17:38:57,598][WARN ][rest.suppressed          ] path: /_cluster/health, params: {pretty=true}
java.lang.ClassCastException: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User
        at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:87)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:144)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
        at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:88)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:853)
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:873)
        at org.elasticsearch.rest.action.admin.cluster.health.RestClusterHealthAction.handleRequest(RestClusterHealthAction.java:62)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:198)
        at org.elasticsearch.rest.RestController$RestHandlerFilter.process(RestController.java:280)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:261)
        at io.fabric8.elasticsearch.plugin.KibanaUserReindexFilter.process(KibanaUserReindexFilter.java:83)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264)
        at com.floragunn.searchguard.filter.SearchGuardRestFilter.process(SearchGuardRestFilter.java:65)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264)
        at com.floragunn.searchguard.filter.SearchGuardRestFilter.process(SearchGuardRestFilter.java:65)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264)
        at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.process(DynamicACLFilter.java:158)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:161)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451)
        at com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport.dispatchRequest(SearchGuardSSLNettyHttpServerTransport.java:162)
        at com.floragunn.searchguard.http.SearchGuardHttpServerTransport.dispatchRequest(SearchGuardHttpServerTransport.java:70)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
        at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
Comment 1 Peter Portante 2017-06-30 19:53:49 EDT
*** Bug 1466963 has been marked as a duplicate of this bug. ***
Comment 2 Peter Portante 2017-07-03 11:39:50 EDT
Also, the sgadmin.sh script is failing with:

Cannot retrieve cluster state due to: class_cast_exception: null. This is not an error, will keep on trying ...
   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
Comment 3 Jan Wozniak 2017-07-04 10:09:51 EDT
> On a 3.5 install from the OpenShift Dedicated team

Am I correct to assume this means they are using downstream images from brew? If yes, then I think we could inspect our images and synchronize upstream and downstream.

> java.lang.ClassCastException: com.floragunn.searchguard.user.User cannot be cast to com.floragunn.searchguard.user.User

When casting an instance of a class to apparently same class throws an exception, it could mean that this class was loaded by two different classloaders. And inspecting the ES downstream image, it looks like we are installing the searchguard plugin twice which could explain this exception. 

1) separately here: http://pkgs.devel.redhat.com/cgit/rpms/logging-elasticsearch-docker/tree/install.sh.prod?h=rhaos-3.5-rhel-7#n27

2) as a part of openshift-elasticsearch-plugin: http://pkgs.devel.redhat.com/cgit/rpms/logging-elasticsearch-docker/tree/install.sh.prod?h=rhaos-3.5-rhel-7#n36

The searchguard became part of openshift-elasticsearch-plugin a few months ago.
https://github.com/fabric8io/openshift-elasticsearch-plugin/issues/78
Comment 4 Jeff Cantrill 2017-07-05 07:51:09 EDT
Closing this one as a duplicate since https://bugzilla.redhat.com/show_bug.cgi?id=1467572#c5 identifies the latest images work properly.  My suspicion is that tested images were some mix of the originals that deployed SG as a plugin and the change to the openshift-elasticsearch-plugin that embed SG

*** This bug has been marked as a duplicate of bug 1467572 ***

Note You need to log in before you can comment on or make changes to this bug.