Saw this stack trace multiple times on a 3.4.1 OCP instance, with logging-elasticsearch 3.4.1-26: [2017-06-06 13:30:08,515][WARN ][rest.suppressed ] path: /_mget, params: {ignore_unavailable=true, preference=1496755806317, timeout=0} java.lang.NullPointerException at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.buildNewResult(KibanaUserReindexAction.java:180) at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:124) at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:210) at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:237) at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:223) at org.elasticsearch.action.get.TransportMultiGetAction$1.finishHim(TransportMultiGetAction.java:122) at org.elasticsearch.action.get.TransportMultiGetAction$1.onResponse(TransportMultiGetAction.java:104) at org.elasticsearch.action.get.TransportMultiGetAction$1.onResponse(TransportMultiGetAction.java:97) at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89) at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:212) at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:173) at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:210) at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:163) at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:210) at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:237) at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:223) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:235) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:221) at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:158) at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.handleResponse(SearchGuardMessageChannelHandler.java:56) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:124) at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.messageReceived(SearchGuardMessageChannelHandler.java:50) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748)
Fixed in origin: https://github.com/fabric8io/openshift-elasticsearch-plugin/pull/83
Commit pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/4187d2dff2077ffd8c8b3dab3d7d60ba3458b10b bump openshift-elasticsearch-plugin to 2.4.4.9 bug 1456413. Fix Kibana security error message bug 1459221. Fix null pointer in KibanaUserReindexAction fix plugin improper evaluation of index pattern fix plugin improper evaluation of how to retrieve a user from 2.4.4.6
Tested with the latest v3.6 images on OCP 3.6.0, logging system worked fine and didn't meet this exception in es log. Set to verified. Test env: # openshift version openshift v3.6.131 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 ansible version: openshift-ansible-playbooks-3.6.131-1.git.0.d87dfaa.el7.noarch Images tested with: openshift3/logging-elasticsearch c601094a6111 openshift3/logging-kibana c91b7ad68dc7 openshift3/logging-fluentd 82367a1102e0 openshift3/logging-curator b609245a72f9 openshift3/logging-auth-proxy 39164e25543c
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1716