Created attachment 1283712 [details] es and kibana logs Description of problem: I have found the following error while investigating this other BZ https://bugzilla.redhat.com/1410694 [2017-05-26 11:51:52,917][ERROR][io.fabric8.elasticsearch.plugin.kibana.KibanaSeed] Error getting default index for C3013111 java.util.concurrent.ExecutionException: RemoteTransportException[[Abigail Brand][127.0.0.1:9300][indices:data/read/get]]; nested: NullPointerException; at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:294) at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:281) at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:120) at io.fabric8.elasticsearch.plugin.kibana.KibanaSeed.getDefaultIndex(KibanaSeed.java:272) at io.fabric8.elasticsearch.plugin.kibana.KibanaSeed.setDashboards(KibanaSeed.java:156) at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.updateCache(DynamicACLFilter.java:238) at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.process(DynamicACLFilter.java:183) at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:161) at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153) at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101) at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451) at com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport.dispatchRequest(SearchGuardSSLNettyHttpServerTransport.java:159) at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60) at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: RemoteTransportException[[Abigail Brand][127.0.0.1:9300][indices:data/read/get]]; nested: NullPointerException; Caused by: java.lang.NullPointerException at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.buildNewResult(KibanaUserReindexAction.java:180) at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:103) at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:210) at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:237) at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:223) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:235) at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:221) at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:158) at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.handleResponse(SearchGuardMessageChannelHandler.java:56) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:124) at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.messageReceived(SearchGuardMessageChannelHandler.java:50) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Version-Release number of selected component (if applicable): /root/buildinfo/Dockerfile-openshift3-logging-elasticsearch-3.4.1-26 /root/buildinfo/Dockerfile-openshift3-logging-fluentd-3.4.1-13 /root/buildinfo/Dockerfile-openshift3-logging-kibana-3.4.1-14 How reproducible: Unknown Steps to Reproduce: 1. Unknown Actual results: NullPointerException in the Elasticsearch logs. Expected results: No NPE Additional info:
Can you provide the version of the ES image this is using so I may better understand where to investigate the fix?
Hi Jeff, the ES image was specified in the original message: logging-elasticsearch-3.4.1-26 Thanks, Ruben
This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1459221 I am keeping this issue open to address the 3.4.x line of ElasticSearch for the required errata
upstream PR https://github.com/fabric8io/openshift-elasticsearch-plugin/pull/88
Jeff, is this fixed now in 3.7 and 3.6 since you have rebuilt the es image?
This is in 3.4, 3.5, 3.6, and master. Fixed in https://github.com/openshift/origin-aggregated-logging/pull/650
The OSE_ES_VER is still 2.4.1.8 in the last openshift3/logging-elasticsearch:v3.4. Can you confirm the logging-elasticsearch have been pushed to brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888? # oc exec logging-es-fp5x3l6g-1-0tnjq env |grep VER RECOVER_AFTER_NODES=0 RECOVER_EXPECTED_NODES=1 RECOVER_AFTER_TIME=5m JAVA_VER=1.8.0 ES_VER=2.4.1 SG_VER=2.4.1.8 SG_VER_RELEASE=2 SG_SSL_VER=2.4.1.16 SG_SSL_VER_RELEASE=1 ES_CLOUD_K8S_VER=2.4.1 OSE_ES_VER=2.4.1.8
Sam, please build some official images for this
The fix was in 3.4.1.44.26, the logging work well. so move bug to verified. oc exec logging-es-38hrwj87-1-0m8uz env |grep VER RECOVER_AFTER_NODES=0 RECOVER_EXPECTED_NODES=1 RECOVER_AFTER_TIME=5m JAVA_VER=1.8.0 ES_VER=2.4.1 SG_VER=2.4.1.8 SG_VER_RELEASE=2 SG_SSL_VER=2.4.1.16 SG_SSL_VER_RELEASE=1 ES_CLOUD_K8S_VER=2.4.1 OSE_ES_VER=2.4.1.10
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:3049