Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1457232

Summary:

NullPointerException - Error getting default index for <user>

Product:

OpenShift Container Platform

Reporter:

Ruben Romero Montes <rromerom>

Component:

Logging

Assignee:

Jeff Cantrill <jcantril>

Status:

CLOSED ERRATA

QA Contact:

Anping Li <anli>

Severity:

high

Docs Contact:

Priority:

high

Version:

3.4.1

CC:

anli, aos-bugs, jcantril, pportant, pweil, rmeggins, rromerom, smunilla

Target Milestone:

---

Target Release:

3.4.z

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

No Doc Update

Doc Text:

undefined

Story Points:

---

Clone Of:

Environment:

Last Closed:

2017-10-25 13:02:19 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Attachments:

Description	Flags
es and kibana logs	none

Description Ruben Romero Montes 2017-05-31 11:47:31 UTC

Created attachment 1283712 [details]
es and kibana logs

Description of problem:
I have found the following error while investigating this other BZ
  https://bugzilla.redhat.com/1410694

[2017-05-26 11:51:52,917][ERROR][io.fabric8.elasticsearch.plugin.kibana.KibanaSeed] Error getting default index for C3013111
java.util.concurrent.ExecutionException: RemoteTransportException[[Abigail Brand][127.0.0.1:9300][indices:data/read/get]]; nested: NullPointerException;
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:294)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:281)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:120)
	at io.fabric8.elasticsearch.plugin.kibana.KibanaSeed.getDefaultIndex(KibanaSeed.java:272)
	at io.fabric8.elasticsearch.plugin.kibana.KibanaSeed.setDashboards(KibanaSeed.java:156)
	at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.updateCache(DynamicACLFilter.java:238)
	at io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter.process(DynamicACLFilter.java:183)
	at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:264)
	at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:161)
	at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:153)
	at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:101)
	at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:451)
	at com.floragunn.searchguard.ssl.http.netty.SearchGuardSSLNettyHttpServerTransport.dispatchRequest(SearchGuardSSLNettyHttpServerTransport.java:159)
	at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:61)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:60)
	at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
	at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: RemoteTransportException[[Abigail Brand][127.0.0.1:9300][indices:data/read/get]]; nested: NullPointerException;
Caused by: java.lang.NullPointerException
	at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.buildNewResult(KibanaUserReindexAction.java:180)
	at io.fabric8.elasticsearch.plugin.KibanaUserReindexAction.apply(KibanaUserReindexAction.java:103)
	at org.elasticsearch.action.support.TransportAction$ResponseFilterChain.proceed(TransportAction.java:210)
	at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:237)
	at org.elasticsearch.action.support.TransportAction$FilteredActionListener.onResponse(TransportAction.java:223)
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:235)
	at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleResponse(TransportSingleShardAction.java:221)
	at org.elasticsearch.transport.netty.MessageChannelHandler.handleResponse(MessageChannelHandler.java:158)
	at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.handleResponse(SearchGuardMessageChannelHandler.java:56)
	at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:124)
	at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.messageReceived(SearchGuardMessageChannelHandler.java:50)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
	at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
	at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
	at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
	at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

Version-Release number of selected component (if applicable):

/root/buildinfo/Dockerfile-openshift3-logging-elasticsearch-3.4.1-26
/root/buildinfo/Dockerfile-openshift3-logging-fluentd-3.4.1-13
/root/buildinfo/Dockerfile-openshift3-logging-kibana-3.4.1-14

How reproducible:
Unknown

Steps to Reproduce:
1. Unknown

Actual results:
NullPointerException in the Elasticsearch logs.

Expected results:
No NPE

Additional info:

Comment 1 Jeff Cantrill 2017-06-02 17:07:36 UTC

Can you provide the version of the ES image this is using so I may better understand where to investigate the fix?

Comment 2 Ruben Romero Montes 2017-06-12 11:50:37 UTC

Hi Jeff,

the ES image was specified in the original message:

logging-elasticsearch-3.4.1-26

Thanks,
Ruben

Comment 4 Jeff Cantrill 2017-06-20 01:51:02 UTC

This is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1459221  I am keeping this issue open to address the 3.4.x line of ElasticSearch for the required errata

Comment 5 Jeff Cantrill 2017-06-21 00:20:52 UTC

upstream PR https://github.com/fabric8io/openshift-elasticsearch-plugin/pull/88

Comment 6 Rich Megginson 2017-09-08 15:35:47 UTC

Jeff, is this fixed now in 3.7 and 3.6 since you have rebuilt the es image?

Comment 7 Jeff Cantrill 2017-09-13 14:23:58 UTC

This is in 3.4, 3.5, 3.6, and master.

Fixed in https://github.com/openshift/origin-aggregated-logging/pull/650

Comment 9 Anping Li 2017-09-15 03:22:59 UTC

The OSE_ES_VER is still 2.4.1.8 in the last openshift3/logging-elasticsearch:v3.4.  Can you confirm the logging-elasticsearch have been pushed to brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888? 

# oc exec logging-es-fp5x3l6g-1-0tnjq env |grep VER
RECOVER_AFTER_NODES=0
RECOVER_EXPECTED_NODES=1
RECOVER_AFTER_TIME=5m
JAVA_VER=1.8.0
ES_VER=2.4.1
SG_VER=2.4.1.8
SG_VER_RELEASE=2
SG_SSL_VER=2.4.1.16
SG_SSL_VER_RELEASE=1
ES_CLOUD_K8S_VER=2.4.1
OSE_ES_VER=2.4.1.8

Comment 11 Jeff Cantrill 2017-09-21 18:41:54 UTC

Sam,  please build some official images for this

Comment 13 Anping Li 2017-10-16 05:38:32 UTC

The fix was in 3.4.1.44.26, the logging work well. so move bug to verified.
oc exec logging-es-38hrwj87-1-0m8uz env |grep VER
RECOVER_AFTER_NODES=0
RECOVER_EXPECTED_NODES=1
RECOVER_AFTER_TIME=5m
JAVA_VER=1.8.0
ES_VER=2.4.1
SG_VER=2.4.1.8
SG_VER_RELEASE=2
SG_SSL_VER=2.4.1.16
SG_SSL_VER_RELEASE=1
ES_CLOUD_K8S_VER=2.4.1
OSE_ES_VER=2.4.1.10

Comment 15 errata-xmlrpc 2017-10-25 13:02:19 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3049