Bug 1377187

Created attachment 1202317 [details]
configmap

Created attachment 1202318 [details]
events

Created attachment 1202319 [details]
es_pod_log_#1

Created attachment 1202320 [details]
es_pod_log_#2

Created attachment 1202661 [details]
es_pod_log_#1_good

Created attachment 1202662 [details]
es_pod_log_#2_good

@Eric, sorry for my mistake--refreshed the es pod logs with good ones.

@Eric, seems the es pod is not recovered now, encountered this exception in its log:

[2016-09-19 15:59:19,414][ERROR][io.fabric8.elasticsearch.plugin.acl.DynamicACLFilter] [Lyja] Error checking ACL when seeding
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:151)
	at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction.checkGlobalBlock(TransportShardSingleOperationAction.java:103)
	at org.elasticsearch.action.support.single.shard.TransportShardSingleOperationAction$AsyncSingleAction.<init>(TransportShardSingleOperationAction.java:132)

Also ES is not connectable from curator:

# oc exec logging-curator-3-hiexf -- curator --host logging-es --use_ssl --certificate /etc/curator/keys/ca --client-cert /etc/curator/keys/cert --client-key /etc/curator/keys/key --loglevel ERROR show indices --all-indices
2016-09-19 22:21:26,084 ERROR     Failed to get indices.
2016-09-19 22:21:26,085 ERROR     No indices found in Elasticsearch
ERROR. No indices found in Elasticsearch.
error: error executing remote command: error executing command in container: Error executing in Docker Container: 1

BTW, I also checked on another OCP environment that the log retention policy of 5 days to the .defaults index, it worked fine to normal indices there...

The cluster is back in a 'green' state and I verified that I could connect with Kibana.

Fluentd is not currently scaled up at this time since we are still discussing the policy for retention with the test namespaces that are created as part of a check in this environment.

@Eric, 

Yes, I'm able to launch the kibana UI now, but get this error: 

Index Patterns 
Warning No default index pattern. You must select or create one to continue.

I wonder if it's caused by the absence of fluentd pod?

And here is the corresponding errors in es:

Error while apply() due to com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all for action indices:admin/get
com.floragunn.searchguard.tokeneval.MalformedConfigurationException: no bypass or execute filters at all
	at com.floragunn.searchguard.tokeneval.TokenEvaluator$Evaluator.validateAndMerge(TokenEvaluator.java:374)
	at com.floragunn.searchguard.tokeneval.TokenEvaluator$Evaluator.<init>(TokenEvaluator.java:362)
	at com.floragunn.searchguard.tokeneval.TokenEvaluator.getEvaluator(TokenEvaluator.java:310)

@Xia,

That exception is a known issue from going to the 'settings' page in Kibana, it is due to not having authorization to look up mappings for the default 'logstash-*' pattern.

The absence of Fluentd pods running would only mean that there are not new log records to view.

Looking in the logs I do see that there is a NPE which was fixed in the OpenShift-Elasticsearch-plugin for the ES 2.x, which is caused by a change in Kibana version when checking for the default index pattern. I'll put in a fix for that today. It is low impact since all it prevents is a user from having their default index pattern set for them in Kibana.

The work around is to manually set the default index pattern in Kibana from the settings page > click on the index pattern you would like to be the default > click on the 'star' icon to set it as your default.

@Eric,

Thanks to the qucik fix -- kibana UI give me the page with "No results found " in title which is expected with the absence of fluentd.

Verified on dedicated-whearntest that fluentd pods are back, and be able to get log entries for different indices on kibana UI. Set to verified.