Bug 1396480

Summary: elasticsearch pod stores logs inside container.
Product: OpenShift Container Platform Reporter: Alexander Koksharov <akokshar>
Component: LoggingAssignee: ewolinet
Status: CLOSED CURRENTRELEASE QA Contact: Junqi Zhao <juzhao>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.1.0CC: aos-bugs, ewolinet, pweil, tdawson
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Feature: Elasticsearch no longer creates log file entries in its log directory in the container/pvc. Reason: To not take up as much space on the container's ephemeral/PVC we updated Elasticsearch's log configuration to only write out to stdout and no longer split log entries to daily rollover files. Result: Elasticsearch no longer creates log files and only writes logs out to stdout
 Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-16 21:00:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Koksharov 2016-11-18 12:56:00 UTC 
Description of problem:

It has been discovered that elasticsearch stores logs under /elasticsearch/<cluster-name>/logs/

This is not necessary as logs are already collected by fluentd. So it is just waste of memory:
sh-4.2$ cd /elasticsearch/logging-es/logs/
sh-4.2$ ls -lah
total 1.9G
drwxr-xr-x. 2 1000050000 root 4.0K Nov 18 00:00 .
drwxr-xr-x. 3 1000050000 root   48 Nov 14 11:05 ..
-rw-r--r--. 1 1000050000 root 232M Nov 18 07:53 logging-es.log
-rw-r--r--. 1 1000050000 root 824K Nov 14 23:59 logging-es.log.2016-11-14
-rw-r--r--. 1 1000050000 root 201M Nov 15 23:59 logging-es.log.2016-11-15
-rw-r--r--. 1 1000050000 root 721M Nov 16 23:59 logging-es.log.2016-11-16
-rw-r--r--. 1 1000050000 root 722M Nov 17 23:59 logging-es.log.2016-11-17
-rw-r--r--. 1 1000050000 root    0 Nov 14 11:05 logging-es_index_indexing_slowlog.log
-rw-r--r--. 1 1000050000 root    0 Nov 14 11:05 logging-es_index_search_slowlog.log
sh-4.2$ 


Version-Release number of selected component (if applicable):
reported for 3.1. still exist in 3.3

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 5 ewolinet 2016-12-14 17:27:19 UTC 
12237861 buildContainer (noarch) completed successfully
koji_builds:
  https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=528578
repositories:
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-deployer:rhaos-3.4-rhel-7-docker-candidate-20161214121213
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-deployer:v3.4.0.36-2
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-deployer:v3.4.0.36
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-deployer:latest
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-deployer:v3.4
  brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/logging-deployer:3.4.0
Comment 7 Junqi Zhao 2016-12-15 03:29:25 UTC 
Tested on OCP 3.4.0, this issue does not exist, elasticsearch will not store logs under /elasticsearch/<cluster-name>/logs/

sh-4.2$ cd /elasticsearch/logging-es/
sh-4.2$ ls -lah
total 0
drwxr-xr-x. 2 1000050000 root  6 Dec 15 03:18 .
drwxrwxrwx. 4 root       root 42 Dec 15 03:18 ..


Image ID:
openshift3/logging-deployer    755d30b7d4de
openshift3/logging-kibana    d5971557d356
openshift3/logging-fluentd    7b11a29c82c1
openshift3/logging-elasticsearch    6716a0ad8b2b
openshift3/logging-auth-proxy    ec334b0c2669
openshift3/logging-curator    9af78fc06248


Close this issue, set it to VERIFIED
Comment 8 Troy Dawson 2017-02-16 21:00:30 UTC 
This bug was fixed with OCP 3.4.0 that is already released.