Bug 1588416 - should fetch log from /elasticsearch/persistent/logging-es/
Summary: should fetch log from /elasticsearch/persistent/logging-es/
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Logging
Version: 3.10.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.10.z
Assignee: Noriko Hosoi
QA Contact: Anping Li
Depends On:
TreeView+ depends on / blocked
Reported: 2018-06-07 09:41 UTC by Anping Li
Modified: 2018-07-30 19:17 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: A utility logging-dump.sh dumps the ElasticSearch logs as part of useful information for troubleshooting. In 3.10, the log location of ElasticSearch has been moved from "/elasticsearch/logging-es[-ops]/logs" to "/elasticsearch/persistent/logging-es[-ops]/logs". Consequence: logging-dump.sh fails to dump ElasticSearch logs with "Unable to get ES logs from pod <ES_POD_NAME>" Fix: In addition to "/elasticsearch/logging-es[-ops]/logs", check the new path "/elasticsearch/persistent/logging-es[-ops]/logs" for the logs files. Result: logging-dump.sh successfully dumps ElasticSearch logs.
Clone Of:
Last Closed: 2018-07-30 19:17:19 UTC

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1816 None None None 2018-07-30 19:17:58 UTC

Description Anping Li 2018-06-07 09:41:10 UTC
Description of problem:
The Elasticsearch log are in /elasticsearch/persistent/logging-es/logs v3.10. 
the log-dump.sh should be updated.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1) curl -O https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/hack/logging-dump.sh

$ grep -A 18 'get_es_logs()' logging-dump.sh
get_es_logs() {
  local pod=$1
  local logs_folder=$2/logs
  echo -- POD $1 Elasticsearch Logs
  if [ ! -d "$logs_folder" ]
    mkdir $logs_folder
  local dc_name=$(oc get po $pod -o jsonpath='{.metadata.labels.deploymentconfig}')
  if [[ $pod == logging-es-ops* ]]
  oc rsync -c elasticsearch -q $pod:$path $logs_folder || echo ---- Unable to get ES logs from pod $pod
  mv -f $logs_folder/logs $logs_folder/$pod
  nice xz $logs_folder/$pod/*

Actual results:
logging-dump.sh use /elasticsearch/logging-es-ops/logs and /elasticsearch/logging-es/logs

Expected results:
logging-dump.sh use /elasticsearch/persistent/logging-es-ops/logs and /elasticsearch/persistent/logging-es/logs

Additional info:

Comment 1 Noriko Hosoi 2018-06-07 20:44:03 UTC
Hi @Anping,

Once the path is set to /elasticsearch/persistent/logging-es-ops/logs and /elasticsearch/persistent/logging-es/logs, does your "oc rsync" work?

Mine fails as follows due to lack of rsync...  Do I need to do something prior to running logging-dump.sh?
++ oc rsync -c elasticsearch $ES_POD:/elasticsearch/persistent/logging-es/logs 
WARNING: cannot use rsync: rsync not available in container

Note: rsh to es pod, then running yum install fails like this.
$ oc rsh $ES_POD
sh-4.2$ yum install rsync
Loaded plugins: fastestmirror, ovl
ovl: Error while doing RPMdb copy-up:
[Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock'
You need to be root to perform this command.

I guess I must be missing something...

Comment 2 Noriko Hosoi 2018-06-07 22:20:08 UTC
Note: if I locally add rsync to the elasticsearch pod 

diff --git a/elasticsearch/Dockerfile.centos7 b/elasticsearch/Dockerfile.centos7
index 77a8202..2cfc36b 100644
--- a/elasticsearch/Dockerfile.centos7
+++ b/elasticsearch/Dockerfile.centos7
@@ -42,6 +42,7 @@ RUN rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch && \
     yum install -y --setopt=tsflags=nodocs \
                 java-${JAVA_VER}-openjdk-headless \
                 elasticsearch-${ES_VER} \
+                rsync \
                 PyYAML && \
     yum clean all

and fix the path as @Anping suggested, the logs of elasticsearch are successfully dumped.

++ path=/elasticsearch/persistent/logging-es/logs
++ oc rsync -c elasticsearch -q logging-es-data-master-zvx580wa-1-ztstj:/elasticsearch/persistent/logging-es/logs logging-20180607_221118/es/logs
++ mv -f logging-20180607_221118/es/logs/logs logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj
++ nice xz logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_deprecation.log logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_index_indexing_slowlog.log logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_index_search_slowlog.log

Note: I'm testing with the build from the master branch.

Comment 3 Anping Li 2018-06-08 04:12:06 UTC
How about use 'oc cp '?
oc cp -c elasticsearch logging-es-data-master-nd2j231v-1-s6vfb:/elasticsearch/persistent/logging-es/logs logs

Comment 4 Noriko Hosoi 2018-06-08 06:14:03 UTC
Thanks, Anping.  Yes, there seems oc cp / tar exists in the pods by default.

It turned out it's not that straightforward. :)

"oc rsync" generates 17 .xz files, while "oc cp" does only 11...

$ du -a logging-20180608_055413-rsync/ | egrep "\.xz$" | wc
     17      34    1732
$ du -a logging-20180608_055616-cp/ | egrep "\.xz$" | wc
     11      22     975

I'm going to continue investigating it.

BTW, rsync exists in your elasticsearch pod, doesn't it?  If it is an upstream only issue, I think it's acceptable to add rsync to Dockerfile.centos7.  Is it ok, Rich, Jeff?

[oc rsync]
$ ls -lR logging-20180608_055413-rsync/es/logs
total 16
drwxr-sr-x. 2 origin origin-git  137 Jun  8 05:54 logging-es-data-master-i5lnodsm-1-glrnz
-rw-r--r--. 1 origin origin-git 3232 Jun  8 05:54 logging-es-data-master-i5lnodsm-1-glrnz-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git  752 Jun  8 05:54 logging-es-data-master-i5lnodsm-1-glrnz-proxy.log.xz
drwxr-sr-x. 2 origin origin-git  149 Jun  8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k
-rw-r--r--. 1 origin origin-git 2880 Jun  8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git  756 Jun  8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k-proxy.log.xz

total 12
-rw-r--r--. 1 origin origin-git 32 Jun  8 02:13 logging-es_deprecation.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun  8 02:13 logging-es_index_indexing_slowlog.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun  8 02:13 logging-es_index_search_slowlog.log.xz

total 12
-rw-r--r--. 1 origin origin-git 32 Jun  8 02:13 logging-es-ops_deprecation.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun  8 02:13 logging-es-ops_index_indexing_slowlog.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun  8 02:13 logging-es-ops_index_search_slowlog.log.xz

[oc cp]
$ ls -lR logging-20180608_055616-cp/es/logs
total 16
-rw-r--r--. 1 origin origin-git 3232 Jun  8 05:56 logging-es-data-master-i5lnodsm-1-glrnz-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git  752 Jun  8 05:56 logging-es-data-master-i5lnodsm-1-glrnz-proxy.log.xz
-rw-r--r--. 1 origin origin-git    0 Jun  8 05:56 logging-es_deprecation.log
-rw-r--r--. 1 origin origin-git    0 Jun  8 05:56 logging-es_index_indexing_slowlog.log
-rw-r--r--. 1 origin origin-git    0 Jun  8 05:56 logging-es_index_search_slowlog.log
-rw-r--r--. 1 origin origin-git 2880 Jun  8 05:56 logging-es-ops-data-master-873opzqr-1-jwx6k-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git  756 Jun  8 05:56 logging-es-ops-data-master-873opzqr-1-jwx6k-proxy.log.xz
-rw-r--r--. 1 origin origin-git    0 Jun  8 05:56 logging-es-ops_deprecation.log
-rw-r--r--. 1 origin origin-git    0 Jun  8 05:56 logging-es-ops_index_indexing_slowlog.log
-rw-r--r--. 1 origin origin-git    0 Jun  8 05:56 logging-es-ops_index_search_slowlog.log

Comment 8 openshift-github-bot 2018-06-14 19:49:50 UTC
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging

Bug 1588416 - should fetch log from /elasticsearch/persistent/logging-es/

As suggested by @Anping Li in the bug, logging-dump.sh is having a support
for the paths which contain "persistent" in addition to the original ones.
Note: this is an adjustment to the change made by pr/1108.

Also, suppressing the stderr output from "oc rsync", e.g., "WARNING: cannot
use rsync: rsync not available in container".  When rsync is not available
in a pod, "oc rsync" falls back to tar.

Merge pull request #1211 from nhosoi/bz1588416

Bug 1588416 - should fetch log from /elasticsearch/persistent/logging…

Comment 10 Anping Li 2018-06-19 07:54:00 UTC
The log file are gathered.

Comment 12 errata-xmlrpc 2018-07-30 19:17:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.