Description of problem: The Elasticsearch log are in /elasticsearch/persistent/logging-es/logs v3.10. the log-dump.sh should be updated. Version-Release number of selected component (if applicable): https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/hack/logging-dump.sh How reproducible: Always Steps to Reproduce: 1) curl -O https://raw.githubusercontent.com/openshift/origin-aggregated-logging/master/hack/logging-dump.sh $ grep -A 18 'get_es_logs()' logging-dump.sh get_es_logs() { local pod=$1 local logs_folder=$2/logs echo -- POD $1 Elasticsearch Logs if [ ! -d "$logs_folder" ] then mkdir $logs_folder fi local dc_name=$(oc get po $pod -o jsonpath='{.metadata.labels.deploymentconfig}') if [[ $pod == logging-es-ops* ]] then path=/elasticsearch/logging-es-ops/logs else path=/elasticsearch/logging-es/logs fi oc rsync -c elasticsearch -q $pod:$path $logs_folder || echo ---- Unable to get ES logs from pod $pod mv -f $logs_folder/logs $logs_folder/$pod nice xz $logs_folder/$pod/* } Actual results: logging-dump.sh use /elasticsearch/logging-es-ops/logs and /elasticsearch/logging-es/logs Expected results: logging-dump.sh use /elasticsearch/persistent/logging-es-ops/logs and /elasticsearch/persistent/logging-es/logs Additional info:
Hi @Anping, Once the path is set to /elasticsearch/persistent/logging-es-ops/logs and /elasticsearch/persistent/logging-es/logs, does your "oc rsync" work? Mine fails as follows due to lack of rsync... Do I need to do something prior to running logging-dump.sh? ++ oc rsync -c elasticsearch $ES_POD:/elasticsearch/persistent/logging-es/logs logging-20180607_203045/es/logs WARNING: cannot use rsync: rsync not available in container Note: rsh to es pod, then running yum install fails like this. $ oc rsh $ES_POD sh-4.2$ yum install rsync Loaded plugins: fastestmirror, ovl ovl: Error while doing RPMdb copy-up: [Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock' You need to be root to perform this command. I guess I must be missing something... Thanks.
Note: if I locally add rsync to the elasticsearch pod diff --git a/elasticsearch/Dockerfile.centos7 b/elasticsearch/Dockerfile.centos7 index 77a8202..2cfc36b 100644 --- a/elasticsearch/Dockerfile.centos7 +++ b/elasticsearch/Dockerfile.centos7 @@ -42,6 +42,7 @@ RUN rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch && \ yum install -y --setopt=tsflags=nodocs \ java-${JAVA_VER}-openjdk-headless \ elasticsearch-${ES_VER} \ + rsync \ PyYAML && \ yum clean all and fix the path as @Anping suggested, the logs of elasticsearch are successfully dumped. ++ path=/elasticsearch/persistent/logging-es/logs ++ oc rsync -c elasticsearch -q logging-es-data-master-zvx580wa-1-ztstj:/elasticsearch/persistent/logging-es/logs logging-20180607_221118/es/logs ++ mv -f logging-20180607_221118/es/logs/logs logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj ++ nice xz logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_deprecation.log logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_index_indexing_slowlog.log logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_index_search_slowlog.log Note: I'm testing with the build from the master branch.
Noriko, How about use 'oc cp '? oc cp -c elasticsearch logging-es-data-master-nd2j231v-1-s6vfb:/elasticsearch/persistent/logging-es/logs logs
Thanks, Anping. Yes, there seems oc cp / tar exists in the pods by default. It turned out it's not that straightforward. :) "oc rsync" generates 17 .xz files, while "oc cp" does only 11... $ du -a logging-20180608_055413-rsync/ | egrep "\.xz$" | wc 17 34 1732 $ du -a logging-20180608_055616-cp/ | egrep "\.xz$" | wc 11 22 975 I'm going to continue investigating it. BTW, rsync exists in your elasticsearch pod, doesn't it? If it is an upstream only issue, I think it's acceptable to add rsync to Dockerfile.centos7. Is it ok, Rich, Jeff? ========================================================================= [oc rsync] $ ls -lR logging-20180608_055413-rsync/es/logs logging-20180608_055413-rsync/es/logs: total 16 drwxr-sr-x. 2 origin origin-git 137 Jun 8 05:54 logging-es-data-master-i5lnodsm-1-glrnz -rw-r--r--. 1 origin origin-git 3232 Jun 8 05:54 logging-es-data-master-i5lnodsm-1-glrnz-elasticsearch.log.xz -rw-r--r--. 1 origin origin-git 752 Jun 8 05:54 logging-es-data-master-i5lnodsm-1-glrnz-proxy.log.xz drwxr-sr-x. 2 origin origin-git 149 Jun 8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k -rw-r--r--. 1 origin origin-git 2880 Jun 8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k-elasticsearch.log.xz -rw-r--r--. 1 origin origin-git 756 Jun 8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k-proxy.log.xz logging-20180608_055413-rsync/es/logs/logging-es-data-master-i5lnodsm-1-glrnz: total 12 -rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es_deprecation.log.xz -rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es_index_indexing_slowlog.log.xz -rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es_index_search_slowlog.log.xz logging-20180608_055413-rsync/es/logs/logging-es-ops-data-master-873opzqr-1-jwx6k: total 12 -rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es-ops_deprecation.log.xz -rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es-ops_index_indexing_slowlog.log.xz -rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es-ops_index_search_slowlog.log.xz [oc cp] $ ls -lR logging-20180608_055616-cp/es/logs logging-20180608_055616-cp/es/logs: total 16 -rw-r--r--. 1 origin origin-git 3232 Jun 8 05:56 logging-es-data-master-i5lnodsm-1-glrnz-elasticsearch.log.xz -rw-r--r--. 1 origin origin-git 752 Jun 8 05:56 logging-es-data-master-i5lnodsm-1-glrnz-proxy.log.xz -rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es_deprecation.log -rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es_index_indexing_slowlog.log -rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es_index_search_slowlog.log -rw-r--r--. 1 origin origin-git 2880 Jun 8 05:56 logging-es-ops-data-master-873opzqr-1-jwx6k-elasticsearch.log.xz -rw-r--r--. 1 origin origin-git 756 Jun 8 05:56 logging-es-ops-data-master-873opzqr-1-jwx6k-proxy.log.xz -rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es-ops_deprecation.log -rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es-ops_index_indexing_slowlog.log -rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es-ops_index_search_slowlog.log
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/913815b82c376a613114ac1c3258355882dd14f0 Bug 1588416 - should fetch log from /elasticsearch/persistent/logging-es/ As suggested by @Anping Li in the bug, logging-dump.sh is having a support for the paths which contain "persistent" in addition to the original ones. Note: this is an adjustment to the change made by pr/1108. Also, suppressing the stderr output from "oc rsync", e.g., "WARNING: cannot use rsync: rsync not available in container". When rsync is not available in a pod, "oc rsync" falls back to tar. https://github.com/openshift/origin-aggregated-logging/commit/9abaa09a899eb84c8e5a05e27b88c0a0aeb8f0c3 Merge pull request #1211 from nhosoi/bz1588416 Bug 1588416 - should fetch log from /elasticsearch/persistent/logging…
The log file are gathered.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816