Bug 1588416
| Summary: | should fetch log from /elasticsearch/persistent/logging-es/ | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Anping Li <anli> |
| Component: | Logging | Assignee: | Noriko Hosoi <nhosoi> |
| Status: | CLOSED ERRATA | QA Contact: | Anping Li <anli> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.10.0 | CC: | aos-bugs, ewolinet, jcantril, nhosoi, rmeggins |
| Target Milestone: | --- | ||
| Target Release: | 3.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: A utility logging-dump.sh dumps the ElasticSearch logs as part of useful information for troubleshooting.
In 3.10, the log location of ElasticSearch has been moved from "/elasticsearch/logging-es[-ops]/logs" to "/elasticsearch/persistent/logging-es[-ops]/logs".
Consequence: logging-dump.sh fails to dump ElasticSearch logs with "Unable to get ES logs from pod <ES_POD_NAME>"
Fix: In addition to "/elasticsearch/logging-es[-ops]/logs", check the new path "/elasticsearch/persistent/logging-es[-ops]/logs" for the logs files.
Result: logging-dump.sh successfully dumps ElasticSearch logs.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-07-30 19:17:19 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Anping Li
2018-06-07 09:41:10 UTC
Hi @Anping, Once the path is set to /elasticsearch/persistent/logging-es-ops/logs and /elasticsearch/persistent/logging-es/logs, does your "oc rsync" work? Mine fails as follows due to lack of rsync... Do I need to do something prior to running logging-dump.sh? ++ oc rsync -c elasticsearch $ES_POD:/elasticsearch/persistent/logging-es/logs logging-20180607_203045/es/logs WARNING: cannot use rsync: rsync not available in container Note: rsh to es pod, then running yum install fails like this. $ oc rsh $ES_POD sh-4.2$ yum install rsync Loaded plugins: fastestmirror, ovl ovl: Error while doing RPMdb copy-up: [Errno 13] Permission denied: '/var/lib/rpm/.dbenv.lock' You need to be root to perform this command. I guess I must be missing something... Thanks. Note: if I locally add rsync to the elasticsearch pod diff --git a/elasticsearch/Dockerfile.centos7 b/elasticsearch/Dockerfile.centos7 index 77a8202..2cfc36b 100644 --- a/elasticsearch/Dockerfile.centos7 +++ b/elasticsearch/Dockerfile.centos7 @@ -42,6 +42,7 @@ RUN rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch && \ yum install -y --setopt=tsflags=nodocs \ java-${JAVA_VER}-openjdk-headless \ elasticsearch-${ES_VER} \ + rsync \ PyYAML && \ yum clean all and fix the path as @Anping suggested, the logs of elasticsearch are successfully dumped. ++ path=/elasticsearch/persistent/logging-es/logs ++ oc rsync -c elasticsearch -q logging-es-data-master-zvx580wa-1-ztstj:/elasticsearch/persistent/logging-es/logs logging-20180607_221118/es/logs ++ mv -f logging-20180607_221118/es/logs/logs logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj ++ nice xz logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_deprecation.log logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_index_indexing_slowlog.log logging-20180607_221118/es/logs/logging-es-data-master-zvx580wa-1-ztstj/logging-es_index_search_slowlog.log Note: I'm testing with the build from the master branch. Noriko, How about use 'oc cp '? oc cp -c elasticsearch logging-es-data-master-nd2j231v-1-s6vfb:/elasticsearch/persistent/logging-es/logs logs Thanks, Anping. Yes, there seems oc cp / tar exists in the pods by default.
It turned out it's not that straightforward. :)
"oc rsync" generates 17 .xz files, while "oc cp" does only 11...
$ du -a logging-20180608_055413-rsync/ | egrep "\.xz$" | wc
17 34 1732
$ du -a logging-20180608_055616-cp/ | egrep "\.xz$" | wc
11 22 975
I'm going to continue investigating it.
BTW, rsync exists in your elasticsearch pod, doesn't it? If it is an upstream only issue, I think it's acceptable to add rsync to Dockerfile.centos7. Is it ok, Rich, Jeff?
=========================================================================
[oc rsync]
$ ls -lR logging-20180608_055413-rsync/es/logs
logging-20180608_055413-rsync/es/logs:
total 16
drwxr-sr-x. 2 origin origin-git 137 Jun 8 05:54 logging-es-data-master-i5lnodsm-1-glrnz
-rw-r--r--. 1 origin origin-git 3232 Jun 8 05:54 logging-es-data-master-i5lnodsm-1-glrnz-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git 752 Jun 8 05:54 logging-es-data-master-i5lnodsm-1-glrnz-proxy.log.xz
drwxr-sr-x. 2 origin origin-git 149 Jun 8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k
-rw-r--r--. 1 origin origin-git 2880 Jun 8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git 756 Jun 8 05:54 logging-es-ops-data-master-873opzqr-1-jwx6k-proxy.log.xz
logging-20180608_055413-rsync/es/logs/logging-es-data-master-i5lnodsm-1-glrnz:
total 12
-rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es_deprecation.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es_index_indexing_slowlog.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es_index_search_slowlog.log.xz
logging-20180608_055413-rsync/es/logs/logging-es-ops-data-master-873opzqr-1-jwx6k:
total 12
-rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es-ops_deprecation.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es-ops_index_indexing_slowlog.log.xz
-rw-r--r--. 1 origin origin-git 32 Jun 8 02:13 logging-es-ops_index_search_slowlog.log.xz
[oc cp]
$ ls -lR logging-20180608_055616-cp/es/logs
logging-20180608_055616-cp/es/logs:
total 16
-rw-r--r--. 1 origin origin-git 3232 Jun 8 05:56 logging-es-data-master-i5lnodsm-1-glrnz-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git 752 Jun 8 05:56 logging-es-data-master-i5lnodsm-1-glrnz-proxy.log.xz
-rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es_deprecation.log
-rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es_index_indexing_slowlog.log
-rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es_index_search_slowlog.log
-rw-r--r--. 1 origin origin-git 2880 Jun 8 05:56 logging-es-ops-data-master-873opzqr-1-jwx6k-elasticsearch.log.xz
-rw-r--r--. 1 origin origin-git 756 Jun 8 05:56 logging-es-ops-data-master-873opzqr-1-jwx6k-proxy.log.xz
-rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es-ops_deprecation.log
-rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es-ops_index_indexing_slowlog.log
-rw-r--r--. 1 origin origin-git 0 Jun 8 05:56 logging-es-ops_index_search_slowlog.log
Commits pushed to master at https://github.com/openshift/origin-aggregated-logging https://github.com/openshift/origin-aggregated-logging/commit/913815b82c376a613114ac1c3258355882dd14f0 Bug 1588416 - should fetch log from /elasticsearch/persistent/logging-es/ As suggested by @Anping Li in the bug, logging-dump.sh is having a support for the paths which contain "persistent" in addition to the original ones. Note: this is an adjustment to the change made by pr/1108. Also, suppressing the stderr output from "oc rsync", e.g., "WARNING: cannot use rsync: rsync not available in container". When rsync is not available in a pod, "oc rsync" falls back to tar. https://github.com/openshift/origin-aggregated-logging/commit/9abaa09a899eb84c8e5a05e27b88c0a0aeb8f0c3 Merge pull request #1211 from nhosoi/bz1588416 Bug 1588416 - should fetch log from /elasticsearch/persistent/logging… The log file are gathered. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:1816 |