Bug 1566138
Summary: | [Docs][RFE][Metrics Install] Document how to extract Elasticsearch logs for troubleshooting | ||
---|---|---|---|
Product: | Red Hat Enterprise Virtualization Manager | Reporter: | Emma Heftman <eheftman> |
Component: | Documentation | Assignee: | Eli Marcus <emarcus> |
Status: | CLOSED CURRENTRELEASE | QA Contact: | rhev-docs <rhev-docs> |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 4.3.0 | CC: | anli, aos-bugs, apinnick, cminkema, dcadzow, emarcus, jcantril, lleistne, lsurette, lsvaty, mtessun, pelauter, rhev-docs, rmeggins, sradco, srevivo |
Target Milestone: | ovirt-4.3.8 | Keywords: | Documentation, FutureFeature, Task |
Target Release: | --- | ||
Hardware: | All | ||
OS: | All | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | 1566047 | Environment: | |
Last Closed: | 2020-01-16 16:08:44 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | Metrics | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1566047 | ||
Bug Blocks: |
Comment 1
Lucy Bopf
2018-05-24 04:08:30 UTC
Hey Shirly Could you please provide some information about - where the logs are located? - when they should be used, i.e. which specific scenarios would require using these logs to resolve the issue? - what is the format of the logs? Thanks! This utility was added to the Elasticsearch image to extract logs since we no longer write to STDOUT by default: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs Hi Rich, Can you pleas provide the steps how to use the util / get the Elasticsearch logs? Also the additional information Emma asked in comment #2 (In reply to Emma Heftman from comment #2) > Hey Shirly > Could you please provide some information about > - where the logs are located? > - when they should be used, i.e. which specific scenarios would require > using these logs to resolve the issue? > - what is the format of the logs? > > Thanks! 1. determine the name of the project used by the logging components oc get projects | grep logging If "openshift-logging" is in that list, use it, otherwise, use "logging". I will refer to this project name as $PROJECT below. 2. get the name of an elasticsearch pod oc -n $PROJECT get pods -l component=es I will refer to the name of an es pod returned from this command as $espod below. 3. examine the log files inside the container oc -n $PROJECT -c elasticsearch $espod -- logs This will just dump the log file to stdout If you want to look for a particular string in the logs use oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string" If you want to tail the logs, use oc -n $PROJECT -c elasticsearch $espod -- logs -f For ref: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs#L27 (In reply to Rich Megginson from comment #6) > (In reply to Emma Heftman from comment #2) > > Hey Shirly > > Could you please provide some information about > > - where the logs are located? > > - when they should be used, i.e. which specific scenarios would require > > using these logs to resolve the issue? > > - what is the format of the logs? > > > > Thanks! > > 1. determine the name of the project used by the logging components > > oc get projects | grep logging > > If "openshift-logging" is in that list, use it, otherwise, use "logging". I > will refer to this project name as $PROJECT below. > > 2. get the name of an elasticsearch pod > > oc -n $PROJECT get pods -l component=es > > I will refer to the name of an es pod returned from this command as $espod > below. > > 3. examine the log files inside the container > > oc -n $PROJECT -c elasticsearch $espod -- logs > > This will just dump the log file to stdout > > If you want to look for a particular string in the logs use > > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string" > > If you want to tail the logs, use > > oc -n $PROJECT -c elasticsearch $espod -- logs -f Thanks Rich. If there is a problem loading a specific pod, how can they export the output to a file in order to send to support? (In reply to Jeff Cantrill from comment #3) > This utility was added to the Elasticsearch image to extract logs since we > no longer write to STDOUT by default: > https://github.com/openshift/origin-aggregated-logging/blob/master/ > elasticsearch/utils/logs How do we run this util? (In reply to Emma Heftman from comment #8) > (In reply to Rich Megginson from comment #6) > > (In reply to Emma Heftman from comment #2) > > > Hey Shirly > > > Could you please provide some information about > > > - where the logs are located? > > > - when they should be used, i.e. which specific scenarios would require > > > using these logs to resolve the issue? > > > - what is the format of the logs? > > > > > > Thanks! > > > > 1. determine the name of the project used by the logging components > > > > oc get projects | grep logging > > > > If "openshift-logging" is in that list, use it, otherwise, use "logging". I > > will refer to this project name as $PROJECT below. > > > > 2. get the name of an elasticsearch pod > > > > oc -n $PROJECT get pods -l component=es > > > > I will refer to the name of an es pod returned from this command as $espod > > below. > > > > 3. examine the log files inside the container > > > > oc -n $PROJECT -c elasticsearch $espod -- logs > > > > This will just dump the log file to stdout > > > > If you want to look for a particular string in the logs use > > > > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string" > > > > If you want to tail the logs, use > > > > oc -n $PROJECT -c elasticsearch $espod -- logs -f > > Thanks Rich. > If there is a problem loading a specific pod, how can they export the output > to a file in order to send to support? Use 'oc -n $PROJECT -c elasticsearch $espod logs' to get the container logs - but since elasticsearch is logging to the file, this container log will contain very little, but it may be useful. The other alternative is to configure elasticsearch with persistent logs - @jcantrill - is this documented? (In reply to Shirly Radco from comment #9) > (In reply to Jeff Cantrill from comment #3) > > This utility was added to the Elasticsearch image to extract logs since we > > no longer write to STDOUT by default: > > https://github.com/openshift/origin-aggregated-logging/blob/master/ > > elasticsearch/utils/logs > > How do we run this util? As I have described in https://bugzilla.redhat.com/show_bug.cgi?id=1566138#c6 (In reply to Rich Megginson from comment #6) > (In reply to Emma Heftman from comment #2) > > Hey Shirly > > Could you please provide some information about > > - where the logs are located? > > - when they should be used, i.e. which specific scenarios would require > > using these logs to resolve the issue? > > - what is the format of the logs? > > > > Thanks! > > 1. determine the name of the project used by the logging components > > oc get projects | grep logging > > If "openshift-logging" is in that list, use it, otherwise, use "logging". I > will refer to this project name as $PROJECT below. > > 2. get the name of an elasticsearch pod > > oc -n $PROJECT get pods -l component=es > > I will refer to the name of an es pod returned from this command as $espod > below. > > 3. examine the log files inside the container > > oc -n $PROJECT -c elasticsearch $espod -- logs > This and below did not work for me in ocp 3.9. I used: oc logs -n $PROJECT -c elasticsearch $espod That worked for the examples below too. Is there a difference in the ocp versions when using the logs util? > This will just dump the log file to stdout > > If you want to look for a particular string in the logs use > > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string" > > If you want to tail the logs, use > > oc -n $PROJECT -c elasticsearch $espod -- logs -f There is no need to specifically configure for specific logs as we moved them to the nodes persistent directory with [1]. This was also backported to 3.9 and is available in v3.9.30-1 or newer. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1568361 (In reply to Shirly Radco from comment #12) > (In reply to Rich Megginson from comment #6) > > (In reply to Emma Heftman from comment #2) > > > Hey Shirly > > > Could you please provide some information about > > > - where the logs are located? > > > - when they should be used, i.e. which specific scenarios would require > > > using these logs to resolve the issue? > > > - what is the format of the logs? > > > > > > Thanks! > > > > 1. determine the name of the project used by the logging components > > > > oc get projects | grep logging > > > > If "openshift-logging" is in that list, use it, otherwise, use "logging". I > > will refer to this project name as $PROJECT below. > > > > 2. get the name of an elasticsearch pod > > > > oc -n $PROJECT get pods -l component=es > > > > I will refer to the name of an es pod returned from this command as $espod > > below. > > > > 3. examine the log files inside the container > > > > oc -n $PROJECT -c elasticsearch $espod -- logs > > > > This and below did not work for me in ocp 3.9. > I used: > oc logs -n $PROJECT -c elasticsearch $espod > > That worked for the examples below too. > > Is there a difference in the ocp versions when using the logs util? oc logs -n $PROJECT -c elasticsearch $espod will always print something. But what it prints depends on the logger setting: oc get -n $PROJECT cm/logging-elasticsearch -o yaml | grep rootLogger rootLogger: ${es.logger.level}, console If you see "console" here, it means "oc logs" will contain all of the elasticsearch logs. If you see "file" here and _not_ "console", it means elasticsearch is logging to a log file and not the console, which means "oc logs" will not show you everything. In that case, you must look at the log file. Note that "file" is now the default, so it will almost always be "file". The "logs" command as discussed elsewhere _can_ be used, but _does not have to be used_ to look at the log file - it is merely a convenience, not required. > > > This will just dump the log file to stdout > > > > If you want to look for a particular string in the logs use > > > > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string" > > > > If you want to tail the logs, use > > > > oc -n $PROJECT -c elasticsearch $espod -- logs -f This bug has not been marked as blocker for oVirt 4.3.0. Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1. (In reply to Shirly Radco from comment #21) > Yes. We need documentation for users to be able to get and attach the logs > to the bugs. You use [1] to pull logs that are persisted to disk or [2] to dump the environment which also grabs these logs too: [1]https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs [2]https://github.com/openshift/origin-aggregated-logging/blob/master/hack/logging-dump.sh Despite this is a documentation bug, QE has no residual capacity for testing this in 4.3.4, re-targeting to 4.3.5 |