Bug 1566138 - [Docs][RFE][Metrics Install] Document how to extract Elasticsearch logs for troubleshooting
Summary: [Docs][RFE][Metrics Install] Document how to extract Elasticsearch logs for t...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: Documentation
Version: 4.3.0
Hardware: All
OS: All
high
high
Target Milestone: ovirt-4.3.8
: ---
Assignee: Eli Marcus
QA Contact: rhev-docs@redhat.com
URL:
Whiteboard:
Depends On: 1566047
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-11 15:28 UTC by Emma Heftman
Modified: 2020-01-16 16:08 UTC (History)
16 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1566047
Environment:
Last Closed: 2020-01-16 16:08:44 UTC
oVirt Team: Metrics
Target Upstream Version:


Attachments (Terms of Use)

Comment 1 Lucy Bopf 2018-05-24 04:08:30 UTC
Emma, I notice this one doesn't have any details attached. Do you have any more information about what's required, or can you flag this to the Metrics team as needing planning input?

Comment 2 Emma Heftman 2018-05-24 07:14:28 UTC
Hey Shirly
Could you please provide some information about 
- where the logs are located? 
- when they should be used, i.e. which specific scenarios would require using these logs to resolve the issue?
- what is the format of the logs?

Thanks!

Comment 3 Jeff Cantrill 2018-05-24 14:21:05 UTC
This utility was added to the Elasticsearch image to extract logs since we no longer write to STDOUT by default: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs

Comment 4 Shirly Radco 2018-05-27 06:21:22 UTC
Hi Rich,

Can you pleas provide the steps how to use the util / get the Elasticsearch logs?

Comment 5 Shirly Radco 2018-05-27 06:22:07 UTC
Also the additional information Emma asked in comment #2

Comment 6 Rich Megginson 2018-05-31 18:45:56 UTC
(In reply to Emma Heftman from comment #2)
> Hey Shirly
> Could you please provide some information about 
> - where the logs are located? 
> - when they should be used, i.e. which specific scenarios would require
> using these logs to resolve the issue?
> - what is the format of the logs?
> 
> Thanks!

1. determine the name of the project used by the logging components

oc get projects | grep logging

If "openshift-logging" is in that list, use it, otherwise, use "logging".  I will refer to this project name as $PROJECT below.

2. get the name of an elasticsearch pod

oc -n $PROJECT get pods -l component=es

I will refer to the name of an es pod returned from this command as $espod below.

3. examine the log files inside the container

oc -n $PROJECT -c elasticsearch $espod -- logs

This will just dump the log file to stdout

If you want to look for a particular string in the logs use

oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"

If you want to tail the logs, use

oc -n $PROJECT -c elasticsearch $espod -- logs -f

Comment 8 Emma Heftman 2018-06-03 11:42:20 UTC
(In reply to Rich Megginson from comment #6)
> (In reply to Emma Heftman from comment #2)
> > Hey Shirly
> > Could you please provide some information about 
> > - where the logs are located? 
> > - when they should be used, i.e. which specific scenarios would require
> > using these logs to resolve the issue?
> > - what is the format of the logs?
> > 
> > Thanks!
> 
> 1. determine the name of the project used by the logging components
> 
> oc get projects | grep logging
> 
> If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> will refer to this project name as $PROJECT below.
> 
> 2. get the name of an elasticsearch pod
> 
> oc -n $PROJECT get pods -l component=es
> 
> I will refer to the name of an es pod returned from this command as $espod
> below.
> 
> 3. examine the log files inside the container
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs
> 
> This will just dump the log file to stdout
> 
> If you want to look for a particular string in the logs use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> 
> If you want to tail the logs, use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs -f

Thanks Rich.
If there is a problem loading a specific pod, how can they export the output to a file in order to send to support?

Comment 9 Shirly Radco 2018-06-03 11:44:41 UTC
(In reply to Jeff Cantrill from comment #3)
> This utility was added to the Elasticsearch image to extract logs since we
> no longer write to STDOUT by default:
> https://github.com/openshift/origin-aggregated-logging/blob/master/
> elasticsearch/utils/logs

How do we run this util?

Comment 10 Rich Megginson 2018-06-04 01:12:42 UTC
(In reply to Emma Heftman from comment #8)
> (In reply to Rich Megginson from comment #6)
> > (In reply to Emma Heftman from comment #2)
> > > Hey Shirly
> > > Could you please provide some information about 
> > > - where the logs are located? 
> > > - when they should be used, i.e. which specific scenarios would require
> > > using these logs to resolve the issue?
> > > - what is the format of the logs?
> > > 
> > > Thanks!
> > 
> > 1. determine the name of the project used by the logging components
> > 
> > oc get projects | grep logging
> > 
> > If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> > will refer to this project name as $PROJECT below.
> > 
> > 2. get the name of an elasticsearch pod
> > 
> > oc -n $PROJECT get pods -l component=es
> > 
> > I will refer to the name of an es pod returned from this command as $espod
> > below.
> > 
> > 3. examine the log files inside the container
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs
> > 
> > This will just dump the log file to stdout
> > 
> > If you want to look for a particular string in the logs use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> > 
> > If you want to tail the logs, use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs -f
> 
> Thanks Rich.
> If there is a problem loading a specific pod, how can they export the output
> to a file in order to send to support?

Use 'oc -n $PROJECT -c elasticsearch $espod logs' to get the container logs - but since elasticsearch is logging to the file, this container log will contain very little, but it may be useful.

The other alternative is to configure elasticsearch with persistent logs - @jcantrill - is this documented?

Comment 11 Rich Megginson 2018-06-04 01:13:22 UTC
(In reply to Shirly Radco from comment #9)
> (In reply to Jeff Cantrill from comment #3)
> > This utility was added to the Elasticsearch image to extract logs since we
> > no longer write to STDOUT by default:
> > https://github.com/openshift/origin-aggregated-logging/blob/master/
> > elasticsearch/utils/logs
> 
> How do we run this util?

As I have described in https://bugzilla.redhat.com/show_bug.cgi?id=1566138#c6

Comment 12 Shirly Radco 2018-06-04 08:17:12 UTC
(In reply to Rich Megginson from comment #6)
> (In reply to Emma Heftman from comment #2)
> > Hey Shirly
> > Could you please provide some information about 
> > - where the logs are located? 
> > - when they should be used, i.e. which specific scenarios would require
> > using these logs to resolve the issue?
> > - what is the format of the logs?
> > 
> > Thanks!
> 
> 1. determine the name of the project used by the logging components
> 
> oc get projects | grep logging
> 
> If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> will refer to this project name as $PROJECT below.
> 
> 2. get the name of an elasticsearch pod
> 
> oc -n $PROJECT get pods -l component=es
> 
> I will refer to the name of an es pod returned from this command as $espod
> below.
> 
> 3. examine the log files inside the container
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs
> 

This and below did not work for me in ocp 3.9.
I used:
oc logs -n $PROJECT -c elasticsearch $espod

That worked for the examples below too.

Is there a difference in the ocp versions when using the logs util?

> This will just dump the log file to stdout
> 
> If you want to look for a particular string in the logs use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> 
> If you want to tail the logs, use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs -f

Comment 13 Jeff Cantrill 2018-06-04 13:41:03 UTC
There is no need to specifically configure for specific logs as we moved them to the nodes persistent directory with [1].  This was also backported to 3.9 and is available in v3.9.30-1 or newer.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1568361

Comment 14 Rich Megginson 2018-06-04 15:41:24 UTC
(In reply to Shirly Radco from comment #12)
> (In reply to Rich Megginson from comment #6)
> > (In reply to Emma Heftman from comment #2)
> > > Hey Shirly
> > > Could you please provide some information about 
> > > - where the logs are located? 
> > > - when they should be used, i.e. which specific scenarios would require
> > > using these logs to resolve the issue?
> > > - what is the format of the logs?
> > > 
> > > Thanks!
> > 
> > 1. determine the name of the project used by the logging components
> > 
> > oc get projects | grep logging
> > 
> > If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> > will refer to this project name as $PROJECT below.
> > 
> > 2. get the name of an elasticsearch pod
> > 
> > oc -n $PROJECT get pods -l component=es
> > 
> > I will refer to the name of an es pod returned from this command as $espod
> > below.
> > 
> > 3. examine the log files inside the container
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs
> > 
> 
> This and below did not work for me in ocp 3.9.
> I used:
> oc logs -n $PROJECT -c elasticsearch $espod
> 
> That worked for the examples below too.
> 
> Is there a difference in the ocp versions when using the logs util?

oc logs -n $PROJECT -c elasticsearch $espod

will always print something.  But what it prints depends on the logger setting:

oc get -n $PROJECT cm/logging-elasticsearch -o yaml | grep rootLogger
    rootLogger: ${es.logger.level}, console

If you see "console" here, it means "oc logs" will contain all of the elasticsearch logs.  If you see "file" here and _not_ "console", it means elasticsearch is logging to a log file and not the console, which means "oc logs" will not show you everything.  In that case, you must look at the log file.  Note that "file" is now the default, so it will almost always be "file".  The "logs" command as discussed elsewhere _can_ be used, but _does not have to be used_ to look at the log file - it is merely a convenience, not required.


> 
> > This will just dump the log file to stdout
> > 
> > If you want to look for a particular string in the logs use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> > 
> > If you want to tail the logs, use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs -f

Comment 16 Sandro Bonazzola 2019-01-28 09:42:10 UTC
This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

Comment 22 Jeff Cantrill 2019-05-13 14:24:10 UTC
(In reply to Shirly Radco from comment #21)
> Yes. We need documentation for users to be able to get and attach the logs
> to the bugs.

You use [1] to pull logs that are persisted to disk or [2] to dump the environment which also grabs these logs too:

[1]https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs
[2]https://github.com/openshift/origin-aggregated-logging/blob/master/hack/logging-dump.sh

Comment 28 Sandro Bonazzola 2019-06-21 08:14:27 UTC
Despite this is a documentation bug, QE has no residual capacity for testing this in 4.3.4, re-targeting to 4.3.5


Note You need to log in before you can comment on or make changes to this bug.