Bug 1566138

Emma, I notice this one doesn't have any details attached. Do you have any more information about what's required, or can you flag this to the Metrics team as needing planning input?

Hey Shirly
Could you please provide some information about 
- where the logs are located? 
- when they should be used, i.e. which specific scenarios would require using these logs to resolve the issue?
- what is the format of the logs?

Thanks!

This utility was added to the Elasticsearch image to extract logs since we no longer write to STDOUT by default: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs

Hi Rich,

Can you pleas provide the steps how to use the util / get the Elasticsearch logs?

Also the additional information Emma asked in comment #2

(In reply to Emma Heftman from comment #2)
> Hey Shirly
> Could you please provide some information about 
> - where the logs are located? 
> - when they should be used, i.e. which specific scenarios would require
> using these logs to resolve the issue?
> - what is the format of the logs?
> 
> Thanks!

1. determine the name of the project used by the logging components

oc get projects | grep logging

If "openshift-logging" is in that list, use it, otherwise, use "logging".  I will refer to this project name as $PROJECT below.

2. get the name of an elasticsearch pod

oc -n $PROJECT get pods -l component=es

I will refer to the name of an es pod returned from this command as $espod below.

3. examine the log files inside the container

oc -n $PROJECT -c elasticsearch $espod -- logs

This will just dump the log file to stdout

If you want to look for a particular string in the logs use

oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"

If you want to tail the logs, use

oc -n $PROJECT -c elasticsearch $espod -- logs -f

For ref: https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs#L27

(In reply to Rich Megginson from comment #6)
> (In reply to Emma Heftman from comment #2)
> > Hey Shirly
> > Could you please provide some information about 
> > - where the logs are located? 
> > - when they should be used, i.e. which specific scenarios would require
> > using these logs to resolve the issue?
> > - what is the format of the logs?
> > 
> > Thanks!
> 
> 1. determine the name of the project used by the logging components
> 
> oc get projects | grep logging
> 
> If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> will refer to this project name as $PROJECT below.
> 
> 2. get the name of an elasticsearch pod
> 
> oc -n $PROJECT get pods -l component=es
> 
> I will refer to the name of an es pod returned from this command as $espod
> below.
> 
> 3. examine the log files inside the container
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs
> 
> This will just dump the log file to stdout
> 
> If you want to look for a particular string in the logs use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> 
> If you want to tail the logs, use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs -f

Thanks Rich.
If there is a problem loading a specific pod, how can they export the output to a file in order to send to support?

(In reply to Jeff Cantrill from comment #3)
> This utility was added to the Elasticsearch image to extract logs since we
> no longer write to STDOUT by default:
> https://github.com/openshift/origin-aggregated-logging/blob/master/
> elasticsearch/utils/logs

How do we run this util?

(In reply to Emma Heftman from comment #8)
> (In reply to Rich Megginson from comment #6)
> > (In reply to Emma Heftman from comment #2)
> > > Hey Shirly
> > > Could you please provide some information about 
> > > - where the logs are located? 
> > > - when they should be used, i.e. which specific scenarios would require
> > > using these logs to resolve the issue?
> > > - what is the format of the logs?
> > > 
> > > Thanks!
> > 
> > 1. determine the name of the project used by the logging components
> > 
> > oc get projects | grep logging
> > 
> > If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> > will refer to this project name as $PROJECT below.
> > 
> > 2. get the name of an elasticsearch pod
> > 
> > oc -n $PROJECT get pods -l component=es
> > 
> > I will refer to the name of an es pod returned from this command as $espod
> > below.
> > 
> > 3. examine the log files inside the container
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs
> > 
> > This will just dump the log file to stdout
> > 
> > If you want to look for a particular string in the logs use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> > 
> > If you want to tail the logs, use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs -f
> 
> Thanks Rich.
> If there is a problem loading a specific pod, how can they export the output
> to a file in order to send to support?

Use 'oc -n $PROJECT -c elasticsearch $espod logs' to get the container logs - but since elasticsearch is logging to the file, this container log will contain very little, but it may be useful.

The other alternative is to configure elasticsearch with persistent logs - @jcantrill - is this documented?

(In reply to Shirly Radco from comment #9)
> (In reply to Jeff Cantrill from comment #3)
> > This utility was added to the Elasticsearch image to extract logs since we
> > no longer write to STDOUT by default:
> > https://github.com/openshift/origin-aggregated-logging/blob/master/
> > elasticsearch/utils/logs
> 
> How do we run this util?

As I have described in https://bugzilla.redhat.com/show_bug.cgi?id=1566138#c6

(In reply to Rich Megginson from comment #6)
> (In reply to Emma Heftman from comment #2)
> > Hey Shirly
> > Could you please provide some information about 
> > - where the logs are located? 
> > - when they should be used, i.e. which specific scenarios would require
> > using these logs to resolve the issue?
> > - what is the format of the logs?
> > 
> > Thanks!
> 
> 1. determine the name of the project used by the logging components
> 
> oc get projects | grep logging
> 
> If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> will refer to this project name as $PROJECT below.
> 
> 2. get the name of an elasticsearch pod
> 
> oc -n $PROJECT get pods -l component=es
> 
> I will refer to the name of an es pod returned from this command as $espod
> below.
> 
> 3. examine the log files inside the container
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs
> 

This and below did not work for me in ocp 3.9.
I used:
oc logs -n $PROJECT -c elasticsearch $espod

That worked for the examples below too.

Is there a difference in the ocp versions when using the logs util?

> This will just dump the log file to stdout
> 
> If you want to look for a particular string in the logs use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> 
> If you want to tail the logs, use
> 
> oc -n $PROJECT -c elasticsearch $espod -- logs -f

There is no need to specifically configure for specific logs as we moved them to the nodes persistent directory with [1].  This was also backported to 3.9 and is available in v3.9.30-1 or newer.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1568361

(In reply to Shirly Radco from comment #12)
> (In reply to Rich Megginson from comment #6)
> > (In reply to Emma Heftman from comment #2)
> > > Hey Shirly
> > > Could you please provide some information about 
> > > - where the logs are located? 
> > > - when they should be used, i.e. which specific scenarios would require
> > > using these logs to resolve the issue?
> > > - what is the format of the logs?
> > > 
> > > Thanks!
> > 
> > 1. determine the name of the project used by the logging components
> > 
> > oc get projects | grep logging
> > 
> > If "openshift-logging" is in that list, use it, otherwise, use "logging".  I
> > will refer to this project name as $PROJECT below.
> > 
> > 2. get the name of an elasticsearch pod
> > 
> > oc -n $PROJECT get pods -l component=es
> > 
> > I will refer to the name of an es pod returned from this command as $espod
> > below.
> > 
> > 3. examine the log files inside the container
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs
> > 
> 
> This and below did not work for me in ocp 3.9.
> I used:
> oc logs -n $PROJECT -c elasticsearch $espod
> 
> That worked for the examples below too.
> 
> Is there a difference in the ocp versions when using the logs util?

oc logs -n $PROJECT -c elasticsearch $espod

will always print something.  But what it prints depends on the logger setting:

oc get -n $PROJECT cm/logging-elasticsearch -o yaml | grep rootLogger
    rootLogger: ${es.logger.level}, console

If you see "console" here, it means "oc logs" will contain all of the elasticsearch logs.  If you see "file" here and _not_ "console", it means elasticsearch is logging to a log file and not the console, which means "oc logs" will not show you everything.  In that case, you must look at the log file.  Note that "file" is now the default, so it will almost always be "file".  The "logs" command as discussed elsewhere _can_ be used, but _does not have to be used_ to look at the log file - it is merely a convenience, not required.


> 
> > This will just dump the log file to stdout
> > 
> > If you want to look for a particular string in the logs use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs | grep "some string"
> > 
> > If you want to tail the logs, use
> > 
> > oc -n $PROJECT -c elasticsearch $espod -- logs -f

This bug has not been marked as blocker for oVirt 4.3.0.
Since we are releasing it tomorrow, January 29th, this bug has been re-targeted to 4.3.1.

(In reply to Shirly Radco from comment #21)
> Yes. We need documentation for users to be able to get and attach the logs
> to the bugs.

You use [1] to pull logs that are persisted to disk or [2] to dump the environment which also grabs these logs too:

[1]https://github.com/openshift/origin-aggregated-logging/blob/master/elasticsearch/utils/logs
[2]https://github.com/openshift/origin-aggregated-logging/blob/master/hack/logging-dump.sh

Despite this is a documentation bug, QE has no residual capacity for testing this in 4.3.4, re-targeting to 4.3.5