Red Hat Bugzilla – Bug 1462584
[upgrade]Empty pods' log after upgrade to OCP 3.6
Last modified: 2017-08-25 16:38:16 EDT
Description of problem:
Deployed logging 3.5 on OCP 126.96.36.199, and then Upgrade OCP to 3.6.112, es and kibana pods' log are empty
Version-Release number of selected component (if applicable):
# openshift version
# oc get po
NAME READY STATUS RESTARTS AGE
logging-curator-1-sdj1h 1/1 Running 0 5h
logging-es-sxdc14oi-1-mxpcm 1/1 Running 0 5h
logging-fluentd-dc7t4 1/1 Running 0 2d
logging-fluentd-n3dvf 1/1 Running 3 2d
logging-kibana-1-d4bls 2/2 Running 1 5h
Steps to Reproduce:
1. Deploy logging on OCP 188.8.131.52
2. Upgrade OCP to 3.6.112
3. Check logging pods' log.
3. es and kibana pods' log are empty
All pods' log should be retrieved
Please provide additional information of how you are retrieving the logs. The ES log appender was changed  so that ES does not degrade due to a logging feed back loop ingesting its own logs.
By Kibana logs are empty, do you mean as in 'oc logs $KIBANA_POD'?
Please provide as much of the relevant details of of the logging stack as identified here: https://github.com/openshift/origin-aggregated-logging/blob/master/docs/issues.md
It is a little weird, kibana pods log could be viewed by console, but there are a few lines there. es pods log could not be viewed by console, but can be found under /elasticsearch/logging-es/logs of es pod, see the attached log
Created attachment 1289420 [details]
Is the symptom reproducible?
Every time you upgrade from 3.5 to 3.6, you see empty log output from "oc logs"? It's not clear to me the issue was triggered by the upgrade or it had existed before the upgrade... Did you have a chance to run "oc logs" on 3.5 just before upgrading it to 3.6?
Also, when you could duplicate the problem, could you try to gather more info?
For instance, please make sure os logs returns nothing.
$ oc logs $espod
Then, could you try oc logs options such as these?
--limit-bytes=1024: Maximum bytes of logs to return. Defaults to no limit.
--previous=true: If true, print the logs for the previous instance of the container in a pod if it exists.
$ oc exec $espod -- ls -l /elasticsearch/logging-es/logs
-rw-r--r--. 1 1000050000 root 8836 Aug 8 00:01 logging-es.log
-rw-r--r--. 1 1000050000 root 19168 Aug 7 23:55 logging-es.log.2017-08-07
$ oc exec $espod -- id
$ oc exec $espod -- cat /elasticsearch/logging-es/logs/logging-es.log
Any other inputs would be appreciated.
(In reply to Noriko Hosoi from comment #4)
> Hi @Junqi,
> Is the symptom reproducible?
> Every time you upgrade from 3.5 to 3.6, you see empty log output from "oc
> logs"? It's not clear to me the issue was triggered by the upgrade or it
> had existed before the upgrade... Did you have a chance to run "oc logs" on
> 3.5 just before upgrading it to 3.6?
I did not have environment now, will attach more info when I do upgrade work later
Closing as this is not a bug but by design . This was changed to avoid a feedback loop observed by Peter Portante where ES ingests its own logs and can get into a state where it is no longer usable.