Bug 1559055 - [Infra] docker logs opendaylight_api command doesn't show the OpenDaylight controller's log
Summary: [Infra] docker logs opendaylight_api command doesn't show the OpenDaylight co...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: z1
: 13.0 (Queens)
Assignee: Janki
QA Contact: Tomas Jamrisko
URL:
Whiteboard: Infra
Depends On:
Blocks: 1571365
TreeView+ depends on / blocked
 
Reported: 2018-03-21 15:39 UTC by Itzik Brown
Modified: 2018-10-18 07:19 UTC (History)
15 users (show)

Fixed In Version: openstack-tripleo-heat-templates-8.0.2-40.el7ost puppet-opendaylight-8.1.2-2.38977efgit.el7ost
Doc Type: Known Issue
Doc Text:
OpenDaylight logging might be missing earlier logs. This is a known issue with journald logging of OpenDaylight (using the “docker logs opendaylght_api” command). The current workaround is to switch OpenDaylight logging to the “file” mechanism which will log inside of the container to /opt/opendaylight/data/logs/karaf.log. To do this, configure the following heat parameter: OpenDaylightLogMechanism: ‘file’.
Clone Of:
Environment:
N/A
Last Closed: 2018-07-19 14:27:12 UTC
Target Upstream Version:


Attachments (Terms of Use)
Container's karaf.log (2.41 MB, text/plain)
2018-04-08 07:53 UTC, Itzik Brown
no flags Details
Docker log command output (53.66 KB, text/plain)
2018-04-08 07:53 UTC, Itzik Brown
no flags Details
org.ops4j.pax.logging.cfg (3.97 KB, text/plain)
2018-04-19 05:10 UTC, Itzik Brown
no flags Details
ODL HA controllers karaf logs (131.84 KB, text/plain)
2018-07-10 12:42 UTC, Noam Manos
no flags Details


Links
System ID Private Priority Status Summary Last Updated
OpenDaylight Bug INTPAK-123 0 None None None 2018-03-21 15:42:48 UTC
OpenDaylight gerrit 66933 0 None None None 2018-03-21 16:02:57 UTC
OpenStack gerrit 566270 0 None master: MERGED tripleo-heat-templates: Mount ODL log file on host node (Ib1275d42daf2896db9e6de6513070a0fba47ed40) 2018-06-29 13:42:15 UTC
OpenStack gerrit 567469 0 None stable/queens: MERGED tripleo-heat-templates: Mount ODL log file on host node (Ib1275d42daf2896db9e6de6513070a0fba47ed40) 2018-06-29 13:42:09 UTC
Red Hat Product Errata RHSA-2018:2214 0 None None None 2018-07-19 14:28:01 UTC

Description Itzik Brown 2018-03-21 15:39:37 UTC
Description of problem:
When running the following command:
# docker logs   opendaylight_api

The output is as follows

+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /opt/opendaylight/data/idmlight.db.mv.db
INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/data/idmlight.db.mv.db to /opt/opendaylight/data/idmlight.db.mv.db
INFO:__main__:Deleting /opt/opendaylight/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml
INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml to /opt/opendaylight/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml
INFO:__main__:Deleting /opt/opendaylight/etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml
INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml to /opt/opendaylight/etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml
INFO:__main__:Deleting /opt/opendaylight/etc/jetty.xml
INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/etc/jetty.xml to /opt/opendaylight/etc/jetty.xml
Running command: '/opt/opendaylight/bin/karaf server'
karaf: JAVA_HOME not set; results may vary
Apache Karaf starting up. Press Enter to open the shell now...
100% [========================================================================]
Karaf started in 24s. Bundle stats: 475 active, 476 total

Version-Release number of selected component (if applicable):
puppet-opendaylight-7.0.0-0.20180216174116.d3b32a4.el7ost.noarch

How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 4 Itzik Brown 2018-03-30 14:13:41 UTC
Logs are present when using docker logs
puppet-opendaylight-8.1.0-0.20180321182556.45c4db7.el7ost.noarch

Comment 7 Itzik Brown 2018-04-08 07:53:22 UTC
Created attachment 1418817 [details]
Container's karaf.log

Comment 8 Itzik Brown 2018-04-08 07:53:54 UTC
Created attachment 1418818 [details]
Docker log command output

Comment 9 Itzik Brown 2018-04-08 07:55:55 UTC
Janki,
I attached the karaf.log from the container and the docker logs command output.
They are different.

Please advise.

Comment 10 Tim Rozet 2018-04-11 21:11:19 UTC
Looks like the docker logs is only picking up the 2nd half of the logs.  It looks like with docker inspect the logging driver being used is journald:

              "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
More info here:

https://docs.docker.com/config/containers/logging/configure/


It looks like for some logging drivers there is config you can add for rollover or max log length.  I'm not sure exactly if this is the problem or not, or if these options are even configurable for journald driver:
https://docs.docker.com/config/containers/logging/journald/


One cool thing I found out is you can do this and actually see the logs in journalctl (however they do not show the full logs, same as docker logs output)
journalctl CONTAINER_NAME=opendaylight_api

If I restart the container I see kolla logging messages as expected:
Apr 11 21:05:31 controller-0 dockerd-current[32771]: + sudo -E kolla_set_configs
Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Validating config file
Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Copying service configuration files
Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Deleting /opt/opendaylight/bin/karaf
Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/bin/karaf to /opt/opendaylight/bin/karaf


I think ODL and the container are working fine.  There was something that prevented journald from collecting the logs.  Not sure what it is yet.  We could switch the driver to use the default json driver:
https://docs.docker.com/config/containers/logging/json-file/#usage

But I'm guessing journald is better and is there is a reason we are already using that driver.

Comment 12 Tim Rozet 2018-04-16 21:01:04 UTC
Did you ever have a chance to test to see whether these missing logs only happen during a deployment and the initial bring up?  Are you able to detect missing logs from a later point in time?  How reproducible is the issue?

Comment 13 Janki 2018-04-17 01:52:18 UTC
I tested this locally before pushing the patch. It was working then. All the logs were seen by "docker logs" even after restart. I also kept ODL running for an hour and the logs were still seen. The problem here, I think, is puppet is not configuring log config file properly. There should be line "karaf.log.console=INFO" in '/opt/opendaylight/etc/org.ops4j.pax.logging.cfg'
after 'log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter' which is missing. The karaf.log file on container should only contain initial 3 4 lines stating that karaf is started and nothing else.

Comment 14 Michael Vorburger 2018-04-18 17:55:37 UTC
From reading all of the above, I'm not crystal clear what exact problem this issue reports.  IMHO it would be helpful if we wrote down the "Expected results:" VS the "Actual results:" and agreed on that, first.

If the Expected results are that there is nothing in the karaf.log, then the logging configuration is wrong, and (it seems) is still sending it to both the karaf.log file as well as STDOUT (given that you ARE seeing things in "docker log").

If the Expected results are that there should be MORE, everything, in "docker logs", and the complaint here specifically is that it is missing things from the start, then that is most certainly not an ODL configuration issue.  It would just seem that there is some "rolling over" happening - on the docker/journald side of things.

Could someone attach the /opt/opendaylight/etc/org.ops4j.pax.logging.cfg from inside the container here so that we can easily have a look at it?  It's probably better not to wonder at TripleO / Puppet sources which patch this file but to see the final result.

Comment 15 Itzik Brown 2018-04-19 05:10:36 UTC
Created attachment 1423904 [details]
org.ops4j.pax.logging.cfg

Comment 16 Michael Vorburger 2018-04-19 16:01:44 UTC
just FTR: The original issue which introduced this was Bug 1489704.

> There should be line "karaf.log.console=INFO" in 
> '/opt/opendaylight/etc/org.ops4j.pax.logging.cfg'

The org.ops4j.pax.logging.cfg which Itzik attached does have this, and that is what is (correctly) causing Karaf to also send its log to the "console" (=STDOUT), where a Container engine (Docker) captures it from and sends it to journald (based on what was said above).

> after 

I am seeing "karaf.log.console=INFO" before not after "log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF}" but that makes no difference (I tested it).

> 'log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter' which is missing. 

No, the attached org.ops4j.pax.logging.cfg which Itzik attached does have this, All is Wehl.

> The karaf.log file on container should only contain initial 3 4 lines stating that karaf is started and nothing else.

No, that's not what you configured in the attached org.ops4j.pax.logging.cfg.  The way it currently is, Karaf will send its log to both STDOUT as well as to the RollingRandomAccessFile karaf.log.%i.  (I've locally tested this; no need for Docker/container/Kolla/TripleO; just try in an upstream Karaf.)

> the karaf.log from the container and the docker logs command output.
> They are different.

As Tim said in #c10 : "It looks like for some logging drivers there is config you can add for rollover or max log length."  I strongly suspect that is the only reason for this.

I'm not crystal clear what exact problem this issue reports.

Comment 18 Sai Sindhur Malleni 2018-04-20 14:24:51 UTC
Just to add here, the karaf logs on the container are getting rotated for every 17M of log data. Earlier they were rotated at every 10G of log data.

Comment 19 Michael Vorburger 2018-04-20 14:56:46 UTC
> the karaf logs on the container are getting rotated for every 17M of log data. > Earlier they were rotated at every 10G of log data.

That's how it's configured: log4j2.appender.rolling.policies.size.size = 16MB

Raise a separate new BZ if you consider this a regression which should be changed back?

Comment 20 Janki 2018-04-20 17:04:33 UTC
I did some playing around locally. I tried adding variations of
"log4j2.rootLogger.appenderRefs = Console/console/stdout/STDOUT/console.stdout". But nothing works.

Commenting "log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile" does the trick. Logs are logged only to the console and NOT to file.

Is this a right way to go?

Comment 21 Michael Vorburger 2018-04-23 09:06:54 UTC
> Commenting "log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile" does > the trick. Logs are logged only to the console and NOT to file.
> Is this a right way to go?

yes, that is the right way to prevent all (rootLogger) logs to be sent to the RollingFile appender.

Strictly speaking though, that is not what this bug report is about.  I would suggest that we conclude this bug report raised by QA with how  "docker log" and/or 'journald' needs to be used to see older log entries, and perhaps just better document that somehwere?  Then "case closed" on this one.

I personally then do think that we also should change things to not log to the karaf.log file anymore as you are getting at here, but would suggest that is looked at in a separate new BZ.

Comment 22 Michael Vorburger 2018-04-24 15:47:19 UTC
> also should change things to not log to the karaf.log file anymore as you are 
> getting at here, but would suggest that is looked at in a separate new BZ.

==> new bug 1571365

Comment 23 Janki 2018-04-30 16:30:30 UTC
I have a setup where ODL Is running for past 2 days.

d46b72b370e8        192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-04-24.1                  "kolla_start"            2 days ago          Up 2 days (unhealthy)                       opendaylight_api


$ date
Mon Apr 30 06:00:23 UTC 2018


$ sudo docker logs opendaylight_api shows logs right from start
Running command: '/opt/opendaylight/bin/karaf server'
karaf: JAVA_HOME not set; results may vary
Apache Karaf starting up. Press Enter to open the shell now...
^M  8% [=====>                                                                  ]^M 16% [===========>                                                            ]^M 25% [==================>                                                     ]^M 33% [=======================>                                                ]^M 41% [=============================>                                          ]^M 50% [====================================>                                   ]^M 58% [=========================================>                              ]^M 66% [===============================================>                        ]^M 75% [======================================================>                 ]^M 83% [===========================================================>            ]^M 91% [=================================================================>      ]^M100% [========================================================================]
Karaf started in 0s. Bundle stats: 12 active, 12 total
2018-04-27T13:15:23,541 | INFO  | pool-1-thread-2  | FeaturesServiceImpl              | 7 - org.apache.karaf.features.core - 4.1.3 | Adding features: standard/[4.1.3,4.1.3], wrap/[0,0.0.0], ssh/[4.1.3,4.1.3], odl-mdsal-trace/[1.7.0.redhat-5,1.7.0.redhat-5], odl-netvirt-openstack/[0.6.0.redhat-5,0.6.0.redhat-5], odl-jolokia/[1.10.0.redhat-5,1.10.0.redhat-5]

last line at 2018-04-30T05:57:55,821

$ journalctl --since "2018-04-27"
-- Logs begin at Fri 2018-04-27 13:01:20 UTC, end at Mon 2018-04-30 06:04:41 UTC. --

So 2 days old are still available in docker logs.

Comment 24 Mike Kolesnik 2018-05-06 08:39:10 UTC
Hi Janki,

What exactly is the bug currently?

Comment 25 Janki 2018-05-09 10:14:40 UTC
Mike,

Bug is journal is rolling over the logs every 2 days 3 hours approximately. We can configure this rollover in journal.conf file. We need to consult journal team (if any) and analyse impact of config change will have on other OpenStack services.

Meanwhile, I can adding logic to mount karaf.log to overcloud controller node from ODL container.
https://review.openstack.org/#/c/566270/

Comment 30 Mike Kolesnik 2018-05-21 12:50:07 UTC
Janki, I see the patch you sent on master got merged, what's the status of this bug?

Comment 31 Janki 2018-05-21 12:56:55 UTC
Cherry-pick to Queens is yet to be merged. CI is failing because of unrelated issue.

https://review.openstack.org/#/c/567469/

Once it is merged, we can access karaf.log will be mounted on controller hosts too and we will be reading logs from /var/log/contaiers/opendaylight.

Comment 32 Janki 2018-05-30 16:36:41 UTC
Hi Itzik

Could you please provide with below command's output on your system where this behavior is observed.

$ journalctl --since yesterday
$ ls -1s /var/log/journal/*/
$ journalctl --disk-usage
[odl@overcloud-controller-0 /opt/opendaylight/data/log]$ ls -sh (Note this gives size of karaf.log inside the container)
$ df -h

Comment 33 Itzik Brown 2018-05-31 06:42:36 UTC
First line of 'journalctl --since yesterday'
-- Logs begin at Thu 2018-05-31 00:32:22 UTC, end at Thu 2018-05-31 06:36:22 UTC. -
====

[root@controller-0 heat-admin]# ls -1s /var/log/journal/*/
total 565304
65540 system@c03707aad97242c7993472e3ba9c2601-00000000000a355a-00056d7599ce8098.journal
65540 system@c03707aad97242c7993472e3ba9c2601-00000000000c2ba2-00056d76da3abcba.journal
65540 system@c03707aad97242c7993472e3ba9c2601-00000000000e208f-00056d781d49f266.journal
65540 system@c03707aad97242c7993472e3ba9c2601-0000000000101916-00056d795f173990.journal
 8196 system.journal
 8192 user-1000.journal
 8192 user-42434.journal
57348 user-42445@0a4f24c4e68a493c802d0b8c0de7c139-00000000000a358c-00056d759a13b7cb.journal
57348 user-42445@0a4f24c4e68a493c802d0b8c0de7c139-00000000000c2bcc-00056d76da922406.journal
57348 user-42445@0a4f24c4e68a493c802d0b8c0de7c139-00000000000e2138-00056d781dc3632d.journal
57348 user-42445@0a4f24c4e68a493c802d0b8c0de7c139-000000000010192c-00056d795f2c1647.journal
 8196 user-42445.journal
 8196 user-42454@0eee94acd5724a06afdf3f37940a0b14-00000000000a3654-00056d759a723c13.journal
 8196 user-42454@0eee94acd5724a06afdf3f37940a0b14-00000000000c2bab-00056d76da4f1359.journal
 8196 user-42454@0eee94acd5724a06afdf3f37940a0b14-00000000000e21fe-00056d781e31fbb5.journal
 8196 user-42454@0eee94acd5724a06afdf3f37940a0b14-000000000010194e-00056d795f79c814.journal
 8192 user-42454.journal

======
[root@controller-0 heat-admin]# journalctl --disk-usage
Archived and active journals take up 552.0M on disk.
=====

()[odl@controller-0 /opt/opendaylight/data/log]$ ls -sh 
total 1.9M
1.9M karaf.log
=====
()[odl@controller-0 /opt/opendaylight/data/log]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay         187G   14G  173G   8% /
tmpfs            32G     0   32G   0% /dev
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/sda2       187G   14G  173G   8% /etc/hosts
devtmpfs         32G     0   32G   0% /dev/log
shm              64M     0   64M   0% /dev/shm
tmpfs            32G     0   32G   0% /proc/scsi
=====

Comment 34 Janki 2018-05-31 12:22:44 UTC
Itzik, df -h from the controller node and not from the ODL container. and also $ date from  controller node please.

Comment 35 Itzik Brown 2018-05-31 12:32:39 UTC
[root@controller-0 heat-admin]# date
Thu May 31 12:31:58 UTC 2018
[root@controller-0 heat-admin]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       187G   14G  173G   8% /
devtmpfs         32G     0   32G   0% /dev
tmpfs            32G   54M   32G   1% /dev/shm
tmpfs            32G   12M   32G   1% /run
tmpfs            32G     0   32G   0% /sys/fs/cgroup
tmpfs           6.3G     0  6.3G   0% /run/user/975
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/5ade1ea00de2f445b1362fcdef1e14381dfd45fdfd37b6c7707b6b36043ce69f/merged
shm              64M     0   64M   0% /var/lib/docker/containers/76fb1260f887e53eb6e77466c2bed652f2f23b4475417b30e0fb450bb60db013/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/e6d2533d8c61fdc2502fc6757499aa2edd9f9b2911187fa7b4faf07bc2d71971/merged
shm              64M     0   64M   0% /var/lib/docker/containers/b60ba67401e8f4382eb53ed3c335ec6d950e6eff0c98d914484212b226f35570/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/893aa8d3ca100930c07365345194f62f5aab2e76f3a2c0583a6a2267d05e2e55/merged
shm              64M     0   64M   0% /var/lib/docker/containers/ee51f7cc33537535b96391db8a87313d5c729dc7b4deb74b2c4046837f017917/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/a369b2d9558efb45ffc461bc4f58fcf68cc9ee81559fb788e289db0a51c5cc4e/merged
shm              64M     0   64M   0% /var/lib/docker/containers/914fa26ca70a87289b26c5d13f80d31579571f3d52d2e954557bf56047f38b28/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/e894c407c91830895a8456a669ff1bc13127e0d5eb34918e27d028698d5e436d/merged
shm              64M     0   64M   0% /var/lib/docker/containers/76bd15627d4e51531547db37286d41694bf43ea4d223864f71f6f32a9508432a/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f52dc362399a55ac3976e1a92e217ce9e13f1863e7b5ca00926054e8ca349277/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d10a6c108e0121786d7c32fef3e54b548edc3371c2ae6e54ef947fde3dcd9f73/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f8bd37c6782375fdc603aaac6f9d1c492ff2e4898b09813940dcd9faecb1c19d/merged
shm              64M     0   64M   0% /var/lib/docker/containers/2a98efbd9c0fdf62c690e127e4ff1a24828b57c36361407908dcc69406329524/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/9afe780361a52fcb0d1723fd86834176a7feaaadde74df1eadb71dfc7d029696/merged
shm              64M     0   64M   0% /var/lib/docker/containers/c8f41604c3207327658c64cae255ee446975711167c4e80e570913d918b1de28/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/68e7d5c6da9d08799eb4c5af344e19b7a2c61984f3256bb2e80f97e5bf42a047/merged
shm              64M     0   64M   0% /var/lib/docker/containers/751a03b46ad3aa57a3c1005604a8da102ae9d60c4bb3c867df02c1c7a8fdb242/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/1cb6a3b4469466430399057890a34943919b2b321dc741da856bbf1d43807700/merged
shm              64M     0   64M   0% /var/lib/docker/containers/301a32527ef21766e7f649578faaf63c6bb126bec0764226fe60a8bde57b6b2d/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/3da1e9cdaf73f4a1de2c36ebd082c3d611381b551fdd328d2527c9bfaa9ed36c/merged
shm              64M     0   64M   0% /var/lib/docker/containers/f322a9c439efb909977abf2df218954fda0dbbe2a9b65269bd7beae7c22b2717/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/50236ffb0229c7672148d942976f0054c31f5111ac2681309842e5fa3d9bf5fd/merged
shm              64M     0   64M   0% /var/lib/docker/containers/8d85eb5ebd20e167fccdce5148c43a2f42e428d9c769210b59a4f507815bbd40/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/3635697f263002c2d8203466ed8225c9a72fbbdd6d84773f4a31d821e5a3af73/merged
shm              64M     0   64M   0% /var/lib/docker/containers/7295970fef0d6d1c6532c18cf0fbba08555fa0c7286ffe5f5bd7e8b584b0342f/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/8e7835c343c883e586006e16f4e13cb4b4834de4695029a2061446e6cca4444d/merged
shm              64M     0   64M   0% /var/lib/docker/containers/f39b0a88befc2a479582cf4585927221dfb439afa573972ab7f3e2f1046fb417/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f804ab6ba6546c4e6de6ca68ba0de47ee37dbfd669044c8abddccfeeebc22dfb/merged
shm              64M     0   64M   0% /var/lib/docker/containers/ae304a67997dd87cd790c4d22177d44de5d355cd8dbef07a7ccba57d92c32e05/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f335e1c82ea8e84c213767b7bc7dc21574222562c0f562c34c16bdced6a52517/merged
shm              64M  4.0K   64M   1% /var/lib/docker/containers/e13663002771e35b4dbaaf9b214aab43af5decf3eb3d704abc54c14351800438/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/9ba7dcd3d43c8aaf4cf644f697f0d44dc4746b0af6c735ed7fdde85ee0b70576/merged
shm              64M     0   64M   0% /var/lib/docker/containers/c1451c1ef4688b3afc4fc6017ad0c502aa77a2f57e8ac551168f77662e124729/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/07120937d02d2e1a4dfc34622d41990c2c4fa83972f1ba239ea14b5d5a41cb6d/merged
shm              64M     0   64M   0% /var/lib/docker/containers/a2d57e0bdd1283787cd96bac13d87342e506a9ac04c0946c19729713ab462345/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/cf1cd7fbc54c89e80d9607befec9b9885a879d04c8a14fc288a296b2499eca79/merged
shm              64M     0   64M   0% /var/lib/docker/containers/2c7417429a27ef7f42cd1a72d124645373e6952fef0fa03a9020ab93bae2fd43/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/0301ed923a35c66de473317b79b40260bfdbc6e7b4f27bbb91ec8ffa1fa23321/merged
shm              64M     0   64M   0% /var/lib/docker/containers/9f755e0f62ee79150004ca53d5c7101c4a5c4ca9359959199a6fc8374270722d/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/7735837127a2eaf2164922efc4ac6e86f4ca0837d93b43afb9e4b4e266502446/merged
shm              64M     0   64M   0% /var/lib/docker/containers/6cad4e22cfa8d1615ecdb055643d141bf78fc0ceccd05a565ca359b0a3918c13/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/39b5b4d5a3b496a2e12a401839582f55b0f43725c36a3c0884376e2b7083f442/merged
shm              64M     0   64M   0% /var/lib/docker/containers/6625384f27c1be2e14b633987670090b4d4c2bff1b99a44da006c135a2f5cd23/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f4ca02366ac8b33485bdb355cdc0559a07ab683bf0f1df64cda6ccf633368c3c/merged
shm              64M     0   64M   0% /var/lib/docker/containers/47a777affb087d32627ab7ff81e240369de4c0726ce0384dd4a0f72d18697a35/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/4f0cfade340518bd0c97850b53834c2c459ab47132c57b44339c6654c553ea97/merged
shm              64M     0   64M   0% /var/lib/docker/containers/c1254d0add4a0340f6ce33e87d72984cde91fc67f7c0b446e90c9092ff99ba03/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/4e18a92b9217104af10ca8b0ffc7a449d251bafcd9363375de0cb0a376092e5d/merged
shm              64M     0   64M   0% /var/lib/docker/containers/b6a9e0ce9bf1ad6b878e8e1f64a7ab1f8c14f322aebe137b0037d320a09ca532/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/cc2164ad6540e7e4a994fb576da53ee37113b6736de43f8c0fea7f67903b80e2/merged
shm              64M     0   64M   0% /var/lib/docker/containers/41d9e399ea527536bdf78738dc8bb60511b1fd4af238c6196cb34727d9d3654c/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/be9afa0a510e71b8514dfafdf88cba53c9668cad773ffb019c45bcfdd8ac0133/merged
shm              64M     0   64M   0% /var/lib/docker/containers/2ac88d7d5485916451d435de5e45cf64d93f6453e693d058764ddcf6ebc3d80d/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/ab720c32b4215c1e03bbf6da5dc72851f1a19059c56d4e17c2f64f65d737af38/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d663799f531c063f028fb5074a0fc7384aa280d40abdaf179906806605ad1a0b/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f7081435e6bdae23da162f369e294fc0bf679acf4dc33fbaa4b790b522172e1e/merged
shm              64M     0   64M   0% /var/lib/docker/containers/5caa78613855712f8a6a361b06097f24882c3554435a0e05bd7fc0ce24c3ff76/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/7ff6119918bec99cf3e61edb62856f2568dc9c9b64280e12f4ddcd60ee9aad52/merged
shm              64M     0   64M   0% /var/lib/docker/containers/78e7d6d08bb2ac5758fdeeb719a78055f8ddd6d26e64663cdea9c25091af2e35/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/79a0b292c8442d7e647c67f91b6eb674e718930f89a0ce0fd30c16f8fea755b6/merged
shm              64M     0   64M   0% /var/lib/docker/containers/6c2b38b3e799ad0a5d45113755e70304f40972b35b9e2418bef1a12485253b5b/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/c21ad6e84f857303aeffbcabda9f2eaa56454ce183a2c259a29730b454509771/merged
shm              64M     0   64M   0% /var/lib/docker/containers/b10f05f02e58b02f9b64f40846989362a1d1d47b1fb2b16544652b96e60b2677/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/6b98fa4ac38740397109f5e140f24584d4058fac76a2583f272594fbf6fca768/merged
shm              64M     0   64M   0% /var/lib/docker/containers/370b205029a7f716fb4e5e74f3b1193a388ea6502bc35935ab9812090e903dee/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/0e6cf250c99e9b8b7f26df1986c48a40555c694945cb95d647bccb285a0de454/merged
shm              64M     0   64M   0% /var/lib/docker/containers/b456ebfc45c1603a124f4773c15b0356d769cd3c0497aa1a0242b8e48d053402/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/15537cb4a545ee82afb08cd3a18139cf0ece9ea03369097e18871375c1cbd540/merged
shm              64M     0   64M   0% /var/lib/docker/containers/127589dc2f66213804360e43a3306c680017e7ad9b17627cd7734b630e293769/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/6fd6f6363d607500ffccc36adbc8d073451d3a509c71b7fe65bceb14e1d19fa1/merged
shm              64M     0   64M   0% /var/lib/docker/containers/a3739d776f25157a4f698c5f2ed22916881b55e050fafc22b530c1b647e83f69/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/7a06b61bc7e8316ec320687d732e26ceb085f0953dea9f0c93c70ef59816fcd9/merged
shm              64M     0   64M   0% /var/lib/docker/containers/8fc0fc4f26ecc23ad09c8e9e31ce58512301e5c1b5ca533396a78f50375ccd79/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/79c1bcdc552a15d3dba16db682041392c5797608091c80d37b740e1ae640e5ca/merged
shm              64M     0   64M   0% /var/lib/docker/containers/21c0c0f6f831c86d35e446d04474af5c7fab4a50c3b47d76258e287d71ec307f/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/1527c6785fdbdb81348a561d3fbef9bdc5c063c796f7f10ebe8020bdab3825da/merged
shm              64M     0   64M   0% /var/lib/docker/containers/96246915c9a306576cec673dbcce7c58a38a687dd810f42f1a19a735fab1ad15/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/7821b5477aa8392fb167dad528ba9b1142505759f3348286f3f1097ffc356946/merged
shm              64M     0   64M   0% /var/lib/docker/containers/c5c2b3e18481844e72a52eef2f3b8419ceefc065ceab5d24a3d23bd781a34312/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/39ef381c9b43da830a80e12be65fdbaa541aca7308300dab6448aa1ff2127484/merged
shm              64M     0   64M   0% /var/lib/docker/containers/2a26a3c8f8fec00d350ba304f0407c5359922968a69a050a114d4bcb69aca10c/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/8f649a5a1a25eba96e63f52d232d696f6f7bf330e687ca09611573fc9f579aea/merged
shm              64M     0   64M   0% /var/lib/docker/containers/3cec928459a9ea5ebf2bd3d1eb62bb71175075ff98a15c5aa8b43187b4bb2662/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/f39a08f13e5f6efc992445086a9407c14d5d4c016e46682d61aac7550dffce21/merged
shm              64M     0   64M   0% /var/lib/docker/containers/44db8ac70e2d3034fa37cd0bbee3e60cc26a2bb8dde5547eb822e2cb22df0824/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/72dbdadafb0986b175dfca674f130f7ba36a4b35cab528177d7d51395c2aaff7/merged
shm              64M     0   64M   0% /var/lib/docker/containers/670633195b4e465c40a7c8307f5b9e2926bf869bb9f028567984f4f1e8896078/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/267d8896601773335827251599d07cfab599fb3f322f21b20d464be482a21ffe/merged
shm              64M     0   64M   0% /var/lib/docker/containers/51fb98e82acd9f69112a2f919fbeb53fc0c952651928695cc2e6f158a7a5a21b/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/2762f604a80bd816f8a78b33b055a0e4d5f1293bc5712cbc061ded0713883d8e/merged
shm              64M     0   64M   0% /var/lib/docker/containers/4ac0387049320f38ac655011a5c42f4eb03a8560f98dc572538ce6cc2679481b/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/aae2f822c32aac750513adc1766fbf2f8ab58464a8c031557b8de94a0ada6716/merged
shm              64M     0   64M   0% /var/lib/docker/containers/e6db38a32563946672d7b6b5b651e737afa25d334d6dc16c547f9b3e5c81630a/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/7481caea6d6c2459b2ebdc23d95b9c0bc5c375ca96902b3e1bbd5f3e5bb9784e/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d87bf59d64e05f0a73c49d5530ebba37cb765587b198aa4536899cda6f15822f/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/42e8677faa5e39acfe8d8b76fc7eb9d6a623c3c36f04411b934e218532615799/merged
shm              64M     0   64M   0% /var/lib/docker/containers/1fc67798df856530ba2edaf059718108d62329901c38dd79667b4c714d39e901/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/caca8443267cf3aa224c712ab7dfcd91e46bd74f3ca8b11ae1f39bbff64fb40a/merged
shm              64M     0   64M   0% /var/lib/docker/containers/4d61971f076d1c35ec0de197e74fb6256d15b098632cb171084ca923e1209b27/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/064b35af37273bcbf83d169248f0c2a2096415d9e82d1ca82bb700388d43ecdd/merged
shm              64M     0   64M   0% /var/lib/docker/containers/0aaec78f04dbb7120d30aff83cc6882251f2b94400fd2a075bc6cd49da35b378/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/77651e06d89510a5795281df816c59d54692c885df687ef6cb0d63a5a7e3408e/merged
shm              64M     0   64M   0% /var/lib/docker/containers/ca7e7a73a2317372dce47e3ac0ddd86f536b252082988f503e4cb5a5bd5bf00d/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/16d9035325c50b8155a40a8295621d1ffc2d146543f17a0611730873801a136f/merged
shm              64M     0   64M   0% /var/lib/docker/containers/23e09a5e487fb06c4499d5d7c2af521b37d9e7183f985326c552d59f646a9b7c/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/7467c4f471daf82b8d34586415cac61bba5f5034c4391c798ca1b6ffa4fb88a7/merged
shm              64M     0   64M   0% /var/lib/docker/containers/e0b82eccd5dcb03e23421ab2907ad0382e88e0f399ba293ee43bf2b06ddf4e63/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/48c0feeeb6f8e2806cf8b26bcb5f056d0bdc82b646022573969cc51700f2bb39/merged
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/43386a89392822c61b69ec231f6f41fcecb5f5eb11ac33cf5c6c52f27b1cd0ee/merged
shm              64M     0   64M   0% /var/lib/docker/containers/eb38fa263786ab06cbb9dfea188a75169b333b5b3d96d43351f749405b010d67/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/99db1c4b719c9b0f96e05c43628d1fb920afbca0e29167b243dbe2decfe5f527/merged
shm              64M     0   64M   0% /var/lib/docker/containers/aac31e1a50cab3790ca4f9846d9f0024f9dd948e469fd95645570a171e66b673/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/1a6e613193f453505c6bb1e757081c2c586ab4bb2ddba9b7426a8958a8a46d21/merged
shm              64M     0   64M   0% /var/lib/docker/containers/fdb03166b422af52ea5908b0f309cc2f21773867ac7d450fea989cab7a7e5902/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/6eac2922335a5934d033068fa6495351d19c5887eacf7a19e13aea38d705bcc4/merged
shm              64M     0   64M   0% /var/lib/docker/containers/1f1c09cf0fcecd4d393ee94e4e4cce3a08f458add8919d5843dad7f55937b463/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/c412d5bc54d6918573c88de801dcab61c3ad6e23c147a84daf9ddde16de9df8f/merged
shm              64M     0   64M   0% /var/lib/docker/containers/39bf58a904c4d9734cf0dcd1474123352776c03da7ff312ada279e31bf41856a/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/1ca903dea1e3f8edc3d36703c282ae313e8912c5bd9e93d5cca187aa99345403/merged
shm              64M     0   64M   0% /var/lib/docker/containers/23b97f771e242d24943d5125db2bc457da2e927c704ff2241c2538e8ba3bc350/shm
tmpfs           6.3G     0  6.3G   0% /run/user/1000
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/06ccf4a5dfacb85b2904140417c2d01bd2af04b6017a5219e634b1009a3e70d1/merged
shm              64M     0   64M   0% /var/lib/docker/containers/4d8346d5045b0b201a44cc8fe3697a59fb23b501e48fef397c5aca90b4451060/shm
overlay         187G   14G  173G   8% /var/lib/docker/overlay2/59c30324a927613d3a22b240c88d012635a8c84fbe606d12f9a8aecfd9c25b33/merged
shm              64M     0   64M   0% /var/lib/docker/containers/f0d2107fdd6fa27e2b489a3aa3ead31f5bacf6b81b549872ca842561d1f27267/shm

Comment 36 Janki 2018-06-14 14:56:06 UTC
Deleting older logs an expected behavior of journal. Even increasing journal space does not guarantee log preservation as not only ODL but other services too use the added space. We will be moving back to logging to file.

Comment 46 Janki 2018-07-10 09:10:01 UTC
Steps to verfiy:

1. Deploy overcloud the rpm that has a fix.
2. Check that karaf.log* files are present on all controller nodes at /var/log/containers/opendaylight

There could be single karaf.log or files from karaf.log, karaf.log.1 to karaf.log.7 depending on when ODL container was started.

Comment 47 Noam Manos 2018-07-10 12:42:52 UTC
Created attachment 1457814 [details]
ODL HA controllers karaf logs

Verifying this bug on OSP13 HA (3 controllers) with ODL, puddle 2018-07-03.3:
All karaf logs are updated. 

On controller-2, which was probably the active one for ODL services, there is more debug data on the recent karaf.log, and there are older logs too (log.N).

Comment 49 errata-xmlrpc 2018-07-19 14:27:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2214


Note You need to log in before you can comment on or make changes to this bug.