Description of problem: When running the following command: # docker logs opendaylight_api The output is as follows + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /opt/opendaylight/data/idmlight.db.mv.db INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/data/idmlight.db.mv.db to /opt/opendaylight/data/idmlight.db.mv.db INFO:__main__:Deleting /opt/opendaylight/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml to /opt/opendaylight/etc/opendaylight/datastore/initial/config/default-openflow-connection-config.xml INFO:__main__:Deleting /opt/opendaylight/etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml to /opt/opendaylight/etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml INFO:__main__:Deleting /opt/opendaylight/etc/jetty.xml INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/etc/jetty.xml to /opt/opendaylight/etc/jetty.xml Running command: '/opt/opendaylight/bin/karaf server' karaf: JAVA_HOME not set; results may vary Apache Karaf starting up. Press Enter to open the shell now... 100% [========================================================================] Karaf started in 24s. Bundle stats: 475 active, 476 total Version-Release number of selected component (if applicable): puppet-opendaylight-7.0.0-0.20180216174116.d3b32a4.el7ost.noarch How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Logs are present when using docker logs puppet-opendaylight-8.1.0-0.20180321182556.45c4db7.el7ost.noarch
Created attachment 1418817 [details] Container's karaf.log
Created attachment 1418818 [details] Docker log command output
Janki, I attached the karaf.log from the container and the docker logs command output. They are different. Please advise.
Looks like the docker logs is only picking up the 2nd half of the logs. It looks like with docker inspect the logging driver being used is journald: "LogConfig": { "Type": "journald", "Config": {} }, More info here: https://docs.docker.com/config/containers/logging/configure/ It looks like for some logging drivers there is config you can add for rollover or max log length. I'm not sure exactly if this is the problem or not, or if these options are even configurable for journald driver: https://docs.docker.com/config/containers/logging/journald/ One cool thing I found out is you can do this and actually see the logs in journalctl (however they do not show the full logs, same as docker logs output) journalctl CONTAINER_NAME=opendaylight_api If I restart the container I see kolla logging messages as expected: Apr 11 21:05:31 controller-0 dockerd-current[32771]: + sudo -E kolla_set_configs Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Validating config file Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Kolla config strategy set to: COPY_ALWAYS Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Copying service configuration files Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Deleting /opt/opendaylight/bin/karaf Apr 11 21:05:31 controller-0 dockerd-current[32771]: INFO:__main__:Copying /var/lib/kolla/config_files/src/opt/opendaylight/bin/karaf to /opt/opendaylight/bin/karaf I think ODL and the container are working fine. There was something that prevented journald from collecting the logs. Not sure what it is yet. We could switch the driver to use the default json driver: https://docs.docker.com/config/containers/logging/json-file/#usage But I'm guessing journald is better and is there is a reason we are already using that driver.
Did you ever have a chance to test to see whether these missing logs only happen during a deployment and the initial bring up? Are you able to detect missing logs from a later point in time? How reproducible is the issue?
I tested this locally before pushing the patch. It was working then. All the logs were seen by "docker logs" even after restart. I also kept ODL running for an hour and the logs were still seen. The problem here, I think, is puppet is not configuring log config file properly. There should be line "karaf.log.console=INFO" in '/opt/opendaylight/etc/org.ops4j.pax.logging.cfg' after 'log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter' which is missing. The karaf.log file on container should only contain initial 3 4 lines stating that karaf is started and nothing else.
From reading all of the above, I'm not crystal clear what exact problem this issue reports. IMHO it would be helpful if we wrote down the "Expected results:" VS the "Actual results:" and agreed on that, first. If the Expected results are that there is nothing in the karaf.log, then the logging configuration is wrong, and (it seems) is still sending it to both the karaf.log file as well as STDOUT (given that you ARE seeing things in "docker log"). If the Expected results are that there should be MORE, everything, in "docker logs", and the complaint here specifically is that it is missing things from the start, then that is most certainly not an ODL configuration issue. It would just seem that there is some "rolling over" happening - on the docker/journald side of things. Could someone attach the /opt/opendaylight/etc/org.ops4j.pax.logging.cfg from inside the container here so that we can easily have a look at it? It's probably better not to wonder at TripleO / Puppet sources which patch this file but to see the final result.
Created attachment 1423904 [details] org.ops4j.pax.logging.cfg
just FTR: The original issue which introduced this was Bug 1489704. > There should be line "karaf.log.console=INFO" in > '/opt/opendaylight/etc/org.ops4j.pax.logging.cfg' The org.ops4j.pax.logging.cfg which Itzik attached does have this, and that is what is (correctly) causing Karaf to also send its log to the "console" (=STDOUT), where a Container engine (Docker) captures it from and sends it to journald (based on what was said above). > after I am seeing "karaf.log.console=INFO" before not after "log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF}" but that makes no difference (I tested it). > 'log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter' which is missing. No, the attached org.ops4j.pax.logging.cfg which Itzik attached does have this, All is Wehl. > The karaf.log file on container should only contain initial 3 4 lines stating that karaf is started and nothing else. No, that's not what you configured in the attached org.ops4j.pax.logging.cfg. The way it currently is, Karaf will send its log to both STDOUT as well as to the RollingRandomAccessFile karaf.log.%i. (I've locally tested this; no need for Docker/container/Kolla/TripleO; just try in an upstream Karaf.) > the karaf.log from the container and the docker logs command output. > They are different. As Tim said in #c10 : "It looks like for some logging drivers there is config you can add for rollover or max log length." I strongly suspect that is the only reason for this. I'm not crystal clear what exact problem this issue reports.
Just to add here, the karaf logs on the container are getting rotated for every 17M of log data. Earlier they were rotated at every 10G of log data.
> the karaf logs on the container are getting rotated for every 17M of log data. > Earlier they were rotated at every 10G of log data. That's how it's configured: log4j2.appender.rolling.policies.size.size = 16MB Raise a separate new BZ if you consider this a regression which should be changed back?
I did some playing around locally. I tried adding variations of "log4j2.rootLogger.appenderRefs = Console/console/stdout/STDOUT/console.stdout". But nothing works. Commenting "log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile" does the trick. Logs are logged only to the console and NOT to file. Is this a right way to go?
> Commenting "log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile" does > the trick. Logs are logged only to the console and NOT to file. > Is this a right way to go? yes, that is the right way to prevent all (rootLogger) logs to be sent to the RollingFile appender. Strictly speaking though, that is not what this bug report is about. I would suggest that we conclude this bug report raised by QA with how "docker log" and/or 'journald' needs to be used to see older log entries, and perhaps just better document that somehwere? Then "case closed" on this one. I personally then do think that we also should change things to not log to the karaf.log file anymore as you are getting at here, but would suggest that is looked at in a separate new BZ.
> also should change things to not log to the karaf.log file anymore as you are > getting at here, but would suggest that is looked at in a separate new BZ. ==> new bug 1571365
I have a setup where ODL Is running for past 2 days. d46b72b370e8 192.168.24.1:8787/rhosp13/openstack-opendaylight:2018-04-24.1 "kolla_start" 2 days ago Up 2 days (unhealthy) opendaylight_api $ date Mon Apr 30 06:00:23 UTC 2018 $ sudo docker logs opendaylight_api shows logs right from start Running command: '/opt/opendaylight/bin/karaf server' karaf: JAVA_HOME not set; results may vary Apache Karaf starting up. Press Enter to open the shell now... ^M 8% [=====> ]^M 16% [===========> ]^M 25% [==================> ]^M 33% [=======================> ]^M 41% [=============================> ]^M 50% [====================================> ]^M 58% [=========================================> ]^M 66% [===============================================> ]^M 75% [======================================================> ]^M 83% [===========================================================> ]^M 91% [=================================================================> ]^M100% [========================================================================] Karaf started in 0s. Bundle stats: 12 active, 12 total 2018-04-27T13:15:23,541 | INFO | pool-1-thread-2 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | Adding features: standard/[4.1.3,4.1.3], wrap/[0,0.0.0], ssh/[4.1.3,4.1.3], odl-mdsal-trace/[1.7.0.redhat-5,1.7.0.redhat-5], odl-netvirt-openstack/[0.6.0.redhat-5,0.6.0.redhat-5], odl-jolokia/[1.10.0.redhat-5,1.10.0.redhat-5] last line at 2018-04-30T05:57:55,821 $ journalctl --since "2018-04-27" -- Logs begin at Fri 2018-04-27 13:01:20 UTC, end at Mon 2018-04-30 06:04:41 UTC. -- So 2 days old are still available in docker logs.
Hi Janki, What exactly is the bug currently?
Mike, Bug is journal is rolling over the logs every 2 days 3 hours approximately. We can configure this rollover in journal.conf file. We need to consult journal team (if any) and analyse impact of config change will have on other OpenStack services. Meanwhile, I can adding logic to mount karaf.log to overcloud controller node from ODL container. https://review.openstack.org/#/c/566270/
Janki, I see the patch you sent on master got merged, what's the status of this bug?
Cherry-pick to Queens is yet to be merged. CI is failing because of unrelated issue. https://review.openstack.org/#/c/567469/ Once it is merged, we can access karaf.log will be mounted on controller hosts too and we will be reading logs from /var/log/contaiers/opendaylight.
Hi Itzik Could you please provide with below command's output on your system where this behavior is observed. $ journalctl --since yesterday $ ls -1s /var/log/journal/*/ $ journalctl --disk-usage [odl@overcloud-controller-0 /opt/opendaylight/data/log]$ ls -sh (Note this gives size of karaf.log inside the container) $ df -h
First line of 'journalctl --since yesterday' -- Logs begin at Thu 2018-05-31 00:32:22 UTC, end at Thu 2018-05-31 06:36:22 UTC. - ==== [root@controller-0 heat-admin]# ls -1s /var/log/journal/*/ total 565304 65540 system 65540 system 65540 system 65540 system 8196 system.journal 8192 user-1000.journal 8192 user-42434.journal 57348 user-42445 57348 user-42445 57348 user-42445 57348 user-42445 8196 user-42445.journal 8196 user-42454 8196 user-42454 8196 user-42454 8196 user-42454 8192 user-42454.journal ====== [root@controller-0 heat-admin]# journalctl --disk-usage Archived and active journals take up 552.0M on disk. ===== ()[odl@controller-0 /opt/opendaylight/data/log]$ ls -sh total 1.9M 1.9M karaf.log ===== ()[odl@controller-0 /opt/opendaylight/data/log]$ df -h Filesystem Size Used Avail Use% Mounted on overlay 187G 14G 173G 8% / tmpfs 32G 0 32G 0% /dev tmpfs 32G 0 32G 0% /sys/fs/cgroup /dev/sda2 187G 14G 173G 8% /etc/hosts devtmpfs 32G 0 32G 0% /dev/log shm 64M 0 64M 0% /dev/shm tmpfs 32G 0 32G 0% /proc/scsi =====
Itzik, df -h from the controller node and not from the ODL container. and also $ date from controller node please.
[root@controller-0 heat-admin]# date Thu May 31 12:31:58 UTC 2018 [root@controller-0 heat-admin]# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 187G 14G 173G 8% / devtmpfs 32G 0 32G 0% /dev tmpfs 32G 54M 32G 1% /dev/shm tmpfs 32G 12M 32G 1% /run tmpfs 32G 0 32G 0% /sys/fs/cgroup tmpfs 6.3G 0 6.3G 0% /run/user/975 overlay 187G 14G 173G 8% /var/lib/docker/overlay2/5ade1ea00de2f445b1362fcdef1e14381dfd45fdfd37b6c7707b6b36043ce69f/merged shm 64M 0 64M 0% /var/lib/docker/containers/76fb1260f887e53eb6e77466c2bed652f2f23b4475417b30e0fb450bb60db013/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/e6d2533d8c61fdc2502fc6757499aa2edd9f9b2911187fa7b4faf07bc2d71971/merged shm 64M 0 64M 0% /var/lib/docker/containers/b60ba67401e8f4382eb53ed3c335ec6d950e6eff0c98d914484212b226f35570/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/893aa8d3ca100930c07365345194f62f5aab2e76f3a2c0583a6a2267d05e2e55/merged shm 64M 0 64M 0% /var/lib/docker/containers/ee51f7cc33537535b96391db8a87313d5c729dc7b4deb74b2c4046837f017917/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/a369b2d9558efb45ffc461bc4f58fcf68cc9ee81559fb788e289db0a51c5cc4e/merged shm 64M 0 64M 0% /var/lib/docker/containers/914fa26ca70a87289b26c5d13f80d31579571f3d52d2e954557bf56047f38b28/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/e894c407c91830895a8456a669ff1bc13127e0d5eb34918e27d028698d5e436d/merged shm 64M 0 64M 0% /var/lib/docker/containers/76bd15627d4e51531547db37286d41694bf43ea4d223864f71f6f32a9508432a/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f52dc362399a55ac3976e1a92e217ce9e13f1863e7b5ca00926054e8ca349277/merged shm 64M 0 64M 0% /var/lib/docker/containers/d10a6c108e0121786d7c32fef3e54b548edc3371c2ae6e54ef947fde3dcd9f73/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f8bd37c6782375fdc603aaac6f9d1c492ff2e4898b09813940dcd9faecb1c19d/merged shm 64M 0 64M 0% /var/lib/docker/containers/2a98efbd9c0fdf62c690e127e4ff1a24828b57c36361407908dcc69406329524/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/9afe780361a52fcb0d1723fd86834176a7feaaadde74df1eadb71dfc7d029696/merged shm 64M 0 64M 0% /var/lib/docker/containers/c8f41604c3207327658c64cae255ee446975711167c4e80e570913d918b1de28/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/68e7d5c6da9d08799eb4c5af344e19b7a2c61984f3256bb2e80f97e5bf42a047/merged shm 64M 0 64M 0% /var/lib/docker/containers/751a03b46ad3aa57a3c1005604a8da102ae9d60c4bb3c867df02c1c7a8fdb242/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/1cb6a3b4469466430399057890a34943919b2b321dc741da856bbf1d43807700/merged shm 64M 0 64M 0% /var/lib/docker/containers/301a32527ef21766e7f649578faaf63c6bb126bec0764226fe60a8bde57b6b2d/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/3da1e9cdaf73f4a1de2c36ebd082c3d611381b551fdd328d2527c9bfaa9ed36c/merged shm 64M 0 64M 0% /var/lib/docker/containers/f322a9c439efb909977abf2df218954fda0dbbe2a9b65269bd7beae7c22b2717/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/50236ffb0229c7672148d942976f0054c31f5111ac2681309842e5fa3d9bf5fd/merged shm 64M 0 64M 0% /var/lib/docker/containers/8d85eb5ebd20e167fccdce5148c43a2f42e428d9c769210b59a4f507815bbd40/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/3635697f263002c2d8203466ed8225c9a72fbbdd6d84773f4a31d821e5a3af73/merged shm 64M 0 64M 0% /var/lib/docker/containers/7295970fef0d6d1c6532c18cf0fbba08555fa0c7286ffe5f5bd7e8b584b0342f/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/8e7835c343c883e586006e16f4e13cb4b4834de4695029a2061446e6cca4444d/merged shm 64M 0 64M 0% /var/lib/docker/containers/f39b0a88befc2a479582cf4585927221dfb439afa573972ab7f3e2f1046fb417/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f804ab6ba6546c4e6de6ca68ba0de47ee37dbfd669044c8abddccfeeebc22dfb/merged shm 64M 0 64M 0% /var/lib/docker/containers/ae304a67997dd87cd790c4d22177d44de5d355cd8dbef07a7ccba57d92c32e05/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f335e1c82ea8e84c213767b7bc7dc21574222562c0f562c34c16bdced6a52517/merged shm 64M 4.0K 64M 1% /var/lib/docker/containers/e13663002771e35b4dbaaf9b214aab43af5decf3eb3d704abc54c14351800438/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/9ba7dcd3d43c8aaf4cf644f697f0d44dc4746b0af6c735ed7fdde85ee0b70576/merged shm 64M 0 64M 0% /var/lib/docker/containers/c1451c1ef4688b3afc4fc6017ad0c502aa77a2f57e8ac551168f77662e124729/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/07120937d02d2e1a4dfc34622d41990c2c4fa83972f1ba239ea14b5d5a41cb6d/merged shm 64M 0 64M 0% /var/lib/docker/containers/a2d57e0bdd1283787cd96bac13d87342e506a9ac04c0946c19729713ab462345/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/cf1cd7fbc54c89e80d9607befec9b9885a879d04c8a14fc288a296b2499eca79/merged shm 64M 0 64M 0% /var/lib/docker/containers/2c7417429a27ef7f42cd1a72d124645373e6952fef0fa03a9020ab93bae2fd43/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/0301ed923a35c66de473317b79b40260bfdbc6e7b4f27bbb91ec8ffa1fa23321/merged shm 64M 0 64M 0% /var/lib/docker/containers/9f755e0f62ee79150004ca53d5c7101c4a5c4ca9359959199a6fc8374270722d/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/7735837127a2eaf2164922efc4ac6e86f4ca0837d93b43afb9e4b4e266502446/merged shm 64M 0 64M 0% /var/lib/docker/containers/6cad4e22cfa8d1615ecdb055643d141bf78fc0ceccd05a565ca359b0a3918c13/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/39b5b4d5a3b496a2e12a401839582f55b0f43725c36a3c0884376e2b7083f442/merged shm 64M 0 64M 0% /var/lib/docker/containers/6625384f27c1be2e14b633987670090b4d4c2bff1b99a44da006c135a2f5cd23/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f4ca02366ac8b33485bdb355cdc0559a07ab683bf0f1df64cda6ccf633368c3c/merged shm 64M 0 64M 0% /var/lib/docker/containers/47a777affb087d32627ab7ff81e240369de4c0726ce0384dd4a0f72d18697a35/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/4f0cfade340518bd0c97850b53834c2c459ab47132c57b44339c6654c553ea97/merged shm 64M 0 64M 0% /var/lib/docker/containers/c1254d0add4a0340f6ce33e87d72984cde91fc67f7c0b446e90c9092ff99ba03/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/4e18a92b9217104af10ca8b0ffc7a449d251bafcd9363375de0cb0a376092e5d/merged shm 64M 0 64M 0% /var/lib/docker/containers/b6a9e0ce9bf1ad6b878e8e1f64a7ab1f8c14f322aebe137b0037d320a09ca532/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/cc2164ad6540e7e4a994fb576da53ee37113b6736de43f8c0fea7f67903b80e2/merged shm 64M 0 64M 0% /var/lib/docker/containers/41d9e399ea527536bdf78738dc8bb60511b1fd4af238c6196cb34727d9d3654c/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/be9afa0a510e71b8514dfafdf88cba53c9668cad773ffb019c45bcfdd8ac0133/merged shm 64M 0 64M 0% /var/lib/docker/containers/2ac88d7d5485916451d435de5e45cf64d93f6453e693d058764ddcf6ebc3d80d/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/ab720c32b4215c1e03bbf6da5dc72851f1a19059c56d4e17c2f64f65d737af38/merged shm 64M 0 64M 0% /var/lib/docker/containers/d663799f531c063f028fb5074a0fc7384aa280d40abdaf179906806605ad1a0b/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f7081435e6bdae23da162f369e294fc0bf679acf4dc33fbaa4b790b522172e1e/merged shm 64M 0 64M 0% /var/lib/docker/containers/5caa78613855712f8a6a361b06097f24882c3554435a0e05bd7fc0ce24c3ff76/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/7ff6119918bec99cf3e61edb62856f2568dc9c9b64280e12f4ddcd60ee9aad52/merged shm 64M 0 64M 0% /var/lib/docker/containers/78e7d6d08bb2ac5758fdeeb719a78055f8ddd6d26e64663cdea9c25091af2e35/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/79a0b292c8442d7e647c67f91b6eb674e718930f89a0ce0fd30c16f8fea755b6/merged shm 64M 0 64M 0% /var/lib/docker/containers/6c2b38b3e799ad0a5d45113755e70304f40972b35b9e2418bef1a12485253b5b/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/c21ad6e84f857303aeffbcabda9f2eaa56454ce183a2c259a29730b454509771/merged shm 64M 0 64M 0% /var/lib/docker/containers/b10f05f02e58b02f9b64f40846989362a1d1d47b1fb2b16544652b96e60b2677/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/6b98fa4ac38740397109f5e140f24584d4058fac76a2583f272594fbf6fca768/merged shm 64M 0 64M 0% /var/lib/docker/containers/370b205029a7f716fb4e5e74f3b1193a388ea6502bc35935ab9812090e903dee/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/0e6cf250c99e9b8b7f26df1986c48a40555c694945cb95d647bccb285a0de454/merged shm 64M 0 64M 0% /var/lib/docker/containers/b456ebfc45c1603a124f4773c15b0356d769cd3c0497aa1a0242b8e48d053402/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/15537cb4a545ee82afb08cd3a18139cf0ece9ea03369097e18871375c1cbd540/merged shm 64M 0 64M 0% /var/lib/docker/containers/127589dc2f66213804360e43a3306c680017e7ad9b17627cd7734b630e293769/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/6fd6f6363d607500ffccc36adbc8d073451d3a509c71b7fe65bceb14e1d19fa1/merged shm 64M 0 64M 0% /var/lib/docker/containers/a3739d776f25157a4f698c5f2ed22916881b55e050fafc22b530c1b647e83f69/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/7a06b61bc7e8316ec320687d732e26ceb085f0953dea9f0c93c70ef59816fcd9/merged shm 64M 0 64M 0% /var/lib/docker/containers/8fc0fc4f26ecc23ad09c8e9e31ce58512301e5c1b5ca533396a78f50375ccd79/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/79c1bcdc552a15d3dba16db682041392c5797608091c80d37b740e1ae640e5ca/merged shm 64M 0 64M 0% /var/lib/docker/containers/21c0c0f6f831c86d35e446d04474af5c7fab4a50c3b47d76258e287d71ec307f/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/1527c6785fdbdb81348a561d3fbef9bdc5c063c796f7f10ebe8020bdab3825da/merged shm 64M 0 64M 0% /var/lib/docker/containers/96246915c9a306576cec673dbcce7c58a38a687dd810f42f1a19a735fab1ad15/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/7821b5477aa8392fb167dad528ba9b1142505759f3348286f3f1097ffc356946/merged shm 64M 0 64M 0% /var/lib/docker/containers/c5c2b3e18481844e72a52eef2f3b8419ceefc065ceab5d24a3d23bd781a34312/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/39ef381c9b43da830a80e12be65fdbaa541aca7308300dab6448aa1ff2127484/merged shm 64M 0 64M 0% /var/lib/docker/containers/2a26a3c8f8fec00d350ba304f0407c5359922968a69a050a114d4bcb69aca10c/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/8f649a5a1a25eba96e63f52d232d696f6f7bf330e687ca09611573fc9f579aea/merged shm 64M 0 64M 0% /var/lib/docker/containers/3cec928459a9ea5ebf2bd3d1eb62bb71175075ff98a15c5aa8b43187b4bb2662/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/f39a08f13e5f6efc992445086a9407c14d5d4c016e46682d61aac7550dffce21/merged shm 64M 0 64M 0% /var/lib/docker/containers/44db8ac70e2d3034fa37cd0bbee3e60cc26a2bb8dde5547eb822e2cb22df0824/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/72dbdadafb0986b175dfca674f130f7ba36a4b35cab528177d7d51395c2aaff7/merged shm 64M 0 64M 0% /var/lib/docker/containers/670633195b4e465c40a7c8307f5b9e2926bf869bb9f028567984f4f1e8896078/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/267d8896601773335827251599d07cfab599fb3f322f21b20d464be482a21ffe/merged shm 64M 0 64M 0% /var/lib/docker/containers/51fb98e82acd9f69112a2f919fbeb53fc0c952651928695cc2e6f158a7a5a21b/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/2762f604a80bd816f8a78b33b055a0e4d5f1293bc5712cbc061ded0713883d8e/merged shm 64M 0 64M 0% /var/lib/docker/containers/4ac0387049320f38ac655011a5c42f4eb03a8560f98dc572538ce6cc2679481b/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/aae2f822c32aac750513adc1766fbf2f8ab58464a8c031557b8de94a0ada6716/merged shm 64M 0 64M 0% /var/lib/docker/containers/e6db38a32563946672d7b6b5b651e737afa25d334d6dc16c547f9b3e5c81630a/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/7481caea6d6c2459b2ebdc23d95b9c0bc5c375ca96902b3e1bbd5f3e5bb9784e/merged shm 64M 0 64M 0% /var/lib/docker/containers/d87bf59d64e05f0a73c49d5530ebba37cb765587b198aa4536899cda6f15822f/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/42e8677faa5e39acfe8d8b76fc7eb9d6a623c3c36f04411b934e218532615799/merged shm 64M 0 64M 0% /var/lib/docker/containers/1fc67798df856530ba2edaf059718108d62329901c38dd79667b4c714d39e901/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/caca8443267cf3aa224c712ab7dfcd91e46bd74f3ca8b11ae1f39bbff64fb40a/merged shm 64M 0 64M 0% /var/lib/docker/containers/4d61971f076d1c35ec0de197e74fb6256d15b098632cb171084ca923e1209b27/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/064b35af37273bcbf83d169248f0c2a2096415d9e82d1ca82bb700388d43ecdd/merged shm 64M 0 64M 0% /var/lib/docker/containers/0aaec78f04dbb7120d30aff83cc6882251f2b94400fd2a075bc6cd49da35b378/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/77651e06d89510a5795281df816c59d54692c885df687ef6cb0d63a5a7e3408e/merged shm 64M 0 64M 0% /var/lib/docker/containers/ca7e7a73a2317372dce47e3ac0ddd86f536b252082988f503e4cb5a5bd5bf00d/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/16d9035325c50b8155a40a8295621d1ffc2d146543f17a0611730873801a136f/merged shm 64M 0 64M 0% /var/lib/docker/containers/23e09a5e487fb06c4499d5d7c2af521b37d9e7183f985326c552d59f646a9b7c/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/7467c4f471daf82b8d34586415cac61bba5f5034c4391c798ca1b6ffa4fb88a7/merged shm 64M 0 64M 0% /var/lib/docker/containers/e0b82eccd5dcb03e23421ab2907ad0382e88e0f399ba293ee43bf2b06ddf4e63/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/48c0feeeb6f8e2806cf8b26bcb5f056d0bdc82b646022573969cc51700f2bb39/merged overlay 187G 14G 173G 8% /var/lib/docker/overlay2/43386a89392822c61b69ec231f6f41fcecb5f5eb11ac33cf5c6c52f27b1cd0ee/merged shm 64M 0 64M 0% /var/lib/docker/containers/eb38fa263786ab06cbb9dfea188a75169b333b5b3d96d43351f749405b010d67/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/99db1c4b719c9b0f96e05c43628d1fb920afbca0e29167b243dbe2decfe5f527/merged shm 64M 0 64M 0% /var/lib/docker/containers/aac31e1a50cab3790ca4f9846d9f0024f9dd948e469fd95645570a171e66b673/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/1a6e613193f453505c6bb1e757081c2c586ab4bb2ddba9b7426a8958a8a46d21/merged shm 64M 0 64M 0% /var/lib/docker/containers/fdb03166b422af52ea5908b0f309cc2f21773867ac7d450fea989cab7a7e5902/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/6eac2922335a5934d033068fa6495351d19c5887eacf7a19e13aea38d705bcc4/merged shm 64M 0 64M 0% /var/lib/docker/containers/1f1c09cf0fcecd4d393ee94e4e4cce3a08f458add8919d5843dad7f55937b463/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/c412d5bc54d6918573c88de801dcab61c3ad6e23c147a84daf9ddde16de9df8f/merged shm 64M 0 64M 0% /var/lib/docker/containers/39bf58a904c4d9734cf0dcd1474123352776c03da7ff312ada279e31bf41856a/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/1ca903dea1e3f8edc3d36703c282ae313e8912c5bd9e93d5cca187aa99345403/merged shm 64M 0 64M 0% /var/lib/docker/containers/23b97f771e242d24943d5125db2bc457da2e927c704ff2241c2538e8ba3bc350/shm tmpfs 6.3G 0 6.3G 0% /run/user/1000 overlay 187G 14G 173G 8% /var/lib/docker/overlay2/06ccf4a5dfacb85b2904140417c2d01bd2af04b6017a5219e634b1009a3e70d1/merged shm 64M 0 64M 0% /var/lib/docker/containers/4d8346d5045b0b201a44cc8fe3697a59fb23b501e48fef397c5aca90b4451060/shm overlay 187G 14G 173G 8% /var/lib/docker/overlay2/59c30324a927613d3a22b240c88d012635a8c84fbe606d12f9a8aecfd9c25b33/merged shm 64M 0 64M 0% /var/lib/docker/containers/f0d2107fdd6fa27e2b489a3aa3ead31f5bacf6b81b549872ca842561d1f27267/shm
Deleting older logs an expected behavior of journal. Even increasing journal space does not guarantee log preservation as not only ODL but other services too use the added space. We will be moving back to logging to file.
Steps to verfiy: 1. Deploy overcloud the rpm that has a fix. 2. Check that karaf.log* files are present on all controller nodes at /var/log/containers/opendaylight There could be single karaf.log or files from karaf.log, karaf.log.1 to karaf.log.7 depending on when ODL container was started.
Created attachment 1457814 [details] ODL HA controllers karaf logs Verifying this bug on OSP13 HA (3 controllers) with ODL, puddle 2018-07-03.3: All karaf logs are updated. On controller-2, which was probably the active one for ODL services, there is more debug data on the recent karaf.log, and there are older logs too (log.N).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2214