Bug 1609333

Summary: [Logs] karaf.log is missing after log rotation
Product: Red Hat OpenStack Reporter: Victor Pickard <vpickard>
Component: puppet-opendaylightAssignee: Victor Pickard <vpickard>
Status: CLOSED WONTFIX QA Contact: Noam Manos <nmanos>
Severity: high Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: aadam, jjoyce, jschluet, mariel, mkolesni, slinaber, tvignaud, vpickard
Target Milestone: z4Keywords: TestOnly, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: Logs
Fixed In Version: puppet-opendaylight-8.2.2-4.9126c8dgit.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-06 16:16:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Victor Pickard 2018-07-27 15:28:17 UTC
Description of problem:

karaf.log is missing after restarting odl container

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. deploy OSP13 in cluster mode (3 controllers)

2. sudo docker restart opendaylight_api

3. ls /var/log/containers/opendaylight



Actual results:

[heat-admin@overcloud-controller-0 opendaylight]$ ls
karaf.log.1  karaf.log.3  karaf.log.4  karaf.log.5  karaf.log.6  karaf.log.7  karaf.log.8

logs are going to karaf.log.1

Expected results:

There should be a karaf.log file present, with logs going to karaf.log

Additional info:

This was on controller-0 of the deployment.

I looked on controller-1 and controller-2, where odl had also been restarted, and saw a slightly different version of the issue. On these nodes, karaf.log.1 is missing.

[heat-admin@overcloud-controller-1 opendaylight]$ ls
karaf.log  karaf.log.2  karaf.log.3  karaf.log.4  karaf.log.5  karaf.log.6  karaf.log.7
[heat-admin@overcloud-controller-1 opendaylight]$ 

[heat-admin@overcloud-controller-2 opendaylight]$ ls
karaf.log  karaf.log.2  karaf.log.3  karaf.log.4  karaf.log.5  karaf.log.6  karaf.log.7
[heat-admin@overcloud-controller-2 opendaylight]$ 

I suspect this is related to the issue reported, so I will not open a separate bz for these.

Comment 1 Janki 2018-08-08 05:15:59 UTC
Vicotr, the latest puddle creates just 2 files - karaf.log and karaf.log.1. Can you please confirm if this is reproducible with the latest puddle.

Comment 2 Victor Pickard 2018-08-24 13:20:25 UTC
Hi Janki,
I just took a look at cloud20 (already deployed, just looking at log issue).

Controller-1
=============

ODL Version:

()[odl@overcloud-controller-1 /]$ rpm -qa|grep openday
opendaylight-8.3.0-3.el7ost.noarch


()[odl@overcloud-controller-1 /var/log/opendaylight]$ ls karaf*
karaf.log.1  karaf.log.2  karaf.log.3


Logs are going to karaf.log.1

There is no karaf.log file.


Controller-0
=============
()[odl@overcloud-controller-0 /var/log/opendaylight]$ ls
karaf.log.1  karaf.log.3

()[odl@overcloud-controller-0 /var/log/opendaylight]$ rpm -qa|grep opendaylight
opendaylight-8.3.0-3.el7ost.noarch
puppet-opendaylight-8.1.2-2.38977efgit.el7ost.noarch


Controller-2
=============
()[odl@overcloud-controller-2 /var/log/opendaylight]$ ls karaf* 
karaf.log.1  karaf.log.2

()[odl@overcloud-controller-2 /var/log/opendaylight]$ rpm -qa|grep opendaylight
opendaylight-8.3.0-3.el7ost.noarch

Comment 3 Victor Pickard 2018-08-24 19:05:36 UTC
Debugged this a bit today, found the issue.

We are missing some config (fileIndex=min, defaults to max), will push a patch u/s shortly.

Reference:
http://logging.apache.org/log4j/2.x/manual/appenders.html#RollingFileAppender

fileIndex
==========

If set to "max" (the default), files with a higher index will be newer than files with a smaller index. If set to "min", file renaming and the counter will follow the Fixed Window strategy described above.


Look at the "Rollover Strategies" section for more details.

Comment 4 Victor Pickard 2018-08-24 19:35:13 UTC
https://git.opendaylight.org/gerrit/#/c/75480/

Comment 5 Victor Pickard 2018-08-29 21:44:19 UTC
Even with the above fix, there is still another issue with rollover. 

It appears to be a time based compression running to gz the files which is causing the issue of karaf.log to go missing. 

Notice the progression below, 1 sec intervals, where we see karaf.log disappear.

Still debugging on this one...

total 332M
-rw-r--r--. 1 odl odl 101M Aug 29 20:48 karaf.log.3
-rw-r--r--. 1 odl odl 101M Aug 29 21:18 karaf.log.2
-rw-r--r--. 1 odl odl 101M Aug 29 21:47 karaf.log.1
-rw-r--r--. 1 odl odl  32M Aug 29 22:00 karaf.log

total 332M
-rw-r--r--. 1 odl odl 101M Aug 29 20:48 karaf.log.3
-rw-r--r--. 1 odl odl 101M Aug 29 21:18 karaf.log.2
-rw-r--r--. 1 odl odl 101M Aug 29 21:47 karaf.log.1
-rw-r--r--. 1 odl odl  32M Aug 29 22:00 karaf.log

total 333M
-rw-r--r--. 1 odl odl 101M Aug 29 20:48 karaf.log.3
-rw-r--r--. 1 odl odl 101M Aug 29 21:18 karaf.log.2
-rw-r--r--. 1 odl odl 101M Aug 29 21:47 karaf.log.1
-rw-r--r--. 1 odl odl  32M Aug 29 22:00 karaf.log
-rw-r--r--. 1 odl odl 128K Aug 29 22:00 karaf.log.1.gz

total 336M
-rw-r--r--. 1 odl odl 101M Aug 29 20:48 karaf.log.3
-rw-r--r--. 1 odl odl 101M Aug 29 21:18 karaf.log.2
-rw-r--r--. 1 odl odl 101M Aug 29 21:47 karaf.log.1
-rw-r--r--. 1 odl odl  32M Aug 29 22:00 karaf.log
-rw-r--r--. 1 odl odl 2.2M Aug 29 22:00 karaf.log.1.gz

total 235M
-rw-r--r--. 1 odl odl 101M Aug 29 20:48 karaf.log.3
-rw-r--r--. 1 odl odl 101M Aug 29 21:18 karaf.log.2
-rw-r--r--. 1 odl odl 3.1M Aug 29 21:47 karaf.log.2.gz
-rw-r--r--. 1 odl odl  32M Aug 29 22:00 karaf.log.1

Comment 14 Franck Baudin 2019-03-06 16:16:38 UTC
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality

Comment 15 Franck Baudin 2019-03-06 16:17:47 UTC
As per depreciation notice [1], closing this bug. Please reopen if relevant for RHOSP13, as this is the only version shipping ODL.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/release_notes/index#deprecated_functionality