Bug 725241

Summary: [vdsm][logs]vdsm keeps rotating the logs even though no new logs are added
Product: Red Hat Enterprise Linux 6 Reporter: Moran Goldboim <mgoldboi>
Component: vdsmAssignee: Yotam Oron <yoron>
Status: CLOSED WORKSFORME QA Contact: Moran Goldboim <mgoldboi>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 6.3CC: abaron, bazulay, danken, iheim, ykaul
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-09-04 12:58:50 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Moran Goldboim 2011-07-24 15:06:31 UTC
Description of problem:
vdsm keeps rotating the logs even though no new logs are added- vdsm become non-operational and thus not writing too much logs, after a day i have noticed that logs were keep rotating each hour though no new logs were added.

ll /var/log/vdsm/vdsm.log* -ltr
.
.
.
-rw-r--r--. 1 vdsm kvm   509520 Jul 23 12:01 /var/log/vdsm/vdsm.log.23.xz
-rw-r--r--. 1 vdsm kvm   483200 Jul 23 13:03 /var/log/vdsm/vdsm.log.22.xz
-rw-r--r--. 1 vdsm kvm 10690560 Jul 24 18:01 /var/log/vdsm/vdsm.log.1
-rw-r--r--. 1 vdsm kvm 20553885 Jul 24 18:01 /var/log/vdsm/vdsm.log


Version-Release number of selected component (if applicable):
vdsm-4.9-81.el6.x86_64

How reproducible:
happened several times

Steps to Reproduce:
1.make vdsm non-oprational and let it run
2.
3.
  
Actual results:
after 100 hours all logs will be rotated.

Expected results:


Additional info:

Comment 2 Yotam Oron 2011-07-25 12:41:50 UTC
http://gerrit.usersys.redhat.com/#change,743

Logs are rotated on an hourly basis, regardless of the log size (they are also rotated when the file exceeds a certain size).
I added minsize to the logrotate conf file, so log files under a certain size (15M) are not rotated.

Comment 3 Yotam Oron 2011-08-09 11:53:53 UTC
Moran, 

What do you mean by 'vdsm become non-operational' ? 
Is it still running ? Are you SURE that no logs are written, since in the ls output you attached, it seems like the logs are not zero-sized.

Comment 4 Moran Goldboim 2011-08-09 12:13:37 UTC
 (In reply to comment #3)
> Moran, 
> 
> What do you mean by 'vdsm become non-operational' ? 
> Is it still running ? Are you SURE that no logs are written, since in the ls
> output you attached, it seems like the logs are not zero-sized.
from what i saw in this example there are 2 problems:
-vdsm log is rotating every hour - which means we are limited by time - 100h
i would prefer a rotate based on min size (can be preformed each hour)
-second problem is that when vdsm was non-operational (on rhevm side)- logs were keeping rotating while vdsm.log file wasn't - need to check and see what is the status regarding it.

Comment 5 Yotam Oron 2011-08-09 12:18:10 UTC
It seems like the minsize solution is the right one for the first issue:
https://bugzilla.redhat.com/show_bug.cgi?id=173088

Comment 6 Dan Kenigsberg 2011-08-09 19:53:30 UTC
This is not my understanding of logrotate(1) man page. But seeing is believing: have you reproduced this from the command line? When I run

logrotate /etc/logrotate.d/vdsm

short logs are NOT rotated.

Comment 7 Yotam Oron 2011-09-01 09:52:24 UTC
I checked again, and Dan is right - logs are not rotated when smaller then 15M.
The minsize indeed has no effect in this case, since logrotate finest time granularity is a day, hence the minsize has not effect here.
I can add a notifempty keyword to make sure that empty log file aren't rotated,
but need to see a reproduction of this bug.

Comment 8 Moran Goldboim 2011-09-01 09:59:36 UTC
(In reply to comment #7)
> I checked again, and Dan is right - logs are not rotated when smaller then 15M.
> The minsize indeed has no effect in this case, since logrotate finest time
> granularity is a day, hence the minsize has not effect here.
> I can add a notifempty keyword to make sure that empty log file aren't rotated,
> but need to see a reproduction of this bug.

Yotam, have you tried to reproduce? - it's suppose to be very easy (just let vdsm run - not connected to rhevm). if it doesn't reproduced to you please let me know.

Comment 9 Yotam Oron 2011-09-01 10:59:04 UTC
Moran,

I had a vdsm running on a server for more than a day with no log rotation problems.

Comment 10 Yotam Oron 2011-09-04 12:58:50 UTC
I had a server running for a few days without logs rotating.
I spoke with Moran and he indicated he had also a server running for a few days without reproduction.
Closing as WORKSFORME.