Bug 108775 - RFE: Logrotate "dateext" patches from Suse
Summary: RFE: Logrotate "dateext" patches from Suse
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: logrotate
Version: rawhide
Hardware: All
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Peter Vrabec
QA Contact:
URL: ftp.suse.com/pub/people/ro/logrotate
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2003-11-01 20:38 UTC by Keith Lofstrom
Modified: 2007-11-30 22:10 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Clone Of:
Environment:
Last Closed: 2005-06-23 15:36:49 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
SUSE's "dateext" logrotate patch (8.29 KB, patch)
2004-12-10 20:26 UTC, David Anderson
no flags Details | Diff
"maxage" extension (3.08 KB, patch)
2004-12-10 20:28 UTC, David Anderson
no flags Details | Diff
patch with the leftover part of the moveout-patch (1.19 KB, patch)
2005-05-03 14:01 UTC, Ruediger Oertel
no flags Details | Diff

Description Keith Lofstrom 2003-11-01 20:38:34 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624

Description of problem:
Request for Enhancement:  Merge Suse logrotate "dateext" patches.

On Redhat systems, logrotate renames files by numbering and
renumbering - messages becomes messages.1, .1 becomes .2, etc.  
This makes life difficult for rsync backups, which saves each 
copy of each renumbered file.

Ruediger Oertel at Suse (ro) has some patches that enhance 
the basic Redhat Logrotate with "dateext".  This allows a dated
extension rather than a numbered one, i.e. messages.20031029 .  
The old logfiles do not get renamed, just discarded after they get 
too old.  This is a lot easier on rsync, and it also is easier to
administer.

Ruediger's patches do not change the current logrotate behavior, just
adds the new features.  His patch to logrotate 3.6.10 can be found at:

  ftp.suse.com/pub/people/ro/logrotate

Suse has been making these patches to RH logrotate for years, and 
I've been running versions of their code for a few weeks while 
still using the old logrotate method for testing.  It would be 
nice to fold this into the main line of fedora/redhat and 
eliminate a needless code fork.

Version-Release number of selected component (if applicable):
3.6.10 and others

Comment 1 Elliot Lee 2003-11-03 20:41:40 UTC
My understanding is that the patch does not play well with some other
logrotate features such as the rotate keyword. Is this correct?

Comment 2 Keith Lofstrom 2003-11-03 21:26:40 UTC
The rotate keyword still appears to function normally.  I am 
running the installed /etc/logrotate.conf file.  It contains 
the default  "rotate 4" command, and indeed the normal logfiles
(boot.log cron mail.log messages mysqld netconf.log rpmpkgs 
secure sendfax spooler up2date xferlog ) are all getting 
rotated in that manner, that is, I have files  [cron, cron.1, 
cron.2, cron.3, cron.4]. pacct is set for "rotate 32" and I 
have all 33 files, no more or less.  If I understand the 
meaning of the "rotate" statement,  the patched version works 
in the expected way for the old-style number scheme.

Of course, logrotate targets configured for "dateext" will 
need to expire files by date and not by number.  That is 
sensible, and these two ways of doing things should probably 
remain somewhat orthogonal to prevent surprising interactions.

However, I have not tested any corner cases or non-default cases,
so I cannot prove there isn't a gotcha in there somewhere.  If 
you have any tests you would like me to try, please let me
(and ro ) know.  I've found Ruediger to be a very helpful
and cooperative fellow, and while I am not so clueful, I would be
glad to run tests and help hunt bugs.

Thank you for your time;   Keith Lofstrom

Comment 3 Keith Lofstrom 2004-03-13 16:45:11 UTC
I have not seen any action on this since November.  I answered the
question from Elliot Lee - this enhancement does NOT interfere with
base functionality.  How do I encourage some action on this RFE?

Comment 4 James Kosin 2004-04-15 18:54:24 UTC
Keith,

I think the point may be WHY?.....

It seems to be easier for me to type 'cat crom.1' than it is to find 
the date last week when the logs were rotated and type 'cat 
crom.20031230'.....

That seems like a lot of typing.  It may help with some people who 
want to keep large numbers of log files (52 for a years worth).  And 
archive them to CD or something.  But, practically, I think you will 
have problems getting people use to typeing that much just to get 
last weeks logs.

Good Luck though,    James

Comment 5 James Kosin 2004-04-15 19:39:19 UTC
Keith,

I think the point may be WHY?.....

It seems to be easier for me to type 'cat crom.1' than it is to find 
the date last week when the logs were rotated and type 'cat 
crom.20031230'.....

That seems like a lot of typing.  It may help with some people who 
want to keep large numbers of log files (52 for a years worth).  And 
archive them to CD or something.  But, practically, I think you will 
have problems getting people use to typeing that much just to get 
last weeks logs.

Good Luck though,    James

Comment 6 Matthew Saltzman 2004-04-15 20:11:48 UTC
Why?  I have a large set of logs that I keep for a long time.  Having
them tagged by date accomplishes three things:

(1) I can see the dates when I do an 'ls' without having to 'ls -l'.

(2) They come sorted in order of increasing date by default.  More 
than ten generations of logs would normally be sorted as log, log.1,
log.10, log.11, ..., log.2, etc.  For the times when you want to 
handle logs in a chronological block (e.g., grep for an event and see 
all occurrences in all logs in chronological order), this would be a
boon. 

(3) Rotating doesn't require renaming all previous generations.  Only
the current log needs to be renamed.  If you have many generations of
logs for many sets of apps, then that can get to be a serious burden
on logrotate.  We were keeping 10 generations of logs for 30-40 Web
sites, and logrotate would run half a day or more trying to keep up.

So, why not give those of us who *do* keep a year's worth of multiple
sets of weekly logs the option to manage them more effectively?  It
doesn't have any effect at all on users that don't care...

Thanks.

Comment 7 Keith Lofstrom 2004-04-15 20:15:05 UTC
This is an *option*.  Nobody is forced to use it, and the old
behavior will keep working fine.  However, if you run rsync for 
backups or mirroring images, then having the same file and the
same data change names from crom.1 to crom.2 means that it
needs to be moved and backed up again, for no reason.  This is
a waste of bandwidth and backup media.

Numerous people have requested this.  A few have groused that it
will cause their code to stop working, but they are neglecting 
that this option does not come into play unless you purposely 
configure it differently than the distro out of the box.

Most people don't need it, granted.  But if you keep forking off
small groups whose needs you ignore, eventually you've forked off
everyone.  SUSE thinks this is important enough that they put in
the effort to maintain a separate source tree, for no other reason 
than that Redhat ignores their request to fold this code back into
the main branch of the tree.  Perhaps that is distro politics as
usual, but it seems like a silly way to grow the Linux business for
everyone.

Keith

Comment 8 Forrest 2004-04-15 22:43:26 UTC
I agree with Keith.  This is simply an option that would enhance
logrotate for those who want it.  Elliott, do you have any information
about the problems that may come up with rotate and other key words?

Comment 9 Marcel Mol 2004-04-16 20:47:42 UTC
I also like to have this feature. Especially for Keiths backup
reasons, but also for properly sorted ls output (although ls -t will
do the same). I was planning to build a patch myself but this will
save me some time.
So please Elliot, consider honoring this request!
Thanks

-Marcel

Comment 10 why the hell do I have to register 2004-06-30 20:19:56 UTC
I would like this feature as well. I mean, it's obvious. I can't 
believe it was met with so much resistance. Maybe I'll switch to suse 
since their developers seem less obtuse. 

BTW, the mysql version shipped with fedora core 2 stable release in 
may, is SIX MONTHS OLD. And I don't mean six months compaired to 
the "testing realease".  I mean, mysql 4 stable was released six 
months before core 2 was released.  You're the center of the LAMP 
architecture and you're screwing up the "M".  

Hell, I'm switching to suse.


  

Comment 11 Elliot Lee 2004-06-30 21:32:30 UTC
Unfortunately, mysql 4 has licensing issues that prevent it from being
included...

For including this patch, well, it's not the highest priority.
Hopefully once Fedora CVS happens (and it will!) then you'll be able
to take over logrotate maintainence from me :)

Comment 12 John Stucki 2004-07-27 19:35:41 UTC
We've been dieing for this.  Please add it to fedora core.  If its
planned to be added, when?  Thanks.

Comment 13 Ruediger Oertel 2004-08-03 23:28:51 UTC
just drop me (ro) a mail for questions about the patch 
or if you need the patch to be ported to some more recent version 
(if there is one) 

Comment 14 John Stucki 2004-08-04 13:18:18 UTC
it appears the Fedora CVS is now open http://www.fedora.us/wiki/FedoraCVS

Thus, hopefully someone will add it soon.  

Just out of curiousity, how does the patch rotate out old logs?  Is it
by the log's mtime or by simply using the largest timestamp in the
file names since log.20040801, 20040802.log, 20040803.log, are
essentially the same as log.1, log.2, log.3 being incremental
integers, just much larger and possibly not sequential (ex.
20040731.log, 20040801.log)?

Comment 15 Ruediger Oertel 2004-08-04 13:52:46 UTC
the current implementation uses the glob result. 
for "found files = n" and "rotateCount = r" it will 
remove the first "n - r" files returned by glob. 
 
Removing by rotateCount is probably a bad idea anyway, 
I'm using an extension "maxage" to drop logfiles older 
than N days, for some cases of use this is even needed, 
some countries have laws not to keep user related log data 
longer than a certain amount of time (AFAIR)... 
 

Comment 16 Ruediger Oertel 2004-08-04 13:54:55 UTC
BTW: patches against version 3.7 can be found in the src-rpm 
at the same location as in comment#1 
 

Comment 17 John Stucki 2004-10-20 19:51:13 UTC
So since the Fedora CVS is up, does anyone want to incorporate the
patches into the main src?  

Comment 18 David Anderson 2004-12-10 16:42:18 UTC
I'm currently rsyncing log files from Australia to the UK, where one 
of our company's offices is. When the log files get rotated, the 
whole load get transferred again because of the current logrotate 
behaviour. 
I've implemented date-based filenames with my own cron job, but I too 
would find this very useful in the default logrotate. Any progress? 

Comment 19 David Anderson 2004-12-10 20:26:36 UTC
Created attachment 108349 [details]
SUSE's "dateext" logrotate patch

I've attached SUSE's "dateext" patch for logrotate 3.7 - extracted from
ftp://ftp.suse.com/pub/people/ro/logrotate/logrotate-3.7-31.2.src.rpm

SUSE have had the "dateext" capability since December 2001. I think that this
fact alone suggests that it just works! (I'm not a C coder, so can't tell you
much else!).

Comment 20 David Anderson 2004-12-10 20:28:27 UTC
Created attachment 108350 [details]
"maxage" extension

"maxage" extension, from the same source. Again, it's been in SUSE's logrotate
since December 2001; again I'd say that's a good enough testimony to its
quality. Pleeaasseeee Mr. Red Hat can we have this?

Comment 21 Gabriele Turchi 2005-02-16 11:29:17 UTC
Maintaining large systems this patch can be very useful. Please, I hate creating
my own rpms...

Comment 22 David Anderson 2005-02-16 11:41:56 UTC
The inactivity on this bug is quite difficult to understand.  
  
If the maintainer is too busy to review it, can he find another 
maintainer?  
  
If he is not too busy, then can a decision be made?  
  
We have here:  
i) A request that many people want  
ii) Tested and reliable patches available  
iii) A 15 month silence from the maintainer (who has been changed in 
the meantime, but the new maintainer is also silent)  
  
This isn't the only entry in bugzilla requesting this; others are 
older still.  
  
This all seems like a bit of a process failure; if there's a good 
reason for the patch not to be applied, what is it?  
The net combined effect of i) ii) and iii) above is just that we 
think "What's the point of reporting things to Red Hat? They just 
ignore us." There's a big de-motivation here for people who have 
posted on this bug to keep on bothering using Bugzilla and helping 
Red Hat improve their products.  

Comment 23 Mark Delahunty 2005-04-29 11:09:14 UTC
I would find this feature very useful to save on backup media and network usage 
and time. I would even patch all the Red Hat installations I admin (old Red 
Hat, RHEL and FC), but I'd then have to remember to do it after every server 
istall (and up2date of logrotate?).

Please, Red Hat, include it and save me some work.

BTW does anybody know how logwatch behaves when the syslogs have the dateext 
extensions?


Comment 24 Lenz Grimmer 2005-05-02 13:20:55 UTC
FWIW, there is a request to include these patches in Debian Linux as well:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=257685

(I too would like to see the inclusion of these patches in Red Hat Linux)

Comment 25 Peter Vrabec 2005-05-03 13:23:12 UTC
http://people.redhat.com/pvrabec/logrotate-3.7.1-11.src.rpm

includes 'dateext' and 'maxage'.

I appreciate you test it and let me know it works.

Comment 26 Ruediger Oertel 2005-05-03 14:01:55 UTC
Created attachment 113961 [details]
patch with the leftover part of the moveout-patch

I guess the rest of the moveout patch (as attached here) could be
applied as well. the moveout patch moves some common code into
the functions mailLogWrapper and removeLogFile and this leftover
patch now makes use of these functions.

Comment 27 David Shafer 2005-10-05 04:02:25 UTC
The University of Iowa would also like to see these patches included.

Comment 28 Peter Vrabec 2005-10-07 09:00:10 UTC
Can somebody tell me what this hunk from maxage patch is suppose to do?

@@ -644,6 +647,22 @@
 	globfree(&globResult);
 	free(glob_pattern);
     } else {
+	if ( log->rotateAge ) {
+	    struct stat fst_buf;
+	    for (i=1; i <= rotateCount; i++) {
+		sprintf(oldName, "%s/%s.%d%s%s", dirName, baseName,
+			rotateCount + 1, fileext, compext);
+		if(!stat(oldName,&fst_buf)
+		    && (((nowSecs - fst_buf.st_mtime)/60/60/24)
+			> log->rotateAge)) {
+		    char * mailFilename = (globResult.gl_pathv)[i];
+		    if (!hasErrors && log->logAddress)
+			hasErrors = mailLogWrapper(mailFilename, mailCommand, logNum, log);
+		    if (!hasErrors)
+			hasErrors = removeLogFile(mailFilename);
+		}
+	    }
+	}
     
     sprintf(oldName, "%s/%s.%d%s%s", dirName, baseName,
             logStart + rotateCount, fileext, compext);


Comment 29 Ruediger Oertel 2005-10-09 23:43:24 UTC
well, it's really been a while since I wrote that code, but at first look: 
for all rotated files (for one given target) see if each file is older 
than the limit set by "rotateAge" and mail the file out if wanted and 
delete the file. 
 

Comment 30 Peter Vrabec 2005-10-10 09:36:22 UTC
for (i=1; i <= rotateCount; i++) {
		sprintf(oldName, "%s/%s.%d%s%s", dirName, baseName,
			rotateCount + 1, fileext, compext);
		if(!stat(oldName,&fst_buf)
		    && (((nowSecs - fst_buf.st_mtime)/60/60/24)
This is not for all rotated files, is it? If rotate==4 it stats oldname(ex.
foo.5) for 4 times.

char * mailFilename = (globResult.gl_pathv)[i];
glob() is not callen before this line => globResult is not allocated.


I'd like to rewrite it to:
      if ( log->rotateAge ) {
            struct stat fst_buf;
            for (i=1; i <= rotateCount; i++) {
                sprintf(oldName, "%s/%s.%d%s%s", rotNames->dirName,
rotNames->baseName,
                        i, fileext, compext);
                if(!stat(oldName,&fst_buf)
                    && (((nowSecs - fst_buf.st_mtime)/60/60/24)
                        > log->rotateAge)) {
                    char * mailFilename = oldName;
                    if (!hasErrors && log->logAddress)
                        hasErrors = mailLogWrapper(mailFilename, mailCommand,
logNum, log);
                    if (!hasErrors)
                        hasErrors = removeLogFile(mailFilename);
                }
            }
        }
What do u think about this? 
Another question is - see bug #169885.
thx. for help





Comment 31 Ruediger Oertel 2005-10-10 23:40:46 UTC
uhmm, the "else" case, MAXAGE without DATEEXT ... 
well, looks like I did never really try or use that :( 
 
yes, your code looks perfectly right. 


Note You need to log in before you can comment on or make changes to this bug.