Created attachment 125976 [details]
/var/log/messages showing logrotate error messages

It looks like your anacron process is exiting / crashing for some reason :
> # service anacron start
> Starting anacron:                                          [  OK  ]
> # service anacron status
> anacron dead but subsys locked
I cannot reproduce this problem here.

You are running on an i386/i686 platform, or x86_64 ?

Please can you gather some further data:

1. Ensure anacron is stopped, and the lockfile is removed:
 # service anacron stop
 # rm -f /var/lock/subsys/anacron
2. Try getting anacron to exit / crash on startup - as root:
 # ulimit -c unlimited
 # cd /var/spool/anacron
 # strace -vfo /tmp/anacron.strace.log anacron -s
The strace should exit when anacron exits. 
3. Please compress and send the log file and any core dump files:
 # tar -cpf /tmp/anacron.strace.log /var/spool/anacron/core* | gzip >
/tmp/anacron-dbg.tar.gz
and send the /tmp/anacron-dbg.tar.gz file to me (jvdias) or
append it to this bug report - thank you.

Created attachment 126119 [details]
anacron strace log

Note, no crash dump was made in the folder you suggested (and I couldn't make
the tar command work as given, so did the tar and gzip separately).

Created attachment 126120 [details]
terminal process and responses

oops: forgot the architecture: i386/i686 AMX Athlon Thunderbird
# uname -a
Linux davidthome 2.6.15-1.2051_FC5 #1 Mon Mar 13 20:06:52 EST 2006 i686 athlon
i386 GNU/Linux

Thank you - I think I can see what is happening now.

When anacron has actually run all the jobs that could be scheduled for a given
day, it exits, with the message 'Normal exit (0 jobs run)' written to 
/var/log/cron - if you do:
  # grep anacron /var/log/cron
you should see the anacron messages - if they are different, please append them
to this bug report.

So, unlike most services run from initscripts, anacron does not stay running
all the time; I guess once it has determined that no jobs can be scheduled
that day, it assumes it will run again the next day the job is to scheduled,
and has no more work to do, so it exits.

anacron is only meant to run jobs that have not been run for a certain time;
not run jobs at certain times, which is crond's job. 

Yes, this whole situation is far from optimal, and will be rectified by 
replacement of anacron with crond; crond will be able to parse anacrontab files,
and record the last time anacron jobs were run, and so will behave like anacron 
and run anacron jobs that have not been run for a certain time - this is work
in progress, and should be done shortly (a few weeks). 

But your real problem is with logrotate:

Feb 15 08:25:23 davidthome logrotate: ALERT exited abnormally with [1]

For some reason, logrotate, whether run by anacron OR crond, is getting an
error which prevents log files being rotated.

Are there any other logrotate messages in /var/log/messages ? 
# grep logrotate /var/log/messages

Does logrotate produce any output messages ? They should appear as mail 
sent to the root or anacrontab MAILTO= user with subject 
"Anacron job 'cron.daily'" . Did you get any such messages ?

If there are no messages from logrotate / cron.daily, then please try this
command as root:
  # strace -fvo /tmp/logrotate.strace /usr/sbin/logrotate /etc/logrotate.conf
and append the /tmp/logrotate.strace file (compressed) and command output to 
this bug report - thanks.

Created attachment 126125 [details]
/var/log/anacron

There is one problem: has trouble with sendmail.
# ls -l /usr/sbin/send*
ls: /usr/sbin/send*: No such file or directory

# rpm -qa|grep mail
procmail-3.22-16.2.1
mailx-8.1.1-44.2.1
mailcap-2.1.20-1
ie. sendmail is not installed. Is this supposed to get installed automatically
?

Created attachment 126126 [details]
grep -B 3 -A 2 logrotate /var/log/messages*

An selinux denied occurs at the same second that logrotate logs, but it doesn't
give me enough hints to say whether it is related.
I have manually renamed /var/log/messages, so the attach is a (possibly)
non-time-linear, listing of the last months messages* files (its a bit ugly).

# mail
Mail version 8.1 6/6/93.  Type ? for help.
"/var/spool/mail/root": 2 messages 2 new
>N  1 root  Sat Feb 11 07:20  19/771   "Anacron job 'cron.dai"
 N  2 root  Sun Feb 12 10:08 802/25844 "LogWatch for davidtho"
&1
From root  Sat Feb 11 07:20:07 2006
Date: Sat, 11 Feb 2006 07:20:06 -0500
From: root (Anacron)
To: root
Content-Type: text/plain; charset="ANSI_X3.4-1968"
Subject: Anacron job 'cron.daily'
/etc/cron.daily/0logwatch:
/usr/sbin/sendmail: Permission denied
&2
...yum updated:
    procmail.i386 3.22-16.2
    sendmail.i386 8.13.5-2.2
    fetchmail.i386 6.3.2.1-1.1
    mailx.i386 8.1.1-44.2
# grep mail /var/log/yum.log*  (reordered by date)
/var/log/yum.log.1:Feb 11 08:52:26 Updated: procmail.i386 3.22-16.2
/var/log/yum.log.1:Feb 11 08:52:54 Updated: mailx.i386 8.1.1-44.2
/var/log/yum.log.1:Feb 11 09:05:53 Updated: sendmail.i386 8.13.5-2.2
/var/log/yum.log.1:Feb 11 09:10:03 Updated: fetchmail.i386 6.3.2.1-1.1
/var/log/yum.log:Feb 20 00:41:38 Updated: mailcap.noarch 2.1.20-1
/var/log/yum.log:Feb 20 00:45:58 Updated: procmail.i386 3.22-16.2.1
/var/log/yum.log:Feb 20 00:46:06 Updated: mailx.i386 8.1.1-44.2.1

I didn't forcefully remove sendmail, but both rpm and the lack of binary on
disk show it isn't installed anymore. Perhaps another occurence like my bug
185177 ?
===
# strace -fvo logrotate.strace /usr/sbin/logrotate /etc/logrotate.conf
error: Ignoring yum.rpmorig, because of .rpmorig ending
error: mysqld:3 unknown user 'mysql'

# rpm -V logrotate
--nada!

Created attachment 126127 [details]
/etc/logrotate.conf

Created attachment 126128 [details]
cat /etc/logrotate.d/*

Just in case something relevant: I do notice a heap of ctrl chars in the mgetty
part, but I don't know if this is normal.

Created attachment 126129 [details]
# strace -fvo logrotate.strace /usr/sbin/logrotate /etc/logrotate.conf

Thanks for the log files - I may have found the problem:

It appears that the 'mysqld' file precedes the 'syslog' file in 
/etc/logrotate.d, you have an existing /var/log/mysqld.log, 
and the 'mysql' user does not exist, causing logrotate to exit with an error:

10372 lstat64("/var/log/mysqld.log", {st_dev=makedev(253, 0), st_ino=951360,
st_mode=S_IFREG|0640, st_nlink=1,
 st_uid=0, st_gid=27, st_blksize=4096, st_blocks=8, st_size=0,
st_atime=2006/03/05-13:49:24, st_mtime=2006/03/
05-13:49:24, st_ctime=2006/03/05-13:49:24}) = 0
... ( so the /var/log/mysqld.log file exists, with gid 27 (which should be 
...   the mysql uid/gid ) ... 
10372 write(2, "error: ", 7)            = 7
10372 write(2, "mysqld:3 unknown user \'mysql\'\n", 30) = 30
...
10372 exit_group(1)                     = ?


Now, probably logrotate should not exit with an error if the owning user
of a log file to be rotated does not exist; it should not attempt to 
rotate that log file, and should emit an error message, but it should proceed
to the next log file, not exit with an error (I've CC'ed the logrotate
maintainer on this bug).

Do you still have the mysql package installed ?
Did you manually remove the mysql user ?

If you still have the mysql package installed, then if you did not
remove the mysql user manually, it is a mysql bug that the mysql user 
does not exist - please open a mysql bug. If you removed the mysql package,
then the mysql package %postun script should not remove the mysql user if 
there are still existing log files, or should remove such log files - it 
is also Red Hat packaging policy not to remove any system userid - so 
that is also a mysql bug.

And anacron will still run the job if sendmail is not installed,
but will be unable to send output mail, but will not complain about
it (an anacron bug that will be fixed when it is replaced by new crond).

Very sorry for this mess. A workaround is to recreate the mysql user:
  # useradd -u 27 -g mysql -n -r -M -s /sbin/nologin -d /var/lib/mysql mysql
or, if you have removed the mysql package, remove the /var/log/mysql* files.

Hi Jason,
*** note: user and group gui tool won't start (error): shows no groups nor users.
*** I didn't manually remove the mysql user
# grep sql /etc/passwd
hsqldb:x:100:101::/var/lib/hsqldb:/sbin/nologin
/\ missing mysql user.
# grep sql /etc/shadow
hsqldb:!!:13190:0:99999:7:::
mysql:!!:13190::::::
/\ mysql is present.
# grep sql /etc/group
hsqldb:x:101:
mysql:x:27:
# grep sql /etc/gshadow
hsqldb:x::
mysql:!::
# rpm -qa|grep mysql|sort
libdbi-dbd-mysql-0.8.1a-1.2.1
mysql-5.0.18-2.1
mysql-connector-odbc-3.51.12-1.2.1
mysql-server-5.0.18-2.1
[root@davidthome ~]# service mysqld status
mysqld is stopped
[root@davidthome ~]# service mysqld start
chown: `mysql:mysql': invalid user
chown: `mysql:mysql': invalid user
*** as you have found!
[root@davidthome ~]# grep mysql /var/log/yum.log*
/var/log/yum.log:Feb 20 00:47:22 Updated: mysql.i386 5.0.18-2.1
/var/log/yum.log:Feb 20 00:52:39 Updated: mysql-server.i386 5.0.18-2.1
/var/log/yum.log:Feb 20 00:52:47 Updated: mysql-connector-odbc.i386 3.51.12-1.2.1
/var/log/yum.log:Feb 20 00:52:47 Updated: libdbi-dbd-mysql.i386 0.8.1a-1.2.1
/var/log/yum.log:Mar 05 13:47:41 Erased: mysql-server
/var/log/yum.log:Mar 05 13:49:24 Installed: mysql-server.i386 5.0.18-2.1

*** I did notice the mysql start user error during startup, and uninstalled /
reinstalled to try to fix it. As you say that process didn't actually fix a
login / user problem.

# useradd -u 27 -g mysql -n -r -M -s /sbin/nologin -d /var/lib/mysql mysql
# grep sql /etc/passwd
hsqldb:x:100:101::/var/lib/hsqldb:/sbin/nologin
mysql:x:27:27::/var/lib/mysql:/sbin/nologin
# grep sql /etc/shadow 
hsqldb:!!:13190:0:99999:7:::
mysql:!!:13222::::::
[root@davidthome ~]# service mysqld status
mysqld is stopped
[root@davidthome ~]# service mysqld start

** after 15 seconds, prompt not returned.
# service mysqld status
mysqld is stopped
#sy-co-se mysql start waits 30 seconds then says:
  service mysqld started, but updates the status to mysqld dead but subsys locked.
# ls -l /var/lock/subsys/mysqld
-rw-r--r-- 1 root root 0 Mar 15 12:17 /var/lock/subsys/mysqld
[root@davidthome ~]# rm /var/lock/subsys/mysqld
rm: remove regular empty file `/var/lock/subsys/mysqld'? y

* retry the service start fails identically.
* pirut remove / install mysql-server (with the fixed mysql user)
* mysql now start OK.
* # /usr/sbin/logrotate /etc/logrotate.conf
error: Ignoring yum.rpmorig, because of .rpmorig ending
[root@davidthome anacron]# ls -l /var/log/me*
-rw------- 1 root root     594 Mar 15 13:28 /var/log/messages
-rw------- 1 root root  371462 Mar 15 13:28 /var/log/messages.1
-rw------- 1 root root  143817 Feb 12 10:08 /var/log/messages.2
-rw------- 1 root root 2932210 Mar 11 13:02 /var/log/messages.2006-03
-rw------- 1 root root   42661 Mar 11 13:13
/var/log/messages.2006-03-11.bad-iso.mount.txt
-rw------- 1 root root  741484 Mar 13 00:36
/var/log/messages.2006-03-12.before-suspend.tt
-rw------- 1 root root  182387 Mar 13 11:38 /var/log/messages.2006-03-13.txt

So, after fixing the username for mysql, reinstalling mysqld, I can now
successfully run Logrotate manually, and it does it's thing.

I am happy if you want to close this bug in which ever way you see fit, or I can
provide more info if you think it is relevant/useful.

OK, well several issues came out of this bug:
 o  anacron should not be exit-ing if it thinks it has no more work to do;
    this will be fixed with the new cron rpm anacron replacement.
 o  Bug 185538 - logrotate should continue to process logrotate.d files after
    it gets an error for a single logrotate.d file.

I'm changing this bug into an enhancement request for the anacron issue.

Changing in notabug - I don't see in point to make cron work with anacron.