Bug 136141

The "exiting on signal 15" is from shutdown. The startup messages are
lost completely with above example steps.

Fedora Core 3 is now maintained by the Fedora Legacy project for security
updates only. If this problem is a security issue, please reopen and
reassign to the Fedora Legacy product. If it is not a security issue and
hasn't been resolved in the current FC5 updates or in the FC6 test
release, reopen and change the version to match.

Thank you!

*sigh*

Yes, I could say this is a security issue, because contrary to
the status messages the logging service is disabled.

Reproducible on FC5.

Yeah, I agree. Adding Security keyword -- someone can take it away if they
disagree. And changing the summary to be more descriptive so maybe it gets a bit
more attention.

still there on Fedora 7

This message is a reminder that Fedora 7 is nearing the end of life. Approximately 30 (thirty) days from now Fedora will stopmaintaining and issuing updates for Fedora 7. It is Fedora's policy to close all bug reports from releases that are no longermaintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '7'.

Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, 
simply change the 'version' to a later Fedora version prior to Fedora 7's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 7 is end 
of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug. If you are unable to change the version, please add a comment here and someone will do it 
for you.

Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by 
events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. If 
possible, it is recommended that you try the newest available Fedora distribution to see if your bug still exists.

Please read the Release Notes for the newest Fedora distribution to make sure it will meet your needs:
http://docs.fedoraproject.org/release-notes/

The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 7 changed to end-of-life (EOL) status on June 13, 2008. 
Fedora 7 is no longer maintained, which means that it will not 
receive any further security or bug fix updates. As a result we 
are closing this bug. 

If you can reproduce this bug against a currently maintained version 
of Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

Reproducible also with rsyslog (rsyslog-3.18.1-2.fc9.i386).

reassign to rsyslog

When rsyslog encounters an error in the configuration file, it generates
a message with syslog.error priority, which can be logged elsewhere.
When running in debug mode, it also writes the message to stderr.
Since version 3.21.1 rsyslog supports a command line option (-N) to validate the configuration file.

The original steps to reproduce it are still valid.
$ rpm -q rsyslog
rsyslog-3.21.10-2.fc10.i386

A simple typo -- as in the example in the original bug report -- results in DoS. With rsyslog.

I'm not sure what you mean by DoS;
If there's a typo in a rule definition, that rule is disabled and a warning is issued where possible. All other valid rules still work.
After making changes to the configuration file, you have a way to verify they are correct.

It might be possible to always run 'rsyslog -N -f <cfg>' from the init script before starting rsyslog, but I'm not sure if this is the right solution. Are you ok with this?

Running it with -N from the init script seems like a great idea.

With DoS I mean that an _entire rule_ is ignored -- silently (!) -- just because there's a subtle syntax error in it. Even a ':' instead of ';' as the separator between the selectors can be used to disable an entire rule. Other typos disable the rule, too. As a consequence, rsyslog not even logs its startup status message anymore as the '*.info' selector is among the ones that are disabled silently. rsyslog doesn't print anything on stderr either. Where do you get a warning?

> After making changes to the configuration file, you have a way to
> verify they are correct.

At least the software ought to help with verification as it parses the rules anyway.

> With DoS I mean that an _entire rule_ is ignored -- silently (!) -- just
> because there's a subtle syntax error in it.

What else would you want to do with a malformed rule?

> silently (!)

If there are other places to log to or rsyslog can write to stderr, it does so.

> Where do you get a warning?

If you run rsyslog in debug mode, it writes to stderr. If there are rules to log messages with the respective priority, they get logged. Provided you won't create typos in those specific rules.

What's your opinion on the proposed solution?

> What else would you want to do with a malformed rule?

Complain about it on stderr. Always.

> If there are other places to log to or rsyslog can write to stderr, it does so.

It doesn't. One needs to run it manually with special options.

> What's your opinion on the proposed solution?

-N is useless. Its noisy and returns exit-code 0 even for a config file with syntax errors.

What would you consider an appropriate action to be taken if there is a syntax error?

I forgot to mention: error messages go to stderr by default. This is disable by specifiying

$ErrorMessagesToStderr off

[Implemented in 3.21.1+]

But if that is not given, all error messages to to stderr. Might it be that nobody takes them from there?

> What would you consider an appropriate action to be taken
> if there is a syntax error? 

With option -N, to exit with a non-zero return-code.

Without option -N, see comment 16.

> I forgot to mention: error messages go to stderr by default. 

Doesn't work in Fedora 10. One mistyped selector, and an entire rule is ignore silently. See comment 14.

comment 16 demands that output is written to stderr. I don't know why this is disabled in Fedora, but rsyslog be default does that.

But what I was really interested in is what shall happen if there is a problem with the rule? Is it sufficient to just push it to stderr? What to do if -N comes back with an error exit code?

This is not really a simple issue, please also have a look at

http://blog.gerhards.net/2008/07/rsyslog-error-reporting-how-to-do-it.html

Rainer

>> I forgot to mention: error messages go to stderr by default. 

> Doesn't work in Fedora 10. One mistyped selector, and an entire rule is ignore
> silently. See comment 14.

By the time rsyslog parses the config file, it is already forked and without access to stderr.

(Rainer, there's actually a bug in that rsyslog always tries to write to stderr. :) When forked, it accidentally writes to the file opened with fd 2. Hope the patch got to you.)

So IMO a reasonable solution is to change the code to return some non-zero value when -N encounters an error, run rsyslog -N first, discard the output as not to spam the screen and if retval != 0, print a warning about config file syntax a then start the daemon.

What do you say?

> By the time rsyslog parses the config file, it is already forked and without
> access to stderr.

ah, yes, that's right...

I just looked, the patch made it to me, but only to the (not visited) spam folder ;) Will apply.

The solution sounds reasonable. Will look what it takes to return the error code.

I have now looked at the code (also, patch is applied). I think it probably is not sufficient to return an exit state. There are some other (exotic) conditions that can stop startup, like not being able to create a queue (for whatever wild reason). In those cases, messages are emitted to stderr, too. So it would probably make more sense to keep stderr open after the fork and continue to emit messages to there.

What do you say?

I have added the exit(1) facility to the code [1], but the question with stderr remains. Feedback appreciated.

[1] http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=23dac82b684e966490de707a44144b3ad0ce2323

Whom is that question directed at? I've answered multiple times before that *I* would prefer to learn about syntax errors on stderr always, e.g. in comment 16. Unfortunately, it seems I'm the only one who considers (and considered) the other behaviour as broken. When I originally learnt about this and reported this in 2004 (!) about syslogd (prior to syslog-ng and rsyslog), I was in good hope that I would meet interest in fixing it for Fedora and derived distributions.

OK, so everything you need is to have the messages printed to stderr. But how do you notice there are some messages? This is unclear to me...

Can't you reuse the same parser that is used for the -N check?

| rsyslogd: unknown priority name "info:mail.none" [try http://www.rsyslog.com/e/3000 ]
| rsyslogd: the last error occured in /etc/rsyslog.conf, line 39
| rsyslogd: warning: selector line without actions will be discarded

Why doesn't rsyslog print the same warnings when running without option -N? Why does it discard selectors silently instead?

I'm of the opinion that it should do everything possible to alert the admin to a problem. Printing warnings to standard error may not be sufficient, but I think it's an important element.

That's at the core of my question: what is "everything possible" - and how to configure what it should do? And should it do this always, or needs it be configurable?

And in regard to comment 27: rsyslog does not discard selectors silently. First of all, it logs them, so whatever action you have configured (provided you did it right), you will see them on. That was the primary method of configuring things so that an admin gets the notifications it wants.

Also, messages are intended to be printed on stderr, but there was a bug which caused them to be not printed if rsyslogd was backgrounded. Based on the feedback, I am now tempted to change it that way that even then messages are printed. However, to do so I would like to hear how these stderr messages be retrieved. It does not make sense to include a facility to output to stderr if nobody will ever see stderr. So far, there was no convincing argument that it would help. Please note that I am overlooking something trivial, but if so, please educate me...

What else can I do to avoid that we don't talk past eachother, please?

> rsyslog does not discard selectors silently. First of all, it logs them, 

It does _not_ log them. At least not in the example I offer in this ticket. Yes, it's bad luck that an entire rule gets disabled, see comment 14, please.

> It does not make sense to include a facility to output
> to stderr if nobody will ever see stderr.

What sort of messages do you refer to?

If I log in to a machine and run "sudo /sbin/service rsyslog restart", why would I not see warnings/errors printed to stderr?

> It does _not_ log them. At least not in the example I offer in this ticket.
> Yes, it's bad luck that an entire rule gets disabled, see comment 14, please.

You almost seem to intentionally misunderstand me. Look at the code, or do a

*.* /path/to/file

in your rsyslog.conf and you will see that rsyslog *does* log these messages. But no point in further arguing this...

I made a test with stderr. I changed rsyslogd so that it keeps fd 2 open after a fork. Then I did 

/sbin/service rsyslog start

Nothing showed up on the terminal. Just to make sure, I added an

fprintf(stderr, "rsyslogd start\n");

right as the first line of main(). Again, I did a

/sbin/service rsyslog start

And again, nothing showed up at the controlling terminal. My conclusion is that /sbin/service does suppress the daemon's stderr. Am I wrong here? What kind of magic do I need to do to make stderr appear?

... hold on... I think I made a mistake, I installed the patched version to the wrong path.

Yes, I made a mistake. Looks like /sbin/service does not prevent stderr from showing up ;)

Find a fix for the recent v3-stable here:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=ff6232d2bee06fbbc25ad83a1cf3bb798884679a

It keeps stdout and stderr open across fork(). Thus, error messages can be seen at wherever they point to. I would still suggest to *log* the emitted error messages, if any, via a rule, because you obviously will not see these error messages during an automated system startup.

Note that there is also an additional small cleanup fix in git (or remove the fprintf(stder, ...) after main() yourself.

I'm not sure whether this is the right solution.

Printing all the syntax errors from the init script can be pretty
ugly. Maybe it would be better to just detect errors and issue one big
red warning that you should check your configuration.

What's worse is that rsyslog will keep spamming the console after
startup.

Regarding the other reasons that can prevent rsyslog from running,
obviously the daemon will not start, which is the first clue that
something is wrong, and the init script will issue a failure
notification. Then it's up to you to investigate.

> obviously the daemon will not start

It does. Please restart at comment 1. Thank you!

We have some issue with the new code:

http://kb.monitorware.com/viewtopic.php?p=17147

I knew I should have put this through the beta phase first (it always hits me if I try to shortcut changes...).

I will now see that I change it so that stderr is closed after the initial initialization. I think this is a good compromise. With that mode, the interactive user will be able to view the error messages, but stderr and stdout will be closed thereafter.

BTW: this case once more reminds me that *all* changes except *real* bug fixes belong into the master branch only. I'll see if there is an easy fix, else I remove that functionality from 4.2.0 and introduce it to 4.5.0 again.

I hope this also addresses Toma's concerns. With the current code base, I can not trivially collect the messages and issue a single warning. On the other hand, a number one problem is that users typically don't look at the startup error messages, even if they are recorded (and they are recorded in default config!). So I think it definitely is a big plus if we can emit them right to where these folks work. If it is ugly, then the config is really ugly and so they deserve an ugly display - hopefully that makes them fix the config ;)

I have crafted a patch:

http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=6511278082a7e1e9602385cd24cdb5e363cb702f

I would appreciate if someone could try it and report back.

This message is a reminder that Fedora 10 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 10.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '10'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 10's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 10 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Fedora 10 changed to end-of-life (EOL) status on 2009-12-17. Fedora 10 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.