Bug 997983 - swift in RDO logs container, object and account to LOCAL2 log facility which floods /var/log/messages
swift in RDO logs container, object and account to LOCAL2 log facility which ...
Product: RDO
Classification: Community
Component: openstack-swift (Show other bugs)
Unspecified Unspecified
medium Severity low
: Milestone3
: Havana
Assigned To: Pete Zaitcev
Shai Revivo
Depends On:
  Show dependency treegraph
Reported: 2013-08-16 12:53 EDT by Perry Myers
Modified: 2016-04-26 10:09 EDT (History)
6 users (show)

See Also:
Fixed In Version: openstack-swift-2.2.0-2.fc22
Doc Type: Release Note
Doc Text:
Since the service configures rsyslog daemon to log into /var/log/swift/swift.log, it is adviseable to add "local0.none" to the line for /var/log/messags in /etc/rsyslog.conf. Doing so prevents double logging.
Story Points: ---
Clone Of:
Last Closed: 2016-03-30 19:00:33 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
Candidate 0 (2.13 KB, patch)
2013-10-08 23:13 EDT, Pete Zaitcev
zaitcev: review? (apevec)
Details | Diff
Candidate 1 (3.26 KB, patch)
2014-05-28 00:41 EDT, Pete Zaitcev
zaitcev: review? (apevec)
Details | Diff

  None (edit)
Description Perry Myers 2013-08-16 12:53:43 EDT
Description of problem:
From the config files for Swift as deployed in RDO:

account-server.conf:log_facility = LOG_LOCAL2
container-server.conf:log_facility = LOG_LOCAL2
object-server.conf:log_facility = LOG_LOCAL2

Should we be logging Swift to /var/log/messages vs. having a separate set of logs in /var/log/swift like all of the other services?
Comment 1 Pete Zaitcev 2013-08-18 10:00:47 EDT
I should note that original Swift used to contain stubs to log into
/var/log/swift.log, but they bitrotted. The log rotation is the
main problem.
Comment 2 Ayal Baron 2013-08-27 05:23:00 EDT
(In reply to Pete Zaitcev from comment #1)
> I should note that original Swift used to contain stubs to log into
> /var/log/swift.log, but they bitrotted. The log rotation is the
> main problem.

Why is swift different in this respect? logrotate should be configured for the logs of all the services in the same way.
Comment 3 Pete Zaitcev 2013-10-08 23:09:48 EDT
As it turned out, I forgot where the main difficulty was. The problem
is how to modify rsyslog.conf. I propose to punt it over to docs.
Comment 4 Pete Zaitcev 2013-10-08 23:13:19 EDT
Created attachment 809633 [details]
Candidate 0
Comment 5 Pete Zaitcev 2013-10-10 16:11:48 EDT
Lon offered a suggestion: modify Swift to log local2.debug. This way
the messages won't end in /var/log/messages with the stock rsyslog.conf,
but we can log them in /var/log/swift/swift.log.
Comment 6 Pete Zaitcev 2014-05-23 22:42:44 EDT
The problem with Lon's suggestion is that the configuration log_level
does not change at what level Swift logs. It only changes what the
threshold or muting is (done with logger.setLevel()). Thus, to make
logger.info() to log at syslog level "debug" requires a special shim
that effects such. It's not impossible, but violates the principle
of least surprise.

A quick sed string would be:
 sed -e "s/cron.none[ \t]*\/var\/log\/messages/cron.none;local0.none\t\/var\/log\/messages/" -i- /etc/rsyslog.conf
Comment 7 Pete Zaitcev 2014-05-28 00:41:28 EDT
Created attachment 899821 [details]
Candidate 1
Comment 8 Tomas Heinrich 2014-05-29 04:10:14 EDT
(In reply to Pete Zaitcev from comment #7)

I'm maintaining rsyslog and let me start by saying that our default setup for allowing applications to automatically drop in rsyslog configuration is suboptimal. I've been thinking about it lately and don't have any elegant solution so far. What we have now is certainly not it.

> Candidate 1

local0.*;local2.*        /var/log/swift/swift.log
&                        ~

I've skimmed through the suggested configuration.
Lets imagine this hypothetical scenario:
The file name would be ordered pretty far lexicographically. Suppose a similar app provides a conf file /etc/rsyslog.d/slug.conf containing:
local0.*        /var/log/slug/slug.log

This looks fairly benign; it doesn't delete the logs, so swift won't be affected. But since "slug" is ordered before "swift" and thus processed first, the slug.log would contain all the "slug" logs including all "swift" logs, breaking your use case of isolating the "swift" logs.
This isn't a very surprising outcome, but I hope it illustrates how hard it is to actually achieve what you wanted.

Another point is that using localN as the filter isn't advisable. There are only eight of such facilities and the chance of collision is high. If you feel you must filter anyway, use a syslog tag (application name) for it. Or even better something unique(-ier).

I'm thinking about restructuring the configuration to better support such usecases, but at the moment, my advice would be:
If you don't want to log to /var/log/messages and don't have non-syslog way of logging to a file, turn logging off and instruct the user to turn it on manually.
Or, if you really, really need to automatically drop in cfg, try to minimize the potential harm. Erasing local0 and local2 is not it.
Comment 9 Pete Zaitcev 2014-10-21 15:27:40 EDT
See also bug 1154947.

Note You need to log in before you can comment on or make changes to this bug.