RDO tickets are now tracked in Jira https://issues.redhat.com/projects/RDO/issues/
Bug 997983 - swift in RDO logs container, object and account to LOCAL2 log facility which floods /var/log/messages
Summary: swift in RDO logs container, object and account to LOCAL2 log facility which ...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: RDO
Classification: Community
Component: openstack-swift
Version: unspecified
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: Milestone3
: Havana
Assignee: Pete Zaitcev
QA Contact: Shai Revivo
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-08-16 16:53 UTC by Perry Myers
Modified: 2023-04-07 10:43 UTC (History)
6 users (show)

Fixed In Version: openstack-swift-2.2.0-2.fc22
Doc Type: Release Note
Doc Text:
Since the service configures rsyslog daemon to log into /var/log/swift/swift.log, it is adviseable to add "local0.none" to the line for /var/log/messags in /etc/rsyslog.conf. Doing so prevents double logging.
Clone Of:
Environment:
Last Closed: 2016-03-30 23:00:33 UTC
Embargoed:


Attachments (Terms of Use)
Candidate 0 (2.13 KB, patch)
2013-10-09 03:13 UTC, Pete Zaitcev
no flags Details | Diff
Candidate 1 (3.26 KB, patch)
2014-05-28 04:41 UTC, Pete Zaitcev
no flags Details | Diff

Description Perry Myers 2013-08-16 16:53:43 UTC
Description of problem:
From the config files for Swift as deployed in RDO:

account-server.conf:log_facility = LOG_LOCAL2
container-server.conf:log_facility = LOG_LOCAL2
object-server.conf:log_facility = LOG_LOCAL2

Should we be logging Swift to /var/log/messages vs. having a separate set of logs in /var/log/swift like all of the other services?

Comment 1 Pete Zaitcev 2013-08-18 14:00:47 UTC
I should note that original Swift used to contain stubs to log into
/var/log/swift.log, but they bitrotted. The log rotation is the
main problem.

Comment 2 Ayal Baron 2013-08-27 09:23:00 UTC
(In reply to Pete Zaitcev from comment #1)
> I should note that original Swift used to contain stubs to log into
> /var/log/swift.log, but they bitrotted. The log rotation is the
> main problem.

Why is swift different in this respect? logrotate should be configured for the logs of all the services in the same way.

Comment 3 Pete Zaitcev 2013-10-09 03:09:48 UTC
As it turned out, I forgot where the main difficulty was. The problem
is how to modify rsyslog.conf. I propose to punt it over to docs.

Comment 4 Pete Zaitcev 2013-10-09 03:13:19 UTC
Created attachment 809633 [details]
Candidate 0

Comment 5 Pete Zaitcev 2013-10-10 20:11:48 UTC
Lon offered a suggestion: modify Swift to log local2.debug. This way
the messages won't end in /var/log/messages with the stock rsyslog.conf,
but we can log them in /var/log/swift/swift.log.

Comment 6 Pete Zaitcev 2014-05-24 02:42:44 UTC
The problem with Lon's suggestion is that the configuration log_level
does not change at what level Swift logs. It only changes what the
threshold or muting is (done with logger.setLevel()). Thus, to make
logger.info() to log at syslog level "debug" requires a special shim
that effects such. It's not impossible, but violates the principle
of least surprise.

A quick sed string would be:
 sed -e "s/cron.none[ \t]*\/var\/log\/messages/cron.none;local0.none\t\/var\/log\/messages/" -i- /etc/rsyslog.conf

Comment 7 Pete Zaitcev 2014-05-28 04:41:28 UTC
Created attachment 899821 [details]
Candidate 1

Comment 8 Tomas Heinrich 2014-05-29 08:10:14 UTC
(In reply to Pete Zaitcev from comment #7)

I'm maintaining rsyslog and let me start by saying that our default setup for allowing applications to automatically drop in rsyslog configuration is suboptimal. I've been thinking about it lately and don't have any elegant solution so far. What we have now is certainly not it.

> Candidate 1

swift.conf:
local0.*;local2.*        /var/log/swift/swift.log
&                        ~


I've skimmed through the suggested configuration.
Lets imagine this hypothetical scenario:
The file name would be ordered pretty far lexicographically. Suppose a similar app provides a conf file /etc/rsyslog.d/slug.conf containing:
local0.*        /var/log/slug/slug.log

This looks fairly benign; it doesn't delete the logs, so swift won't be affected. But since "slug" is ordered before "swift" and thus processed first, the slug.log would contain all the "slug" logs including all "swift" logs, breaking your use case of isolating the "swift" logs.
This isn't a very surprising outcome, but I hope it illustrates how hard it is to actually achieve what you wanted.

Another point is that using localN as the filter isn't advisable. There are only eight of such facilities and the chance of collision is high. If you feel you must filter anyway, use a syslog tag (application name) for it. Or even better something unique(-ier).

I'm thinking about restructuring the configuration to better support such usecases, but at the moment, my advice would be:
If you don't want to log to /var/log/messages and don't have non-syslog way of logging to a file, turn logging off and instruct the user to turn it on manually.
Or, if you really, really need to automatically drop in cfg, try to minimize the potential harm. Erasing local0 and local2 is not it.

Comment 9 Pete Zaitcev 2014-10-21 19:27:40 UTC
See also bug 1154947.


Note You need to log in before you can comment on or make changes to this bug.