Bug 1387229 - RFE: Reduce useless logging noise
Summary: RFE: Reduce useless logging noise
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: RFEs
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: ---
Assignee: RHOS Maint
QA Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-10-20 11:52 UTC by Marko Myllynen
Modified: 2018-08-06 11:52 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-07-13 18:17:40 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Marko Myllynen 2016-10-20 11:52:27 UTC
Description of problem:
By default we're seeing on OSP nodes, especially on controllers, the following kind of kernel messages (see dmesg or journalctl -k output) being flooded although they do not convey any relevant information at all:

SELinux: initialized (dev sysfs, type sysfs), uses genfs_contexts

Also on some nodes, especially on the undercloud node, we're seeing these messages (again, see dmesg or journalctl -k):

systemd-journald[PID]: Failed to set file attributes: Operation not supported

Which again is totally useless, especially considering that the offending file name was not logged so an administrator can't do much about this.

Finally, on some nodes we also see:

systemd[1]: Configuration file /run/systemd/system/*service is marked world-inaccessible. This has no effect as configuration data is accessible via APIs without restrictions. Proceeding anyway.

There are certainly other examples as well but these three seem to be causing the most noise. Their downside in practise is that a) reading logs is harder (or need use time to define and set filtering), b) important message may go by unnoticed due to noise c) disk space is wasted, d) and most importantly important messages may get rotated as growing logs are removed. For example, by default systemd caps its journal logs at 4 GB which on a busy OSP controller node might be a week's worth of logs, or even less.

In more practical terms, in case of an issue it's a good idea to check whether the underlying hardware and kernel are working as expected. This can be done with dmesg or journalctl -k but due to above mentioned log flooding this information is quickly rotated and lost, making troubleshooting and RCAs in some cases impossible.

We should avoid useless log noise by default on all OSP nodes.

Thanks.

Comment 6 Mike Burns 2018-07-13 18:17:40 UTC
Please open bugs for specific components with examples on what we should be reducing.


Note You need to log in before you can comment on or make changes to this bug.