Red Hat Bugzilla – Bug 1337102
pmlogger should avoid redundant logging
Last modified: 2016-09-26 00:15:03 EDT
Description of problem: When doing something like this after starting the pmlogger service using the default configuration: echo "log mandatory on 10sec kernel.all.load" | pmlc -P echo "log mandatory on 10min kernel.all.load" | pmlc -P pmlogger logs these logging changes in its logs (rightly so). However, if doing the latter several times in a row (for example for a script) pmlogger would still log the current state even if there was no change at all in logging configuration making the log messages redundant. The issue with these redundant log messages is that they make it much harder to spot the actual changes from the log file and of course eat quite a lot of disk space in the long run. While one could use "query" to determine the current status and then call pmlc only if needed. But parsing pmlc output and adding logic whether or not to call pmlc doesn't sound optimal in this case and doesn't guarantee that all calls to pmlc are done after such checks. Thanks.
(In reply to Marko Myllynen from comment #0) > Description of problem: > When doing something like this after starting the pmlogger service using the > default configuration: > > echo "log mandatory on 10sec kernel.all.load" | pmlc -P > echo "log mandatory on 10min kernel.all.load" | pmlc -P > > pmlogger logs these logging changes in its logs (rightly so). However, if > doing the latter several times in a row (for example for a script) pmlogger > would still log the current state even if there was no change at all in > logging configuration making the log messages redundant. > > While one could use "query" to determine the current status and then call > pmlc only if needed. But parsing pmlc output and adding logic whether or not > to call pmlc doesn't sound optimal in this case and doesn't guarantee that > all calls to pmlc are done after such checks. > It's very unlikely pmlogger is going to be able to figure this out I think - the way it groups sets of metrics, the different intervals, and the different starting offsets above - all conspire to make this a difficult problem to solve in the general case. I assume you are doing this pmlc invocation from pmie? (not really mentioned here, but IIRC that was mentioned elsewhere) If so, it might be simpler to manage the state there to avoid this situation. Either using the action holdoff logic, or using a script that holds state in the filesystem, or a combination of those and/or your pmlc query idea. I've seen cases where people wrote custom PMDAs with storable metrics indicating to pmie whether certain situations currently hold (like, we have currently logging X at Y interval from pmlc), so that it can choose to take additional action or not. Lots of options, so I suspect changing pmlogger for this is probably not going to happen. (Although, if someone with deeper pmlogger-fu than I have thinks its feasible - please go ahead!). cheers.
Ok, if it sounds tricky then using scripts is probably the most feasible way forward. The action holdoff logic works to a certain degree, perhaps pmie actually could be enhanced a bit: one idea I had in mind was to allow to define variables on the pmie side as part of actions (if this is already possible then I've missed it). So something like: some_limit = 10 some_check = " some_inst ( some.metric ) "; all_sample ( $some_check @0..9 ) > $some_limit && $some_knob == 0 -> some_knob = 1 & shell 10 min "/usr/local/sbin/some_script on"; all_sample ( $some_check @0..9 ) > $some_limit && $some_knob == 1 -> some_knob = 0 & shell 10 min "/usr/local/sbin/some_script off"; Thanks.
As discussed in #c1. The pmie macro support you're after in #c2 exists already Marko - macros are implemented as straight string replacement, so any part of the predicate can be parameterised, exactly as you've done there in your example for $some_check. They are not variables though, so the $some_knob wouldn't work (but there are other ways to achieve that like managing filesystem state, or storable metric values when direct rule feedback needed).