Bug 1217403
Summary: | [RFE] separate system-level logs of cron cartridge from gear-level logs | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Evgheni Dereveanchin <ederevea> | |
Component: | ImageStreams | Assignee: | Timothy Williams <tiwillia> | |
Status: | CLOSED ERRATA | QA Contact: | libra bugs <libra-bugs> | |
Severity: | medium | Docs Contact: | ||
Priority: | low | |||
Version: | 2.2.0 | CC: | erich, jialiu, jokerman, libra-bugs, libra-onpremise-devel, mmccomas, tiwillia | |
Target Milestone: | --- | |||
Target Release: | --- | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | openshift-origin-cartridge-cron-1.25.4.2-1.el6op | Doc Type: | Enhancement | |
Doc Text: |
Feature: Tag cron's system-level messages as they are sent to syslog.
Reason: Cron's system-level messages may get lost among the gear-level cron logs, which may result in important messages being missed by system administrators.
Result: Tagging cron's system-level messages allows administrators to set up syslog rules that would seperate system-level cron log messages from gear-level messages.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1328495 (view as bug list) | Environment: | ||
Last Closed: | 2016-08-24 19:43:03 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1328495 |
Description
Evgheni Dereveanchin
2015-04-30 10:04:56 UTC
I think this requires two changes. One change to the following line: https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-cron/bin/cron_runjobs.sh#L40 To introduce a TAG flag to the system messages. Example: function log_message() { msg=${1-""} [ -z "$msg" ] && return 0 logger -i -s "user-cron-jobs" -p user.info -t "cron_sys_log:" "`date`: $msg" } and a change to the docs on how to pick up tag's in the openshift syslog plugin. 2: Update the rsyslog configuration to account for this tag (and filter the logs out). Example: module(load="mmopenshift") action(type="mmopenshift") :syslogtag, isequal, "cron_sys_log:" /var/log/ABC.log if $!OpenShift!OPENSHIFT_APP_UUID != '' then *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift") else { *.info;mail.none;authpriv.none;cron.none action(type="omfile" file="/var/log/messages") } (In reply to Eric Rich from comment #3) > I think this requires two changes. One change to the following line: > > https://github.com/openshift/origin-server/blob/master/cartridges/openshift- > origin-cartridge-cron/bin/cron_runjobs.sh#L40 > > To introduce a TAG flag to the system messages. > > Example: > > function log_message() { > msg=${1-""} > [ -z "$msg" ] && return 0 > logger -i -s "user-cron-jobs" -p user.info -t "cron_sys_log:" > "`date`: $msg" > } > The following should provide a fix for this: https://github.com/openshift/origin-server/pull/6381 Commit pushed to master at https://github.com/openshift/origin-server https://github.com/openshift/origin-server/commit/96ab0c03099d35a23f31089a22f60fab31957559 ...cartridge-cron/bin/cron_runjobs.sh: Feature Enhancement for syslog tagging. Bug 1217403 Bugzilla link https://bugzilla.redhat.com/show_bug.cgi?id=1217403 Adding the ability to filter/tag cron messages with syslog Re-test this bug with 2.2/2016-08-11.1, and following comment 11, add the following line to /etc/rsyslog.conf :syslogtag, contains, "cron_sys_log:" /var/log/ABC.log Then restart rsyslog service. Check logs, both system-level and gear-level logs are still sent to /var/log/openshift_gears, and another copy of system-level logs are sent to /var/log/ABC.log, but according to initial report, the request is asking to separate system-level logs from gear-level, is it an expected behavior? # vim /var/log/ABC.log ... Aug 12 14:50:04 node1 cron_sys_log:[25470]: user-cron-jobs Fri Aug 12 14:50:04 CST 2016: :START: minutely cron run for openshift user 'jialiu-php54app-1' Aug 12 14:50:04 node1 cron_sys_log:[25498]: user-cron-jobs Fri Aug 12 14:50:04 CST 2016: :END: minutely cron run for openshift user 'jialiu-php54app-1' # vim /var/log/openshift_gears Aug 12 14:50:04 node1 cron_sys_log:[25470]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 user-cron-jobs Fri Aug 12 14:50:04 CST 2016: :START: minutely cron run for openshift user 'jialiu-php54app-1' Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 Fri Aug 12 14:50:04 CST 2016: START minutely cron run Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 12 14:50:04 node1 run-parts(/var/lib/openshift/jialiu-php54app-1/app-root/runtime/repo//.openshift/cron/minutely)[25489]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 starting test.sh Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 /var/lib/openshift/jialiu-php54app-1/app-root/runtime/repo//.openshift/cron/minutely/test.sh: Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 TEST ------- Fri Aug 12 14:50:04 CST 2016 Aug 12 14:50:04 node1 run-parts(/var/lib/openshift/jialiu-php54app-1/app-root/runtime/repo//.openshift/cron/minutely)[25495]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 finished test.sh Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 Fri Aug 12 14:50:04 CST 2016: END minutely cron run - status=0 Aug 12 14:50:04 node1 cron_minutely[25476]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 12 14:50:04 node1 cron_sys_log:[25498]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 user-cron-jobs Fri Aug 12 14:50:04 CST 2016: :END: minutely cron run for openshift user 'jialiu-php54app-1' The below entry is also catching the cron_sys_log entries: *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift") To remedy this, the new entry we are adding should be placed before the mmopenshift configuration and a '&~' added to the next line to stop further filtering of the log message. See below for an example: :syslogtag, contains, "cron_sys_log:" /var/log/ABC.log &~ action(type="mmopenshift") if $!OpenShift!OPENSHIFT_APP_UUID != '' then # annotate and log syslog output from gears specially *.* action(type="omfile" file="/var/log/openshift_gears" template="OpenShift") else # otherwise send syslog where it usually goes *.info;mail.none;authpriv.none;cron.none action(type="omfile" file="/var/log/messages") Following comment 14, the behavior is working as expection, so the bug is verified. ==> /var/log/openshift_gears <== Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 Mon Aug 15 10:18:04 CST 2016: START minutely cron run Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 15 10:18:04 node1 run-parts(/var/lib/openshift/jialiu-php54app-1/app-root/runtime/repo//.openshift/cron/minutely)[9232]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 starting test.sh Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 /var/lib/openshift/jialiu-php54app-1/app-root/runtime/repo//.openshift/cron/minutely/test.sh: Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 TEST ------- Mon Aug 15 10:18:04 CST 2016 Aug 15 10:18:04 node1 run-parts(/var/lib/openshift/jialiu-php54app-1/app-root/runtime/repo//.openshift/cron/minutely)[9238]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 finished test.sh Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 Mon Aug 15 10:18:04 CST 2016: END minutely cron run - status=0 Aug 15 10:18:04 node1 cron_minutely[9219]: app=php54app ns=jialiu appUuid=57a8643882611da20d00029b gearUuid=jialiu-php54app-1 __________________________________________________________________________ ==> /var/log/ABC.log <== Aug 15 10:18:04 node1 cron_sys_log:[9213]: user-cron-jobs Mon Aug 15 10:18:04 CST 2016: :START: minutely cron run for openshift user 'jialiu-php54app-1' Aug 15 10:18:04 node1 cron_sys_log:[9241]: user-cron-jobs Mon Aug 15 10:18:04 CST 2016: :END: minutely cron run for openshift user 'jialiu-php54app-1' Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-1773.html |