Red Hat Bugzilla – Bug 1096900
Log messages are being interpolated by syslog_logger.rb, causing random app operation failures
Last modified: 2014-06-18 16:00:15 EDT
+++ This bug was initially created as a clone of Bug #1095914 +++ Description of problem: The syslog_logger.rb implementation in openshift-origin-node is making incorrect use of the Ruby syslog library. The code is treating the format string parameter as a formatted message parameter. This causes the raw input string (a log message) to be interpolated underneath in the syslog implementation. Any log messages dispatched via this code which contain the '%' character can result in exceptions unless followed by another character that happens to make the string conform to the printf substitution rules. These exceptions can occur anywhere in the openshift-origin-node code, or the MCollective agent. The code should be changed to use a generic '%s' format string, passing the processed log message as the format args. This will make the call log the input verbatim. Version-Release number of selected component (if applicable): How reproducible: Configure platform logging for syslog via /etc/openshift/node.conf: PLATFORM_LOG_CLASS=SyslogLogger PLATFORM_SYSLOG_THRESHOLD=LOG_INFO PLATFORM_SYSLOG_TRACE_ENABLED=1 PLATFORM_LOG_FILE=/var/log/openshift/node/platform.log PLATFORM_LOG_LEVEL=INFO PLATFORM_TRACE_LOG_FILE=/var/log/openshift/node/platform-trace.log PLATFORM_TRACE_LOG_LEVEL=DEBUG PLATFORM_LOG_CONTEXT_ENABLED=1 PLATFORM_LOG_CONTEXT_ATTRS=request_id,app_uuid One example that happens to easily produce the offending log messages is: rhc app create sinkerror jbossas-7 -pp --no-git --noprompt --from-code=git://github.com/openshift/kitchensink-example.git Steps to Reproduce: 1. 2. 3. Actual results: The MCollective agent will crash with an exception such as: May 8 12:45:45 ex-std-node3 mcollectived[1965]: agents.rb:139:in `rescue in block in dispatch' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.23.9/lib/openshift-origin-node/utils/logger/syslog_logger.rb:61:in `log' ... The app will fail to be created due to the crash. Expected results: Correct log output to Syslog and a successfully created application. Additional info: --- Additional comment from Dan Mace on 2014-05-08 16:24:24 EDT --- https://github.com/openshift/origin-server/pull/5401 --- Additional comment from Meng Bo on 2014-05-08 23:36:26 EDT --- Checked on devenv-stage_830, issue has been fixed. With the platform log send to syslog set as comment#0 in node.conf, created the kitchensink jbossas app several times. And no error found in /var/log/ruby193-mcollective.log Move bug to verified.
https://github.com/openshift/enterprise-server/pull/283
commit 56a50d0fc7c1a98af1f5a87fb5d2def418a55274 Commit: Luke Meyer <lmeyer@redhat.com> CommitDate: Thu May 29 11:59:50 2014 -0400 syslog_logger: Prevent unwanted syslog message interpolation #cherrypick origin-server: commit f2328073c329defdb91e2aa009994bdf84570cbe Author: Dan Mace <ironcladlou@gmail.com> Date: Thu May 8 16:15:49 2014 -0400 Prevent unwanted syslog message interpolation Stop treating raw log input as a syslog format string, and instead pass the input as a format arg to the '%s' format string. This will prevent syslog from trying to interpolate random log messages as format strings. Resolve bug https://bugzilla.redhat.com/show_bug.cgi?id=1095914
Check on puddle [2.1.z/2014-06.12] With the platform log send to syslog set as comment#0 in node.conf, create eapmongo app with jbosseap-6 and mongodb-2.4 cartridges ten times. No error found in /var/log/messages Move bug to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHSA-2014-0764.html