Bug 803101

Summary:	rsyslog appears to truncate messages when logging over the network
Product:	Red Hat Enterprise Linux 5	Reporter:	Brian Hourigan <bhourigan>
Component:	rsyslog	Assignee:	Tomas Heinrich <theinric>
Status:	CLOSED WORKSFORME	QA Contact:	BaseOS QE Security Team <qe-baseos-security>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	5.9	CC:	dgherman, gdestuynder, pvrabec, theinric
Target Milestone:	rc
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-04-19 11:28:11 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Brian Hourigan 2012-03-14 00:29:21 UTC

Description of problem:

I noticed RHEL5 (RHEL6 hosts are unaffected) hosts (i386 and x86_64) were only logging ~50 first characters of log messages when logging to the network. When you remove syslog headers (date, actual header, program name) the message is truncated to about 10 characters.

This means the network logging is unusable, and that logs from programs logging only to the network (like auditd) are lost. I have reproduced this on different hosts even when using the stock rsyslog config.

You can check by running tcpdump -i bond0 -XX port 514 (or 2514 for auditd).

When using the stock syslogd on those systems, messages are not truncated.
When using file logging via rsyslogd, messages are not truncated.
When using network logging via rsyslogd, messages ARE truncated.

And I believe this is caused by one of the RHEL patches to rsyslog.

A workaround is to rollback to rsyslog-3.22.1-3.el5 (previous version)


Version-Release number of selected component (if applicable):

Name        : rsyslog                      Relocations: (not relocatable)
Version     : 3.22.1                            Vendor: Red Hat, Inc.
Release     : 7.el5                         Build Date: Thu 01 Dec 2011 07:41:54 AM PST
Install Date: Mon 20 Feb 2012 07:28:53 PM PST      Build Host: hs20-bc2-3.build.redhat.com
Group       : System Environment/Daemons    Source RPM: rsyslog-3.22.1-7.el5.src.rpm

Comment 1 Brian Hourigan 2012-03-14 00:41:14 UTC

Here's a tcpdump capturing the truncated messages:

17:34:38.493632 IP xxx.mozilla.com.50174 >
xxx2.mozilla.com.syslog: SYSLOG daemon.debug, length: 253
0x0000: 0026 b956 5685 68b5 99bd fcb0 0800 4500 .&.VV.h.......E.
0x0010: 0119 0000 4000 4011 8f8e 0a08 4b1f 0a08 ....@.@.....K...
0x0020: 4b17 c3fe 0202 0105 ab5c 3c33 313e 4d61 K........\<31>Ma
0x0030: 7220 3133 2031 373a 3334 3a33 3820 6173 r.13.17:34:38.as
0x0040: 2d6d 6772 3120 6e72 7065 5b37 3831 315d -mgr1.nrpe[7811]
0x0050: 3a20 436f 6d6d 616e 6420 636f 6d70 6c65 :.Command.comple
17:34:38.494774 IP xxx.mozilla.com.50174 >
xxx2.mozilla.com.syslog: SYSLOG daemon.debug, length: 228
0x0000: 0026 b956 5685 68b5 99bd fcb0 0800 4500 .&.VV.h.......E.
0x0010: 0100 0000 4000 4011 8fa7 0a08 4b1f 0a08 ....@.@.....K...
0x0020: 4b17 c3fe 0202 00ec ab43 3c33 313e 4d61 K........C<31>Ma
0x0030: 7220 3133 2031 373a 3334 3a33 3820 6173 r.13.17:34:38.as
0x0040: 2d6d 6772 3120 6e72 7065 5b37 3831 315d -mgr1.nrpe[7811]
0x0050: 3a20 5265 7475 726e 2043 6f64 653a 2030 :.Return.Code:.0
17:34:38.494835 IP xxx.mozilla.com.50174 >
xxx2.mozilla.com.syslog: SYSLOG daemon.debug, length: 74
0x0000: 0026 b956 5685 68b5 99bd fcb0 0800 4500 .&.VV.h.......E.
0x0010: 0066 0000 4000 4011 9041 0a08 4b1f 0a08 .f..@.@..A..K...
0x0020: 4b17 c3fe 0202 0052 aaa9 3c33 313e 4d61 K......R..<31>Ma
0x0030: 7220 3133 2031 373a 3334 3a33 3820 6173 r.13.17:34:38.as
0x0040: 2d6d 6772 3120 6e72 7065 5b37 3831 315d -mgr1.nrpe[7811]
0x0050: 3a20 436f 6e6e 6563 7469 6f6e 2066 726f :.Connection.fro

Comment 2 Tomas Heinrich 2012-03-14 13:07:28 UTC

This bug is filed against rhel 5.9, should that be 5.8?
Based on the tcpdump output, I assume "logging over the network" means forwarding via UDP.

The tcpdump output shows "length: 253", but the output that follows ends with the "0x0050: ..." line - the first 80 bytes, which means not all the data are shown.

I've done some tests with rsyslog-3.22.1-7.el5 over UDP / TCP with tcpdump and netcat and everything seems to work.