Bug 1323916

Summary:	logger unnecessarily splits messages sent via stdin into 1024 byte chunks
Product:	Red Hat Enterprise Linux 7	Reporter:	Ryan Sawhill <rsawhill>
Component:	util-linux	Assignee:	Karel Zak <kzak>
Status:	CLOSED ERRATA	QA Contact:	Radka Brychtova <rskvaril>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	7.2	CC:	bblaskov, gentoo, gerald.prock, rsawhill, rskvaril
Target Milestone:	rc
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	util-linux-2.23.2-36.el7	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-01 21:41:11 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1203710, 1380361

Description Ryan Sawhill 2016-04-05 04:58:09 UTC

Description of problem:

  logger splits stdin into 1 KiB chunks. This is especially odd because logger doesn't seem to do any splitting when log messages are passed via argv.

  Why does this matter?
  logger is regularly used by applications as a cheap way to get logs to a local or remote (r)?syslog server. One of the most common users of this is Apache httpd, where httpd spawns a single logger command when started and uses that to send logs. In this situation, it's not at all uncommon for log entries to be larger than 1k. If you google it, you can numerous examples of httpd users complaining about this limitation. 

 I assume the 1024-byte limit is a hold-over from the syslog rfc (http://www.faqs.org/rfcs/rfc3164.html) but in 2016 this is just comical. rsyslog and journald have no such limit.

Version-Release number of selected component (if applicable):

  util-linux-2.23.2-26.el7

Steps to Reproduce:

  1. Use `logger $(printf 'a%.0s' {1..8192})` and notice the journal and rsyslog end up with a single 8k message.
  2. Use `printf 'b%.0s' {1..1025} | logger` and the journal/rsyslog output make it clear that logger splits stdin into 1k-chunks -- you end up with 2 separate log entries.

Expected results:

  I'd like to see logger treat stdin in the same way it treats argv -- i.e., practically unlimited message length.
 
Additional info:

  FYI rsyslog v7 in RHEL 7 is supposed to have a default $MaxMessageSize setting of 2k as seen here: http://www.rsyslog.com/doc/v7-stable/configuration/global/index.html; however, despite our rsyslog-doc package saying the same thing, we've clearly raised the default far above 2k -- probably in response to some customer request. (I didn't check the source.) All the more reason to fix this issue with logger.

Comment 1 Karel Zak 2016-04-05 08:59:37 UTC

The current upstream version provides "--size <bytes>" command line option to overwrite the default (still 1024). Maybe we can backport the option to RHEL7.

I'm not sure if we want to change the default due to backward compatibility.

Comment 2 Ryan Sawhill 2016-04-05 15:06:43 UTC

If we can backport that --size option that upstream already has, that would be awesome and good enough for me; however, I have to say that "backward compatibility" doesn't really make sense to me given that logger accepts virtually unlimited messages when passed on the cmdline as argument(s). ... Which is why I filed this as a bug and not an RFE -- it's inconsistent.

In any case, thanks so much for the prompt response Karel!

Comment 3 Ryan Sawhill 2016-04-12 04:24:48 UTC

Well it took me a few hours to write a logger clone in python that does virtually everything RHEL7's logger cmd does. (I didn't implement --stderr and it only reads input from stdin.)

I have no love for perl, but I want to see how the performance compares, so I spent a few more (half dozen?) much more painful hours doing the same in perl. *shudder*

Neither of these logger clones suffer from the stdin message-splitting limitation of the stock logger command. Both are linked from:
https://access.redhat.com/solutions/2220491

Hopefully we can get an officially-supported method sooner rather than later.

Comment 4 Ryan Sawhill 2016-04-13 17:51:06 UTC

(In reply to Ryan Sawhill from comment #3)
> Well it took me a few hours to write a logger clone in python that does
> virtually everything RHEL7's logger cmd does. (I didn't implement --stderr
> and it only reads input from stdin.)

I have to correct this for the record: I spent a couple more hours on it and my logger.py now accepts logs via cmdline args as well as -f/--file. Also added features from upstream: --id for custom IDs and -e/--skip-empty for fileinput. Finally, added a new --ppid option.

Comment 7 Florian Crouzat 2016-12-30 10:10:07 UTC

So any chance to get the upstream version of logger which supports the --size option backported into RHEL7 any time or even better, any time soon ?

I can reproduce the issue with rhel7.3 and util-linux-2.23.2-33.el7.x86_64

If not, Ryan, any chance you published your python script somewhere so I can try it ? I definitely need a fix/workaround to this 1024-byte limit  ...

Comment 8 Ryan Sawhill 2016-12-30 17:05:35 UTC

(In reply to Florian Crouzat from comment #7)
...
> If not, Ryan, any chance you published your python script somewhere so I can
> try it ? I definitely need a fix/workaround to this 1024-byte limit  ...

Hi Florian [and anyone else with the same question]. The perl & python scripts are linked from the KCS solution in comment #3 -- direct link:

  https://github.com/ryran/loggerclones

Of course the obvious disclaimer applies: I do not offer these in my capacity as a Red Hat employee; treat them as 3rd-party/unsupported/no-warranty -- any further queries about them should be directed to that github repo. :)

Comment 9 Karel Zak 2017-01-03 10:13:04 UTC

Frankly, if you're fine with script as logger replacement than you can probably use logger binary from Fedora where is no such restriction :-)  (or rebuild it from upstream github)

Anyway, from my point of view the --size backport seems fine, going to suggest it for 7.4.

Comment 10 Florian Crouzat 2017-01-03 10:35:22 UTC

(In reply to Karel Zak from comment #9)
> Frankly, if you're fine with script as logger replacement than you can
> probably use logger binary from Fedora where is no such restriction :-)  (or
> rebuild it from upstream github)
> 
> Anyway, from my point of view the --size backport seems fine, going to
> suggest it for 7.4.

That would be great. I'll stay tuned. Will this BZ updates inform me weither or not it will be shipped in 7.4 ?

(In reply to Ryan Sawhill from comment #8)
> https://github.com/ryran/loggerclones

Thanks, I'll wait for a conclusion on the above topic (7.4) before trying your scripts as I'd rather remain in the stable branch, but I'll have a look at it in case the option is not backported.

Comment 15 errata-xmlrpc 2017-08-01 21:41:11 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2186