Description of problem: Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. It causes two kinds of problems: 1) even gz archives exceeding 5k lines (i.e. task export) are truncated, what makes them impossible to extract - known issue filed as bz1274694. 2) Collected logs have "gaps" in timeline, what means we lose track of events happened that time. Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log. So we don't collect *some* log records in close history while we collect some logs in very past history. What about to rather collect all newest files, until some per-directory limit is reached? Something like that is done in sosreport [2] where it works well. Proposed pseudo-code: add_files() { # $1 is path / directory, $2 is optional limit set limit to $2 or to $MAXLINES by default for i in $(sort $1 from newest to oldest); do collect whole $i decrement limit by size of $i if [ $limit < 0 ]; then break; fi done add_files "/etc/{httpd,apache2}/conf/* no-limit add_files "/var/log/foreman/*" 20MB add_files /var/log/{httpd,apache2}/*error_log* add_files /var/log/{httpd,apache2}/foreman-ssl_access_ssl.log* 20MB .. add_files $task-export_filename no-limit .. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1274694 [2] https://github.com/sosreport/sos/blob/master/sos/plugins/__init__.py#L508 Version-Release number of selected component (if applicable): foreman-debug-1.11.0.49-1.el7sat.noarch How reproducible: 100% Steps to Reproduce: 1. Generate logfiles (i.e. production.log*) each bigger than 5k lines 2. collect foreman-debug and unpack it 3. Check what lgofiles are collected Actual results: Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning) Expected results: Collected logifles completely cover past few days without a gap; older logfiles not collected. Additional info:
+1 on this approach as opposed to the current one
TL;DR -1 does not make much sense, will make things worse actually. This approach has some downsides as well. There are directories with multiple logs very often. If you happen to have two log file sets, one updated often and the other not, you will end up with some files ignored completely. And we have that! /var/log/foreman/production.log /var/log/foreman/cron.log /var/log/foreman/apipie_cache.log /var/log/foreman/db_seed.log /var/log/foreman/plugins /var/log/foreman/db_migrate.log With your approach, you will end up having all production.XXX.log files but once you hit the hard limit, you won't see the others. If a customer have let's say 2 GBs of Apache logs, you somehow need to do the cut at the end of the day, otherwise you'll end up with gigabytes tarballs which is not useful. Today, you can use -m0 option to get all (untruncated) logs. Everytime you encounter a problem with incomplete log file, simply ask the customer to do the full log. I think 5000 lines is a sane default to get orientated in the problem. If you always prefer bigger tarball, ask your customers to use -m0 option. (*) (*) the -m0 is recent change, you can use -m 99999999 for older versions of Satellite. The broken gzip was fixed recently and it's awaiting for 6.2 z-stream errata
And to be fair, I think we should change this RFE to: Add an option to ignore files older than X months And we should provide a sane default (let's say 6 months).
Since the upstream work affects 3 different projects (foreman, smart proxy and katello packaging), 3 upstream issues created accordingly for the 3 upstream PRs accordingly.
Upstream bug component is Uncategorized
List of upstream PRs (it's getting broader): - https://github.com/Katello/katello-packaging/pull/297 - (fixes #17114) - https://github.com/theforeman/smart-proxy/pull/461 - (fixes #17069) - https://github.com/theforeman/foreman/pull/3881 - (fixes #17044) - https://github.com/theforeman/foreman-tasks/pull/209 - (fixes #17179) - https://github.com/theforeman/foreman-packaging/pull/1404 - (applies #17179)
Upstream bug component is Tasks Plugin
As far as I can tell all upstream PRs have been merged, is that correct Pavel? If so please move this bug to POST.
Right, all upstream PRs merged. Here is the complete definitive list: - https://github.com/Katello/katello-packaging/pull/297 - (fixes #17114) - https://github.com/theforeman/smart-proxy/pull/461 - (fixes #17069) - https://github.com/theforeman/foreman/pull/3881 - (fixes #17044) - https://github.com/theforeman/foreman-tasks/pull/209 - (fixes #17179) - https://github.com/theforeman/foreman-packaging/pull/1423 - (applies #17179)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. > > > > For information on the advisory, and where to find the updated files, follow the link below. > > > > If the solution does not work for you, open a new bug report. > > > > https://access.redhat.com/errata/RHSA-2018:0336