Bug 1372884

Summary: foreman-debug to collect whole newest (log)files instead of tailing all (log)files
Product: Red Hat Satellite Reporter: Pavel Moravec <pmoravec>
Component: LoggingAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Perry Gagne <pgagne>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: aruzicka, cwelton, inecas, jcallaha, oshtaier, pdwyer, pmoravec
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: All   
OS: Linux   
URL: http://projects.theforeman.org/issues/16680
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-02-21 16:54:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Pavel Moravec 2016-09-03 08:23:30 UTC 
Description of problem:
Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. It causes two kinds of problems:

1) even gz archives exceeding 5k lines (i.e. task export) are truncated, what makes them impossible to extract - known issue filed as bz1274694​.

2) Collected logs have "gaps" in timeline, what means we lose track of events happened that time.

Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log.

So we don't collect *some* log records in close history while we collect some logs in very past history.

What about to rather collect all newest files, until some per-directory limit is reached?

Something like that is done in sosreport [2] where it works well.

Proposed pseudo-code:

add_files() {   # $1 is path / directory, $2 is optional limit
  set limit to $2 or to $MAXLINES by default

  for i in $(sort $1 from newest to oldest); do
    collect whole $i
    decrement limit by size of $i
    if [ $limit < 0 ]; then break; fi
  done

add_files "/etc/{httpd,apache2}/conf/* no-limit
add_files "/var/log/foreman/*" 20MB
add_files /var/log/{httpd,apache2}/*error_log*
add_files /var/log/{httpd,apache2}/foreman-ssl_access_ssl.log* 20MB
..
add_files $task-export_filename no-limit
..

​[1] https://bugzilla.redhat.com/show_bug.cgi?id=1274694​
​[2] https://github.com/sosreport/sos/blob/master/sos/plugins/__init__.py#L508​


Version-Release number of selected component (if applicable):
foreman-debug-1.11.0.49-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Generate logfiles (i.e. production.log*) each bigger than 5k lines
2. collect foreman-debug and unpack it
3. Check what lgofiles are collected


Actual results:
Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning)


Expected results:
Collected logifles completely cover past few days without a gap; older logfiles not collected.


Additional info:
Comment 1 Ivan Necas 2016-09-05 06:40:15 UTC 
+1 on this approach as opposed to the current one
Comment 2 Lukas Zapletal 2016-09-05 07:07:19 UTC 
TL;DR -1 does not make much sense, will make things worse actually.

This approach has some downsides as well. There are directories with 
multiple logs very often. If you happen to have two log file sets, one 
updated often and the other not, you will end up with some files ignored
completely. And we have that!

/var/log/foreman/production.log
/var/log/foreman/cron.log
/var/log/foreman/apipie_cache.log
/var/log/foreman/db_seed.log
/var/log/foreman/plugins
/var/log/foreman/db_migrate.log

With your approach, you will end up having all production.XXX.log files
but once you hit the hard limit, you won't see the others.

If a customer have let's say 2 GBs of Apache logs, you somehow need to 
do the cut at the end of the day, otherwise you'll end up with gigabytes
tarballs which is not useful.

Today, you can use -m0 option to get all (untruncated) logs. Everytime 
you encounter a problem with incomplete log file, simply ask the 
customer to do the full log.

I think 5000 lines is a sane default to get orientated in the problem. 
If you always prefer bigger tarball, ask your customers to use -m0 
option. (*)

(*) the -m0 is recent change, you can use -m 99999999 for older versions
of Satellite. The broken gzip was fixed recently and it's awaiting for 
6.2 z-stream errata
Comment 3 Lukas Zapletal 2016-09-05 07:08:41 UTC 
And to be fair, I think we should change this RFE to:

Add an option to ignore files older than X months

And we should provide a sane default (let's say 6 months).
Comment 5 Pavel Moravec 2016-10-24 12:08:36 UTC 
Since the upstream work affects 3 different projects (foreman, smart proxy and katello packaging), 3 upstream issues created accordingly for the 3 upstream PRs accordingly.
Comment 6 Bryan Kearney 2016-10-24 12:18:31 UTC 
Upstream bug component is Uncategorized
Comment 7 Pavel Moravec 2016-11-02 13:21:04 UTC 
List of upstream PRs (it's getting broader):

- https://github.com/Katello/katello-packaging/pull/297
  - (fixes #17114)
- https://github.com/theforeman/smart-proxy/pull/461
  - (fixes #17069)
- https://github.com/theforeman/foreman/pull/3881
  - (fixes #17044)
- https://github.com/theforeman/foreman-tasks/pull/209
  - (fixes #17179)
- https://github.com/theforeman/foreman-packaging/pull/1404
  - (applies #17179)
Comment 8 Bryan Kearney 2016-11-02 14:14:12 UTC 
Upstream bug component is Tasks Plugin
Comment 9 Tomer Brisker 2016-12-20 09:18:38 UTC 
As far as I can tell all upstream PRs have been merged, is that correct Pavel?
If so please move this bug to POST.
Comment 10 Pavel Moravec 2016-12-20 11:10:32 UTC 
Right, all upstream PRs merged. Here is the complete definitive list:


- https://github.com/Katello/katello-packaging/pull/297
  - (fixes #17114)
- https://github.com/theforeman/smart-proxy/pull/461
  - (fixes #17069)
- https://github.com/theforeman/foreman/pull/3881
  - (fixes #17044)
- https://github.com/theforeman/foreman-tasks/pull/209
  - (fixes #17179)
- https://github.com/theforeman/foreman-packaging/pull/1423
  - (applies #17179)
Comment 15 Satellite Program 2018-02-21 16:54:37 UTC 
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336