Bug 1372884 - foreman-debug to collect whole newest (log)files instead of tailing all (log)files
Summary: foreman-debug to collect whole newest (log)files instead of tailing all (log)...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Logging
Version: 6.2.0
Hardware: All
OS: Linux
unspecified
high vote
Target Milestone: Unspecified
Assignee: Pavel Moravec
QA Contact: Perry Gagne
URL: http://projects.theforeman.org/issues...
Whiteboard:
Keywords: Triaged
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-09-03 08:23 UTC by Pavel Moravec
Modified: 2019-04-01 20:27 UTC (History)
7 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2018-02-21 16:54:37 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Foreman Issue Tracker 16680 None None None 2016-09-24 10:16 UTC
Foreman Issue Tracker 17044 None None None 2016-10-24 12:08 UTC
Foreman Issue Tracker 17069 None None None 2016-10-24 12:06 UTC
Foreman Issue Tracker 17179 None None None 2016-11-02 13:21 UTC
Red Hat Bugzilla 1274694 None None None 2018-12-27 11:31 UTC

Internal Trackers: 1274694

Description Pavel Moravec 2016-09-03 08:23:30 UTC
Description of problem:
Currently foreman-debug truncates any (log)file it collects after 5000 lines, to shrink the size of created tarball. It causes two kinds of problems:

1) even gz archives exceeding 5k lines (i.e. task export) are truncated, what makes them impossible to extract - known issue filed as bz1274694​.

2) Collected logs have "gaps" in timeline, what means we lose track of events happened that time.

Assume I want to investigate some problem with relevant logs scattered in past 2 days, and these logs are kept in files A.log, A.1.log and A.2.log, each exceeding 5k lines. Then I get tails of the logfiles (plus tails of A.9999.log from very past) but dont know what happened at the beginning of say A.1.log.

So we don't collect *some* log records in close history while we collect some logs in very past history.

What about to rather collect all newest files, until some per-directory limit is reached?

Something like that is done in sosreport [2] where it works well.

Proposed pseudo-code:

add_files() {   # $1 is path / directory, $2 is optional limit
  set limit to $2 or to $MAXLINES by default

  for i in $(sort $1 from newest to oldest); do
    collect whole $i
    decrement limit by size of $i
    if [ $limit < 0 ]; then break; fi
  done

add_files "/etc/{httpd,apache2}/conf/* no-limit
add_files "/var/log/foreman/*" 20MB
add_files /var/log/{httpd,apache2}/*error_log*
add_files /var/log/{httpd,apache2}/foreman-ssl_access_ssl.log* 20MB
..
add_files $task-export_filename no-limit
..

​[1] https://bugzilla.redhat.com/show_bug.cgi?id=1274694​
​[2] https://github.com/sosreport/sos/blob/master/sos/plugins/__init__.py#L508​


Version-Release number of selected component (if applicable):
foreman-debug-1.11.0.49-1.el7sat.noarch


How reproducible:
100%


Steps to Reproduce:
1. Generate logfiles (i.e. production.log*) each bigger than 5k lines
2. collect foreman-debug and unpack it
3. Check what lgofiles are collected


Actual results:
Currently tails of all logfiles (even 2 years old, irrelevant for current problem investigation) are collected. But head of first log-rotated logfile is missing (causing we dont know what happened yesterday morning)


Expected results:
Collected logifles completely cover past few days without a gap; older logfiles not collected.


Additional info:

Comment 1 Ivan Necas 2016-09-05 06:40:15 UTC
+1 on this approach as opposed to the current one

Comment 2 Lukas Zapletal 2016-09-05 07:07:19 UTC
TL;DR -1 does not make much sense, will make things worse actually.

This approach has some downsides as well. There are directories with 
multiple logs very often. If you happen to have two log file sets, one 
updated often and the other not, you will end up with some files ignored
completely. And we have that!

/var/log/foreman/production.log
/var/log/foreman/cron.log
/var/log/foreman/apipie_cache.log
/var/log/foreman/db_seed.log
/var/log/foreman/plugins
/var/log/foreman/db_migrate.log

With your approach, you will end up having all production.XXX.log files
but once you hit the hard limit, you won't see the others.

If a customer have let's say 2 GBs of Apache logs, you somehow need to 
do the cut at the end of the day, otherwise you'll end up with gigabytes
tarballs which is not useful.

Today, you can use -m0 option to get all (untruncated) logs. Everytime 
you encounter a problem with incomplete log file, simply ask the 
customer to do the full log.

I think 5000 lines is a sane default to get orientated in the problem. 
If you always prefer bigger tarball, ask your customers to use -m0 
option. (*)

(*) the -m0 is recent change, you can use -m 99999999 for older versions
of Satellite. The broken gzip was fixed recently and it's awaiting for 
6.2 z-stream errata

Comment 3 Lukas Zapletal 2016-09-05 07:08:41 UTC
And to be fair, I think we should change this RFE to:

Add an option to ignore files older than X months

And we should provide a sane default (let's say 6 months).

Comment 5 Pavel Moravec 2016-10-24 12:08:36 UTC
Since the upstream work affects 3 different projects (foreman, smart proxy and katello packaging), 3 upstream issues created accordingly for the 3 upstream PRs accordingly.

Comment 6 Bryan Kearney 2016-10-24 12:18:31 UTC
Upstream bug component is Uncategorized

Comment 7 Pavel Moravec 2016-11-02 13:21:04 UTC
List of upstream PRs (it's getting broader):

- https://github.com/Katello/katello-packaging/pull/297
  - (fixes #17114)
- https://github.com/theforeman/smart-proxy/pull/461
  - (fixes #17069)
- https://github.com/theforeman/foreman/pull/3881
  - (fixes #17044)
- https://github.com/theforeman/foreman-tasks/pull/209
  - (fixes #17179)
- https://github.com/theforeman/foreman-packaging/pull/1404
  - (applies #17179)

Comment 8 Bryan Kearney 2016-11-02 14:14:12 UTC
Upstream bug component is Tasks Plugin

Comment 9 Tomer Brisker 2016-12-20 09:18:38 UTC
As far as I can tell all upstream PRs have been merged, is that correct Pavel?
If so please move this bug to POST.

Comment 10 Pavel Moravec 2016-12-20 11:10:32 UTC
Right, all upstream PRs merged. Here is the complete definitive list:


- https://github.com/Katello/katello-packaging/pull/297
  - (fixes #17114)
- https://github.com/theforeman/smart-proxy/pull/461
  - (fixes #17069)
- https://github.com/theforeman/foreman/pull/3881
  - (fixes #17044)
- https://github.com/theforeman/foreman-tasks/pull/209
  - (fixes #17179)
- https://github.com/theforeman/foreman-packaging/pull/1423
  - (applies #17179)

Comment 15 pm-sat@redhat.com 2018-02-21 16:54:37 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA.
> > 
> > For information on the advisory, and where to find the updated files, follow the link below.
> > 
> > If the solution does not work for you, open a new bug report.
> > 
> > https://access.redhat.com/errata/RHSA-2018:0336


Note You need to log in before you can comment on or make changes to this bug.