Bug 1938932

Summary:	sosreport not collecting SSSD logs
Product:	Red Hat Enterprise Linux 7	Reporter:	Tomas Halman <thalman>
Component:	sos	Assignee:	Jan Jansky <jjansky>
Status:	CLOSED ERRATA	QA Contact:	Maros Kopec <makopec>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.0	CC:	agk, atikhono, bmr, cww, fkrska, jreznik, mhradile, plambri, pmoravec, ppolawsk, sbradley, theute
Target Milestone:	rc	Keywords:	OtherQA, Triaged, ZStream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	sos-3.9-5.el7_9.6	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-21 01:06:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Tomas Halman 2021-03-15 09:15:23 UTC

Description of problem: sosreport usually doesn't collect all sssd log. 

The issue is that SSSD creates individual log files for its
components. To be able to track the issue we need all of them.

With one wildcard copy set we usually get just one truncated
log file and this is not very useful for solving issues. We
need to track the request accross logs to understand the
problem. Also log file names are specific for paricular
configuration.


How reproducible:
Allways


Steps to Reproduce:
1. in /var/log/sssd create log files that are bigger than sosreport size limit
2. run sosreport
3. check the archive - log files are not collected

Actual results:
Only one (tailed) or few logs in sosreport

Expected results:
All (tailed eventually) log-files in sosreport.


Additional info:
Proposed solution is to list log-files individually, not as one copy-set.

Upstream PR: https://github.com/sosreport/sos/pull/2445

Comment 2 Pavel Moravec 2021-03-20 11:06:32 UTC

not sure if https://github.com/sosreport/sos/pull/2444 is also worth to backport for RHEL7? (cf RHEL8 bz1940502 that requires both PRs)

Comment 4 Pavel Moravec 2021-03-22 11:57:08 UTC

> Pawel created one set to add the files from sss/mc (memory cache). But those files are binary blobs (big enough to go over default limits) and tailing them doesn't make much sense.

> is binary file also "tailed"?

Yes, by default any file over 25MB is tailed on this limit. There is (a command line option and) the option sizelimit of the add_copy_spec method to change this, where sizelimit=0 means no limit. But:
- be careful to use sizelimit=0, as if the to-be-collected file can be arbitrarily huge, sosreport will generate arbitrarily huge tarball (in arbitrarily huge time..)
- the sizelimit is applied to *each* file or pattern in the list. While I think our aim is to:
  - collect whole memory cache (or no cache? is it valuable for some debugging?)
  - collect limited amount of logs

So we should have two add_copy_specs, I guess? One for just logs, one for the memory cache (or one jst for logs without memory cache?).

Is somebody able to provide filepatterns / globs for those?

Comment 5 Alexey Tikhonov 2021-03-22 13:01:34 UTC

(In reply to Pavel Moravec from comment #4)
>
> While I think our aim is to:
>   - collect whole memory cache (or no cache? is it valuable for some
> debugging?)

Right, we need either whole cache or nothing, truncates doesn't make sense.

With default settings total size of all caches is about 25mb in total.
Vast majority of users never change this setting.
I think that's ok to either add cache files one by one or use 'sizelimit=50mb'.

Comment 6 Alexey Tikhonov 2021-03-22 14:20:32 UTC

(In reply to Alexey Tikhonov from comment #5)
> 
> With default settings total size of all caches is about 25mb in total.

JFTR: I mean SSSD settings here.

Comment 7 Paweł Poławski 2021-03-24 15:11:46 UTC

I prepared PR which adds cache files one by one:
https://github.com/sosreport/sos/pull/2462

Comment 8 Paweł Poławski 2021-03-26 12:09:32 UTC

If it is possible it would be good to have all 3 PRs included in RHEL7.
In the worst case scenario must-have one is: https://github.com/sosreport/sos/pull/2445

Comment 9 Paweł Poławski 2021-04-06 08:58:25 UTC

Hi Pavel,

At the moment all 3 of SSSD related PRs has been merged upstream [1][2][3].

[1] https://github.com/sosreport/sos/pull/2445
[2] https://github.com/sosreport/sos/pull/2444
[3] https://github.com/sosreport/sos/pull/2462

Comment 10 Jan Jansky 2021-04-06 10:15:57 UTC

Merging to legacy-3.9 branch

https://github.com/sosreport/sos/pull/2476

Depending on merge date it should be possible to release to next z-stream batch (Tue 2021-06-08)

Comment 11 Jan Jansky 2021-04-08 12:41:09 UTC

Pawel i am preparing all 3 PRs into next batch z-stream, will you be able to test applied changes and role as OtherQA when candidate package will be available or do you know who may serve as OtherQA?

Comment 12 Paweł Poławski 2021-04-12 10:30:17 UTC

Hi Jan, I love to help with testing and verification. If this will be about downloading your z-stream build candidate package and manual verification that it perform as planned?

Comment 13 Jan Jansky 2021-04-12 11:20:56 UTC

Hi Pawel, that is exactly what we want, i will post here link for candidate when it will be ready.

Comment 24 errata-xmlrpc 2021-07-21 01:06:48 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (sos bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:2804