Bug 1764472
| Summary: | remove --all-logs from RHOSP preset to prevent high memory usage of sosreport | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Takashi Kajinami <tkajinam> |
| Component: | sos | Assignee: | Jan Jansky <jjansky> |
| Status: | CLOSED ERRATA | QA Contact: | Miroslav Hradílek <mhradile> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.6 | CC: | agk, bmr, bshephar, cww, dhill, jhunsaker, jjansky, lyarwood, mhradile, plambri, rlondhe, sbradley |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | sos-3.9-2.el7 | Doc Type: | Release Note |
| Doc Text: |
On OpenStack deployments, sosreport does not further collect all logs by default.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-09-29 20:55:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1784466 | ||
|
Description
Takashi Kajinami
2019-10-23 06:36:15 UTC
Technical background: (*) --all-logs option is added due to rhosp preset loaded, that contains that option: https://github.com/sosreport/sos/blob/master/sos/policies/redhat.py#L217-L218 (*) this causes journalctl since current boot - without a size limit so huge here - is collected: https://github.com/sosreport/sos/blob/master/sos/plugins/logs.py#L53 (*) sadly, whole that content is kept in memory (known issue we have on todo list to improve) I would separate two issues here: - sosreport should not keep whole command output in memory - should sosreport on RHOSP collect all logs by default? Is there such a requirement? Is the request to drop --all-logs driven just by the high memory requirements, or by redundancy of those collected logs? Jake, Lee, could you comment please? Lee, please see previous comment for the needinfo. (In reply to Pavel Moravec from comment #2) > Technical background: > (*) --all-logs option is added due to rhosp preset loaded, that contains > that option: > https://github.com/sosreport/sos/blob/master/sos/policies/redhat.py#L217-L218 > (*) this causes journalctl since current boot - without a size limit so huge > here - is collected: > https://github.com/sosreport/sos/blob/master/sos/plugins/logs.py#L53 > (*) sadly, whole that content is kept in memory (known issue we have on todo > list to improve) > > > I would separate two issues here: > - sosreport should not keep whole command output in memory > - should sosreport on RHOSP collect all logs by default? Is there such a > requirement? > > Is the request to drop --all-logs driven just by the high memory > requirements, or by redundancy of those collected logs? > > Jake, Lee, could you comment please? I'm okay dropping this for OSP *if* we clearly advertise this to CEE (and in turn our customers). Generally speaking I'm not a fan of --all-logs being a default option, so I'm good with removing this from the RHOSP preset. I agree with Lee though, we'll just need to make sure this is properly conveyed to the support teams. Hi all,
Thanks for working on this topic, and sorry for my late update.
> Is the request to drop --all-logs driven just by the high memory requirements, or by redundancy of those collected logs?
AFAIK sosreport didn't add --all-logs in old versions, and the change to enable it
was introduced recently.
Because I didn't expect that most of people expect to have --all-logs by default.
I think we can start with just disabling it.
On the other hand, we still have risk to cause huge memory consumption
when we explicitly set --all-logs when we try to gather logs.
I'll file another bz to improve memory usage to gather journal in sosreport,
so that we can continue to discuss about this topic.
Please let me know if you have any concerns about filing an independent bz.
Thank you,
Takashi
(In reply to Takashi Kajinami from comment #6) > On the other hand, we still have risk to cause huge memory consumption > when we explicitly set --all-logs when we try to gather logs. > I'll file another bz to improve memory usage to gather journal in sosreport, > so that we can continue to discuss about this topic. > > Please let me know if you have any concerns about filing an independent bz. There is already a BZ for this: https://bugzilla.redhat.com/show_bug.cgi?id=1726023 And as I stated we are aware of this problem: > (*) sadly, whole that content is kept in memory (known issue we have on todo list to improve) but it will be tricky to fix this (since e.g. we parse that output for potentially-mutliline secret data to scrub). --all-logs is removed from RHOSP preset in upstream and that will reach RHEL7.9 whole cmd output kept in memory - we are aware of the problem (see sos #1506), no idea how to efficiently fix, probably wont reach RHEL7. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (sos bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:4034 |