RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1647935 - sosreport shouldnt store whole cmdoutput in memory
Summary: sosreport shouldnt store whole cmdoutput in memory
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: sos
Version: 8.4
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: Upgrades and Supportability
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-11-08 15:08 UTC by Udi Kalifon
Modified: 2021-03-15 07:31 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-03-15 07:31:17 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Output from sosreport (10.76 KB, text/plain)
2018-11-22 06:49 UTC, Udi Kalifon
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github sosreport sos issues 1506 0 'None' open Large command outputs should not be completely stored in memory 2021-02-04 09:16:13 UTC

Description Udi Kalifon 2018-11-08 15:08:56 UTC
Description of problem:
I am running sosreport on my openstack undercloud (OSP14 RHEL7.6). When it finishes collecting all the logs it doesn't tell me any more where to find the result.


Version-Release number of selected component (if applicable):
sos-3.6-11.el7_6.noarch


How reproducible:
100%


Steps to Reproduce:
1. su to root
2. run sosreport


Actual results:
When sosreport finishes - it doesn't tell you where it saved the file to, and we can't find it anywhere.

Comment 2 Bryn M. Reeves 2018-11-08 15:32:18 UTC
Please post the output that you do get, with -vv (extra verbosity) and --debug (report exceptions to the terminal). There's not a lot to go on here otherwise.

Comment 3 Udi Kalifon 2018-11-22 06:49:14 UTC
Created attachment 1507880 [details]
Output from sosreport

Attached is the output from the command

Comment 4 Bryn M. Reeves 2018-11-22 10:57:04 UTC
Please run the command with extra verbosity as requested in comment #2. The output in comment #3 has only standard verbosity and is not useful for debugging.

Also, since you are running with --debug as requested, when the Python traceback is shown you will need to exit the debugger by entering <CTRL>-d (alternately, since you've provided the --debug output already, you can just drop that option from the command - the exceptions are written to sos_logs/ anyway). If you do run without --debug then please keep the report directory (at least sos_logs) in case we need any further information from it.

Comment 5 Bryn M. Reeves 2018-11-22 11:06:16 UTC
For reference you should expect to see output like this (preamble text removed for brevity):

# sosreport -vv --batch -o system
set sysroot to '/' (default)

sosreport (version 3.6)

[...]

 Setting up archive ...
[archive:TarFileArchive] initialised empty FileCacheArchive at '/var/tmp/sos.ShISFI/sosreport-localhost-2018-11-22-xunzzvy'
[sos.sosreport:setup] executing 'sosreport -vv --batch -o system'
[sos.sosreport:setup] using '' preset defaults (--threads 4)
[sos.sosreport:setup] effective options now: --batch --onlyplugins system --threads 4 -vv
 Setting up plugins ...
[plugin:system] adding forbidden path '/proc/sys/net/ipv4/route/flush'
[plugin:system] adding forbidden path '/proc/sys/net/ipv6/route/flush'
[plugin:system] adding forbidden path '/proc/sys/net/ipv6/neigh/*/retrans_time'
[plugin:system] adding forbidden path '/proc/sys/net/ipv6/neigh/*/base_reachable_time'
 Running plugins. Please wait ...

  Starting 1/1   system          [Running: system]
[plugin:system] collecting path '/proc/sys'
[plugin:system] recursively adding 'abi' from '/proc/sys'
[plugin:system] recursively adding 'vsyscall32' from '/proc/sys/abi'
[plugin:system] copying path '/proc/sys/abi/vsyscall32' to archive:'/proc/sys/abi/vsyscall32'
[plugin:system] recursively adding 'crypto' from '/proc/sys'
[plugin:system] recursively adding 'fips_enabled' from '/proc/sys/crypto'
[plugin:system] copying path '/proc/sys/crypto/fips_enabled' to archive:'/proc/sys/crypto/fips_enabled'
[plugin:system] recursively adding 'debug' from '/proc/sys'
[plugin:system] recursively adding 'exception-trace' from '/proc/sys/debug'
[plugin:system] copying path '/proc/sys/debug/exception-trace' to archive:'/proc/sys/debug/exception-trace'
[plugin:system] recursively adding 'kprobes-optimization' from '/proc/sys/debug'
[plugin:system] copying path '/proc/sys/debug/kprobes-optimization' to archive:'/proc/sys/debug/kprobes-optimization'
[plugin:system] recursively adding 'dev' from '/proc/sys'
[plugin:system] recursively adding 'cdrom' from '/proc/sys/dev'
[plugin:system] recursively adding 'autoclose' from '/proc/sys/dev/cdrom'
[plugin:system] copying path '/proc/sys/dev/cdrom/autoclose' to archive:'/proc/sys/dev/cdrom/autoclose'
[plugin:system] recursiv

(Do _not_ include "-o system" in your run - this limits the execution to just the system plugin which is unaffected here)

Comment 10 Pavel Moravec 2018-12-10 09:06:39 UTC
Observed the reproducer and the cause is:

- the system where sosreport was run is OpenStack director (undercloud)
- so rhosp preset was used
- this means sosreport had automatically enabled options --all-logs -k process.lsof=off
- that caused "journalctl --no-pager --all --boot --output_verbose" command output was collected, additionally to default non-verbose (and few times smaller) variant
- that command output is 2GB huge
- sosreport keeps the whole output in memory, which requires few times more RSS at some time
- the system didnt have so much free memory, thus OOM killer terminated sosreport execution (visible via /var/log/messages)

Workaround: prevent enabling --all-logs, i.e. run:

sosreport --preset rhel -k process.lsof=off


Anyway, we again hit the well known problem of keeping whole cmd output in memory - sosreport should process a cmd output by chunks or offload it to the disk.

https://github.com/sosreport/sos/issues/1506 raised for this.


I would like to have it fixed in 7.7, though it depends on devel capacity (nontrivial change with regression potential).

Comment 11 Pavel Moravec 2019-03-29 11:33:12 UTC
Scope of 7.7 closed, rescheduling for potential inclusion in 7.8.

Comment 12 Pavel Moravec 2020-11-04 21:15:56 UTC
This long pain in sos hasnt been fixed yet. I still have it on my radar but no idea of smart small fix and no time for solid complex fix.

Rescheduling to RHEL8 for potential inclusion in 8.5 or later.

Comment 15 RHEL Program Management 2021-03-15 07:31:17 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.


Note You need to log in before you can comment on or make changes to this bug.