Bug 1647935
| Summary: | sosreport shouldnt store whole cmdoutput in memory | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Udi Kalifon <ukalifon> | ||||
| Component: | sos | Assignee: | Pavel Moravec <pmoravec> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | Upgrades and Supportability <upgrades-and-supportability> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 8.4 | CC: | agk, bmr, jjansky, plambri, sbradley, theute, ukalifon | ||||
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-03-15 07:31:17 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Udi Kalifon
2018-11-08 15:08:56 UTC
Please post the output that you do get, with -vv (extra verbosity) and --debug (report exceptions to the terminal). There's not a lot to go on here otherwise. Created attachment 1507880 [details]
Output from sosreport
Attached is the output from the command
Please run the command with extra verbosity as requested in comment #2. The output in comment #3 has only standard verbosity and is not useful for debugging. Also, since you are running with --debug as requested, when the Python traceback is shown you will need to exit the debugger by entering <CTRL>-d (alternately, since you've provided the --debug output already, you can just drop that option from the command - the exceptions are written to sos_logs/ anyway). If you do run without --debug then please keep the report directory (at least sos_logs) in case we need any further information from it. For reference you should expect to see output like this (preamble text removed for brevity): # sosreport -vv --batch -o system set sysroot to '/' (default) sosreport (version 3.6) [...] Setting up archive ... [archive:TarFileArchive] initialised empty FileCacheArchive at '/var/tmp/sos.ShISFI/sosreport-localhost-2018-11-22-xunzzvy' [sos.sosreport:setup] executing 'sosreport -vv --batch -o system' [sos.sosreport:setup] using '' preset defaults (--threads 4) [sos.sosreport:setup] effective options now: --batch --onlyplugins system --threads 4 -vv Setting up plugins ... [plugin:system] adding forbidden path '/proc/sys/net/ipv4/route/flush' [plugin:system] adding forbidden path '/proc/sys/net/ipv6/route/flush' [plugin:system] adding forbidden path '/proc/sys/net/ipv6/neigh/*/retrans_time' [plugin:system] adding forbidden path '/proc/sys/net/ipv6/neigh/*/base_reachable_time' Running plugins. Please wait ... Starting 1/1 system [Running: system] [plugin:system] collecting path '/proc/sys' [plugin:system] recursively adding 'abi' from '/proc/sys' [plugin:system] recursively adding 'vsyscall32' from '/proc/sys/abi' [plugin:system] copying path '/proc/sys/abi/vsyscall32' to archive:'/proc/sys/abi/vsyscall32' [plugin:system] recursively adding 'crypto' from '/proc/sys' [plugin:system] recursively adding 'fips_enabled' from '/proc/sys/crypto' [plugin:system] copying path '/proc/sys/crypto/fips_enabled' to archive:'/proc/sys/crypto/fips_enabled' [plugin:system] recursively adding 'debug' from '/proc/sys' [plugin:system] recursively adding 'exception-trace' from '/proc/sys/debug' [plugin:system] copying path '/proc/sys/debug/exception-trace' to archive:'/proc/sys/debug/exception-trace' [plugin:system] recursively adding 'kprobes-optimization' from '/proc/sys/debug' [plugin:system] copying path '/proc/sys/debug/kprobes-optimization' to archive:'/proc/sys/debug/kprobes-optimization' [plugin:system] recursively adding 'dev' from '/proc/sys' [plugin:system] recursively adding 'cdrom' from '/proc/sys/dev' [plugin:system] recursively adding 'autoclose' from '/proc/sys/dev/cdrom' [plugin:system] copying path '/proc/sys/dev/cdrom/autoclose' to archive:'/proc/sys/dev/cdrom/autoclose' [plugin:system] recursiv (Do _not_ include "-o system" in your run - this limits the execution to just the system plugin which is unaffected here) Observed the reproducer and the cause is: - the system where sosreport was run is OpenStack director (undercloud) - so rhosp preset was used - this means sosreport had automatically enabled options --all-logs -k process.lsof=off - that caused "journalctl --no-pager --all --boot --output_verbose" command output was collected, additionally to default non-verbose (and few times smaller) variant - that command output is 2GB huge - sosreport keeps the whole output in memory, which requires few times more RSS at some time - the system didnt have so much free memory, thus OOM killer terminated sosreport execution (visible via /var/log/messages) Workaround: prevent enabling --all-logs, i.e. run: sosreport --preset rhel -k process.lsof=off Anyway, we again hit the well known problem of keeping whole cmd output in memory - sosreport should process a cmd output by chunks or offload it to the disk. https://github.com/sosreport/sos/issues/1506 raised for this. I would like to have it fixed in 7.7, though it depends on devel capacity (nontrivial change with regression potential). Scope of 7.7 closed, rescheduling for potential inclusion in 7.8. This long pain in sos hasnt been fixed yet. I still have it on my radar but no idea of smart small fix and no time for solid complex fix. Rescheduling to RHEL8 for potential inclusion in 8.5 or later. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |