Bug 1704957

Summary: Very poor performance of html report generation
Product: Red Hat Enterprise Linux 7 Reporter: Kevin Clevenger <kcleveng>
Component: sosAssignee: Pavel Moravec <pmoravec>
Status: CLOSED ERRATA QA Contact: Miroslav HradĂ­lek <mhradile>
Severity: medium Docs Contact: Michal Stubna <mstubna>
Priority: medium    
Version: 7.6CC: agk, bmr, ebarrera, kyoneyam, mhradile, plambri, pmoravec, sbradley, sreber, yuokada
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: sos-3.8-1.el7 Doc Type: Bug Fix
Doc Text:
.`sosreport` now generates HTML reports faster Previously, when the `sosreport` utility collected tens of thousands of files, generation of HTML report was very slow. This update provides changes to the text report code, improving the report structure and formatting. Additionally, support for reports in the JSON file format has been added. As a result, HTML reports are now generated without delay.
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-03-31 20:04:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1722515    

Comment 1 Bryn M. Reeves 2019-05-01 10:30:19 UTC
We tried to do this a number of years ago: at the time, CEE (then GSS), told us that many associates relied on the HTML report for quickly browsing a report and that disabling it by default was not acceptable. If that situation has now changed, we'd have no problem with changing the default but we do need to ensure that we have the buy-in of the wider support group before we can make any final decision.

Comment 2 Pavel Moravec 2019-05-02 08:17:02 UTC
Disabling HTML report is rather alleviating the problem than solving it - if one adds more block devices / paths, the plugin will timeout again.

Is collection of all the paths necessary? For all the block devices? Shall not we rather stop collecting some of those (if we are certain they are out of interest for troubleshooting any problem)?

And what particular paths are such numerous / causing the timeout problem?

(anyway this reminds me similar https://bugzilla.redhat.com/show_bug.cgi?id=1683904 )

Comment 6 Pavel Moravec 2019-06-30 11:04:19 UTC
Anyway, someone in the poll had an interesting idea to use JSON reports (instead of plaintext and html ones) - I raised it as

https://github.com/sosreport/sos/issues/1713

Comment 7 Pavel Moravec 2019-07-10 14:03:47 UTC
*** Bug 1727016 has been marked as a duplicate of this bug. ***

Comment 8 Pavel Moravec 2019-07-15 11:33:27 UTC
See the proposal in https://github.com/sosreport/sos/issues/1713 .

Sosreport generates html report inefficiently when some plugin collects many files (many = tens or hundreds of thousands). Rather than disabling this feature, I would vote for improving the inefficiency. Within the above PR, I propose an implementation where HTML report (for a plugin with 100k files collected) is generated within fractions of a second - basically almost as fast as text report.

I am preparing a test for 1M files collected one plugin, to see how my approach scales, but I dont expect any issues.

Please share your concern about the approach OR about the way of implementation (there can be different subjective views about the reports content or formatting, and mine proposed might not be the best one).

Comment 10 Pavel Moravec 2019-09-04 20:17:41 UTC
FYI the upstream PR was merged and backported to RHEL7.8 candidate. If you want to test it, please use below repo (or download the package from brew):



A yum repository for the build of sos-3.8-1.el7 (task 23200819) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/sos-3.8-1.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/noarch/sos-3.8-1.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/noarch/sos-3.8-1.el7.noarch.rpm

Build output will be available for the next 13 days.

Comment 15 Pavel Moravec 2019-12-27 08:20:24 UTC
*** Bug 1784234 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2020-03-31 20:04:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1127

Comment 18 Pavel Moravec 2020-05-11 13:43:55 UTC
*** Bug 1754562 has been marked as a duplicate of this bug. ***