Bug 1704957 - Very poor performance of html report generation
Summary: Very poor performance of html report generation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sos
Version: 7.6
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Pavel Moravec
QA Contact: Miroslav Hradílek
mstubna
URL:
Whiteboard:
: 1727016 1754562 1784234 (view as bug list)
Depends On:
Blocks: 1722515
TreeView+ depends on / blocked
 
Reported: 2019-04-30 21:42 UTC by Kevin Clevenger
Modified: 2020-05-11 13:43 UTC (History)
10 users (show)

Fixed In Version: sos-3.8-1.el7
Doc Type: Bug Fix
Doc Text:
.`sosreport` now generates HTML reports faster Previously, when the `sosreport` utility collected tens of thousands of files, generation of HTML report was very slow. This update provides changes to the text report code, improving the report structure and formatting. Additionally, support for reports in the JSON file format has been added. As a result, HTML reports are now generated without delay.
Clone Of:
Environment:
Last Closed: 2020-03-31 20:04:09 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github sosreport sos issues 1713 'None' closed [sosreport] format of reports (plain/html/json?) 2020-07-06 08:47:06 UTC
Red Hat Knowledge Base (Solution) 4265251 None None sosreport is taking a long time on OpenShift Container Platform - Node 2019-07-10 14:03:46 UTC
Red Hat Product Errata RHEA-2020:1127 None None None 2020-03-31 20:04:59 UTC

Comment 1 Bryn M. Reeves 2019-05-01 10:30:19 UTC
We tried to do this a number of years ago: at the time, CEE (then GSS), told us that many associates relied on the HTML report for quickly browsing a report and that disabling it by default was not acceptable. If that situation has now changed, we'd have no problem with changing the default but we do need to ensure that we have the buy-in of the wider support group before we can make any final decision.

Comment 2 Pavel Moravec 2019-05-02 08:17:02 UTC
Disabling HTML report is rather alleviating the problem than solving it - if one adds more block devices / paths, the plugin will timeout again.

Is collection of all the paths necessary? For all the block devices? Shall not we rather stop collecting some of those (if we are certain they are out of interest for troubleshooting any problem)?

And what particular paths are such numerous / causing the timeout problem?

(anyway this reminds me similar https://bugzilla.redhat.com/show_bug.cgi?id=1683904 )

Comment 6 Pavel Moravec 2019-06-30 11:04:19 UTC
Anyway, someone in the poll had an interesting idea to use JSON reports (instead of plaintext and html ones) - I raised it as

https://github.com/sosreport/sos/issues/1713

Comment 7 Pavel Moravec 2019-07-10 14:03:47 UTC
*** Bug 1727016 has been marked as a duplicate of this bug. ***

Comment 8 Pavel Moravec 2019-07-15 11:33:27 UTC
See the proposal in https://github.com/sosreport/sos/issues/1713 .

Sosreport generates html report inefficiently when some plugin collects many files (many = tens or hundreds of thousands). Rather than disabling this feature, I would vote for improving the inefficiency. Within the above PR, I propose an implementation where HTML report (for a plugin with 100k files collected) is generated within fractions of a second - basically almost as fast as text report.

I am preparing a test for 1M files collected one plugin, to see how my approach scales, but I dont expect any issues.

Please share your concern about the approach OR about the way of implementation (there can be different subjective views about the reports content or formatting, and mine proposed might not be the best one).

Comment 10 Pavel Moravec 2019-09-04 20:17:41 UTC
FYI the upstream PR was merged and backported to RHEL7.8 candidate. If you want to test it, please use below repo (or download the package from brew):



A yum repository for the build of sos-3.8-1.el7 (task 23200819) is available at:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/

You can install the rpms locally by putting this .repo file in your /etc/yum.repos.d/ directory:

http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/sos-3.8-1.el7.repo

RPMs and build logs can be found in the following locations:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/noarch/

The full list of available rpms is:
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/noarch/sos-3.8-1.el7.src.rpm
http://brew-task-repos.usersys.redhat.com/repos/official/sos/3.8/1.el7/noarch/sos-3.8-1.el7.noarch.rpm

Build output will be available for the next 13 days.

Comment 15 Pavel Moravec 2019-12-27 08:20:24 UTC
*** Bug 1784234 has been marked as a duplicate of this bug. ***

Comment 17 errata-xmlrpc 2020-03-31 20:04:09 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:1127

Comment 18 Pavel Moravec 2020-05-11 13:43:55 UTC
*** Bug 1754562 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.