Bug 1124462 - do not display all ftp files on web UI by default - manager page is slow and/or may hang or crash browser if there's a lot of ftp files
Summary: do not display all ftp files on web UI by default - manager page is slow and/...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: retrace-server
Version: epel7
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
Assignee: Matej Marušák
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-29 14:27 UTC by Dave Wysochanski
Modified: 2017-04-18 21:18 UTC (History)
5 users (show)

Fixed In Version: retrace-server-1.16-1.fc23 retrace-server-1.17.0-1.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-18 21:18:35 UTC


Attachments (Terms of Use)

Description Dave Wysochanski 2014-07-29 14:27:18 UTC
Description of problem:
Recently we had a customer upload a vmcore split into very small chunks (256KB) and there were thousands of files.  From this point on, anyone else using the web UI got either very slow or crashed browser when they tried to load it.

I think it may be better to change the default behavior away from loading the full list of ftp files on the manager page.  The the manager page can be simple and just show existing tasks.

Maybe we update the 'filter' line to have a checkbox to load the files from ftp, or wait for someone to enter something into the filter box, or do some action such as clicking the 'filter' button or a new button to 'load list of ftp files'.

Version-Release number of selected component (if applicable):
retrace-server-1.11-4.el6.noarch


How reproducible:
Every time loaded, though the thousands of files may be removed in the next several days.


Steps to Reproduce:
1. go to the 'manager' web UI page when there's thousands of files on the ftp server


Actual results:
web UI is slow, hangs or crashes browser


Expected results:
web UI is more responsive.  If you need to find a file with a specific pattern, that should be a non-default action.


Additional info:
This is probably a low-frequency problem, but if it happens, it makes the webUI much less usable.  We put in a ticket to remove these files, but the request was denied, at least in part due to the mechanics of how the ftp server works.

The format of the vmcore split files is vmcore_part_XXXX.  I think there are 98,000 of these files.

Comment 1 Marc Milgram 2014-07-29 14:40:01 UTC
Would it be possible to have it display files that match some string?  Frequently the customer indicates that he uploaded  core but fails to list the name of the file.  I search for the case number and frequently find the core.

Or would this be as slow?

Comment 2 Guy Streeter 2014-07-29 14:49:12 UTC
There is already a "filter" field at the top of the page. The problem is that it shows unfiltered as soon as you go to the page. I think making it wait til you put something in the filter field is a great idea. And I suppose it could take an empty filter and show everything.

While I'm at it: I can never remember if the filter needs/uses wildcards, or if it just looks for what you enter as a substring. Maybe a bit of label text or a mouseover popup could explain that.

Comment 3 Kyle Walker 2014-07-31 13:52:32 UTC
Just to chime in. Since there is the possibility of a future customer uploading an unreasonable number of files in the future, I completely agree that populating the "FTP Files" on load should be avoided. It seems like unnecessary load when you can just use a '*' wildcard after the fact and achieve the same end state.

To that note, the filter filed does accept wildcards and usually requires them. For example:

00930333      Returns:
    No files
*00930333*    Returns 
   00930333-vmcore


However, in poking around, I don't believe the fix is that straightforward. Searching through the FTP server contents seems to be the bottleneck in this interaction:

$ time curl https://optimus.gsslab.rdu2.redhat.com/manager 1>/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.9M  100 12.9M    0     0  2156k      0  0:00:06  0:00:06 --:--:-- 3431k

real	0m6.170s
user	0m0.119s
sys	0m0.103s


$ time curl https://optimus.gsslab.rdu2.redhat.com/manager?filter=None 1>/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4932  100  4932    0     0    912      0  0:00:05  0:00:05 --:--:--  1218

real	0m5.409s
user	0m0.047s
sys	0m0.132s


The filter behaviour still needs to traverse the contents of the FTP server and returns in roughly the same amount of time from what I see.

It's just my 2 cents, but to bypass the issue at this particular time, I would recommend determining the filename exactly and just plug it into the URL for the task creation page:

https://optimus.gsslab.rdu2.redhat.com/manager/<Filename>

Comment 4 Harshula Jayasuriya 2014-08-01 07:47:30 UTC
Is there some magic javascript library that dynamically loads the list of files as they are received from the FTP server? Such a mechanism would allow the rest of the page to be viewable and usable.

Comment 6 Dave Wysochanski 2015-07-10 22:27:08 UTC
Apparently optimus UI is now suffering from this problem again due to someone uploading a ton of small split files starting with 'x' (for example, "xzaei") though doesn't appear to be quite as bad as before.

Last time we opened a ticket with IT and they wouldn't remove the files, stating they would "auto-remove" within 30 days.

Maybe we should re-think the web UI though this bug still remains a fairly low priority in relation to other retrace-server bugs.

Comment 7 Kyle Walker 2016-03-14 17:00:46 UTC
I went ahead and put together a patch that addresses this condition and submitted it upstream.

   https://github.com/abrt/retrace-server/pull/108


In short, I moved the FTP query operation into a separate AJAX call. The only additional requirement that this includes is the ability to access the google-hosted jQuery library similar to:

   <script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>

Kyle Walker
Software Maintenance Engineer - SEG
North America

Comment 8 Dave Wysochanski 2016-05-31 21:03:28 UTC
I just tested kyle's patch on our development server and it looks like it definitely fixes this problem.

Even with a constant creating stream of files on the ftp server:
# for i in $(seq 1 10000000); do touch /var/ftp/pub/file.$i; done

The 'manager' interface does not hang.  I can't claim to fully understand the patch though I do grok enough to see we won't block at the same place and this looks like it fixes the problem.

Assuming Jakub and/or Michal is ok with it, hopefully it can be merged soon.

Comment 9 Fedora Update System 2016-06-03 09:46:02 UTC
retrace-server-1.16-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-4cd8a3c0e5

Comment 10 Fedora Update System 2016-06-03 09:46:14 UTC
retrace-server-1.16-1.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-81eb6f786a

Comment 11 Fedora Update System 2016-06-03 09:46:30 UTC
retrace-server-1.16-1.fc23 has been submitted as an update to Fedora 23. https://bodhi.fedoraproject.org/updates/FEDORA-2016-d84c5abe3f

Comment 12 Fedora Update System 2016-06-03 09:46:39 UTC
retrace-server-1.16-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-2e49e9538e

Comment 13 Fedora Update System 2016-06-04 18:26:24 UTC
retrace-server-1.16-1.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-81eb6f786a

Comment 14 Fedora Update System 2016-06-05 03:58:43 UTC
retrace-server-1.16-1.fc23 has been pushed to the Fedora 23 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-d84c5abe3f

Comment 15 Fedora Update System 2016-06-05 07:18:15 UTC
retrace-server-1.16-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-4cd8a3c0e5

Comment 16 Fedora Update System 2016-06-05 07:51:44 UTC
retrace-server-1.16-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-2e49e9538e

Comment 17 Dave Wysochanski 2016-06-07 09:17:57 UTC
I checked retrace-server-1.16-1.el6 and the loading of the manager page is much improved in the presence of a large number of ftp files.  I simulated the large number of ftp files by having a loop creating a lot of files in the ftp server directory and then on the retrace-server machine, trying to load the 'manager' page.

Comment 18 Fedora Update System 2016-06-13 20:55:50 UTC
retrace-server-1.16-1.fc23 has been pushed to the Fedora 23 stable repository. If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2016-07-01 06:19:20 UTC
retrace-server-1.16-2.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-2e49e9538e

Comment 20 Fedora Update System 2016-07-01 06:19:48 UTC
retrace-server-1.16-2.fc24 has been submitted as an update to Fedora 24. https://bodhi.fedoraproject.org/updates/FEDORA-2016-81eb6f786a

Comment 21 Fedora Update System 2016-07-01 07:57:01 UTC
retrace-server-1.16-2.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-4cd8a3c0e5

Comment 22 Fedora Update System 2016-07-02 20:17:20 UTC
retrace-server-1.16-2.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-4cd8a3c0e5

Comment 23 Fedora Update System 2016-07-02 20:21:12 UTC
retrace-server-1.16-2.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2016-2e49e9538e

Comment 24 Fedora Update System 2016-07-02 20:31:22 UTC
retrace-server-1.16-2.fc24 has been pushed to the Fedora 24 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-81eb6f786a

Comment 25 Fedora Update System 2017-03-30 14:11:19 UTC
retrace-server-1.17.0-1.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-3d55370e77

Comment 26 Fedora Update System 2017-03-30 14:12:12 UTC
retrace-server-1.17.0-1.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-9390d60e0d

Comment 27 Fedora Update System 2017-03-31 03:47:38 UTC
retrace-server-1.17.0-1.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-9390d60e0d

Comment 28 Fedora Update System 2017-03-31 03:49:00 UTC
retrace-server-1.17.0-1.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2017-3d55370e77

Comment 29 Fedora Update System 2017-04-18 21:18:35 UTC
retrace-server-1.17.0-1.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.