Red Hat Bugzilla – Bug 172557
CVE-2005-3510 Possible Tomcat DoS
Last modified: 2007-04-18 13:33:56 EDT
A denial of service flaw has been made public that can affect Tomcat 5.5
versions prior to 5.5.12 when serving directories containing lots of files. The
details of this flaw are sketchy as the reporter did not give many details in
the public report:
However there is a message on tomcat-user list that gives a little more
background into how to trigger the flaw:
There doesn't seem to be any other details and the list of affected versions
should not be trusted without verification. One write up of this flaw mentions
it may affect Windows only, but I'm not sure how they came to that conclusion.
We'll rate this as moderate for now, although it could be low or not an issue at
all depending on your analysis of how this affects RHAPS2.
I've confirmed that this issue is reproducable on linux with tomcat 5.5.11.
Running "ab -c 100 -n 10000 http://*server*/test" against a tomcat instance
where test is a directory with 100 files will cause the tomcat instance to keel
over fairly quickly. With my test case there is no significant impact to the
machine other than permantently hosting the tomcat instance. The same test runs
fine against tomcat 5.5.12.
I've now successfully reproduced the problem on 5.5.9, 5.5.11, and
5.5.12. I believe the problem is related to the strategy used by the
default servlet to cache filesystem access. This strategy is for the
most part unchanged between 5.5.9 and 5.5.12, and so I believe the
problem is present in all intermediate versions.
The best way I've found to demonstrate the problem is to run a series
of benchmarks against a directory listing with increasing concurrency
levels, but a constant number of total requests.
c | total | mean
1 | 12.250582 seconds | 122.506 ms
10 | 8.476202 seconds | 847.620 ms
20 | 7.917447 seconds | 1583.489 ms
30 | 7.818699 seconds | 2345.610 ms
40 | 9.72974 seconds | 3629.190 ms
50 | 9.952440 seconds | 4976.220 ms
60 | 14.325583 seconds | 8595.350 ms
70 | 58.615614 seconds | 41030.931 ms
80 | -- | --
90 | -- | --
100 | -- | --
c = number of concurrent requests (see the -c flag for ab)
total = total time for benchmark to complete
mean = the average time until each request is complete
-- = benchmark timed out
In all cases the total number of requests is 100 and the test
directory being listed contains 1000 empty files.
As you can see, the total time for the benchmark to complete increases
rapidly as the number of concurrent requests increases from 50 to 70.
My first guess was that this is somehow related to the cache used for
filesystem access since the drop off in performance happens right
around the point where the time to service requests exceeds the
default 5 second timeout for the cache. In order to test this theory I
reran the same set of benchmarks after adjusting the default timeout
to be 10 minutes. This yielded the following results:
c | total | mean
1 | 13.869870 seconds | 138.699 ms
10 | 8.692605 seconds | 869.261 ms
20 | 8.722117 seconds | 1744.423 ms
30 | 8.912768 seconds | 2673.831 ms
40 | 8.511701 seconds | 3404.680 ms
50 | 9.987676 seconds | 4993.838 ms
60 | 11.397467 seconds | 6838.480 ms
70 | 10.919906 seconds | 7643.934 ms
80 | 19.230226 seconds | 15384.181 ms
90 | 25.684044 seconds | 23115.639 ms
100 | 26.872654 seconds | 26872.654 ms
While this does seem to confirm that the problem is somehow related to
the caching strategy, I am unsatisfied with this as a complete
explanation because a simple microbenchmark in python or Java easily
indicates that performing the given number of file accesses without
*any* caching either in serial or in parallel takes significantly less
time than the server requires to respond or timeout and recover during
the above benchmarks. Also, although it seems to happen rarely under
my test conditions, the server does occasionally become permanently
unresponsive. This seems to occur when for some reason the server runs
out of memory and can no longer service incoming requests.
Given these results, I suspect that in addition to a difficult to
trigger memory leak there are likely some significant concurrency
bottlenecks either in the DefaultServlet implementation or in tomcat
itself. Some immediate workarounds to address this issue would be to
change the default tomcat install to disable directory listing and/or
reduce the number of threads used to service requests. To properly fix
the problem will require detailed profiling of both tomcat and the
default servlet implementation.
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.