Bug 41057 - Tomcat leaks memory on every request
Summary: Tomcat leaks memory on every request
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 5
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 5.5.20
Hardware: All Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2006-11-28 10:07 UTC by vilks56
Modified: 2006-12-02 12:12 UTC (History)
0 users



Attachments
some JRockit Memory Leak Detector pictures and the script to send data (60.85 KB, patch)
2006-11-28 10:13 UTC, vilks56
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description vilks56 2006-11-28 10:07:27 UTC
POST parameter values stay referenced by a static field in
org.apache.tomcat.util.buf.StringCache - thus the GC can never release the memory.

Steps to Reproduce:
1) install and start tomcat 5.5
(http://apache.zone-h.org/tomcat/tomcat-5/v5.5.20/bin/apache-tomcat-5.5.20.zip)

2) see the example webapp "Request Parameters Example" included in the install
(http://localhost:8080/servlets-examples/servlet/RequestParamExample)

3) run script "send.sh" to create heavy load
#!/bin/bash

export LEAK_COMMAND='curl --silent
http://localhost:8080/servlets-examples/servlet/RequestParamExample --output
response.htm --data @send.txt'

#create initial POST parameter with value about 10kB

echo "firstname=" > send.txt
for((i=0;$i<=10000;i=$(($i+1))));do
  echo -n "x" >> send.txt
done


#send the request
for((i=0;$i<=1000;i=$(($i+1))));do
  echo $i
  # modify the param value to send (append i)
  echo -n $i >> send.txt
  
  $LEAK_COMMAND

done

4) running the script causes heap usage to increase for some 40MB
Comment 1 vilks56 2006-11-28 10:13:01 UTC
Created attachment 19184 [details]
some JRockit Memory Leak Detector pictures and the script to send data
Comment 2 Remy Maucherat 2006-11-28 15:46:47 UTC
You seem to be in a hurry. You did not seem to notice it, but the cache is
bounded and also configurable. Using the Servlet API provided methods for
parameters handling with very large POSTs is inefficient and will lead to large
amounts of garbage collections.

If you'd like me to look into adjusting the defaults or some of the behavior of
the cache, you should send me an explanation I can understand (unlike these
scripts).
Comment 3 Remy Maucherat 2006-11-28 16:06:48 UTC
Ok, I understand the script I think. The first one generate a:
"firstname=xxxxxxx[long]xxxxxxxx" line. The second makes a number of requests
while appending a number to it. Solution: the string cache is useful, but
there's little point caching large strings.
Comment 4 vilks56 2006-11-29 06:36:18 UTC
(In reply to comment #3)
> Ok, I understand the script I think. 

Yes, the script works exactly how You said. I created it just to isolate the
problem from my real app where I need to POST many large Strings (could be
megabytes). 

First, I don't understand why there is a need to cache request parameters for a
longer duration than it takes to process the request. It would be useful only in
very rare cases: if clients would make the exact same request many times.

Second, I could not confgure the cahce in a useful way. These are default 
StringCache settings in catalina.properties:

# String cache configuration.
tomcat.util.buf.StringCache.byte.enabled=true
#tomcat.util.buf.StringCache.char.enabled=true
#tomcat.util.buf.StringCache.trainThreshold=500000
#tomcat.util.buf.StringCache.cacheSize=5000

The byte cahce is enabled by default and the memrory leak occurs.

I tried to avoid the leak by trying different configurations and here is what I
discovered:

1) The leak does not appear if the byte cache is disabled
(...StringCache.byte.enabled=false). But disabled cache can't be useful at all.

2) I don't know what is the purpose of trainTreshold property, but that is the
only property that can help: setting it low enough (even extremely low) avoids
the leak. For example, lowering trainTreshold to 100 for 10kB requests avoids
the leak, but isn't that the same as disabling the cache?

3) Limiting the cache size does not work as expected - for example, with
cacheSize=1 the large requests still lead to exhaustion of heap memory. Digging
with Memory Leak Detector shows that _all_ the request parameters still stay in
heap, referenced by a static field in org.apache.tomcat.util.buf.StringCache.
Comment 5 Remy Maucherat 2006-11-29 08:21:41 UTC
This cache caches the result of the conversion of byte[] to String. This is an
expensive operation since it includes many allocations and B2C  operations(which
makes the cache lookup cost insignificant as long as no sync occurs), and it's a
bit of an experiment.

The way the cache works is that it looks for a while at the strings the webapp
is using, build usage data (for example, the attributes names, etc, are going to
be used oftem), and then generates a static cache out of that.

The cache is supposed to be relatively small, however, so if it can get too big
it's bug. I added a limit on the String size in the code along with a new system
property, which should be all that is needed.
Comment 6 Mark Thomas 2006-12-02 12:12:26 UTC
This has been fixed by Remy in SVN for 6.0.x and I have ported his fix to 5.5.x.

The fix will be included in 5.5.21 and 6.0.3 onwards.