|Summary:||Tomcat leaks memory on every request|
|Component:||Catalina||Assignee:||Tomcat Developers Mailing List <dev>|
|Attachments:||some JRockit Memory Leak Detector pictures and the script to send data|
Description vilks56 2006-11-28 10:07:27 UTC
POST parameter values stay referenced by a static field in org.apache.tomcat.util.buf.StringCache - thus the GC can never release the memory. Steps to Reproduce: 1) install and start tomcat 5.5 (http://apache.zone-h.org/tomcat/tomcat-5/v5.5.20/bin/apache-tomcat-5.5.20.zip) 2) see the example webapp "Request Parameters Example" included in the install (http://localhost:8080/servlets-examples/servlet/RequestParamExample) 3) run script "send.sh" to create heavy load #!/bin/bash export LEAK_COMMAND='curl --silent http://localhost:8080/servlets-examples/servlet/RequestParamExample --output response.htm --data @send.txt' #create initial POST parameter with value about 10kB echo "firstname=" > send.txt for((i=0;$i<=10000;i=$(($i+1))));do echo -n "x" >> send.txt done #send the request for((i=0;$i<=1000;i=$(($i+1))));do echo $i # modify the param value to send (append i) echo -n $i >> send.txt $LEAK_COMMAND done 4) running the script causes heap usage to increase for some 40MB
Comment 1 vilks56 2006-11-28 10:13:01 UTC
Created attachment 19184 [details] some JRockit Memory Leak Detector pictures and the script to send data
Comment 2 Remy Maucherat 2006-11-28 15:46:47 UTC
You seem to be in a hurry. You did not seem to notice it, but the cache is bounded and also configurable. Using the Servlet API provided methods for parameters handling with very large POSTs is inefficient and will lead to large amounts of garbage collections. If you'd like me to look into adjusting the defaults or some of the behavior of the cache, you should send me an explanation I can understand (unlike these scripts).
Comment 3 Remy Maucherat 2006-11-28 16:06:48 UTC
Ok, I understand the script I think. The first one generate a: "firstname=xxxxxxx[long]xxxxxxxx" line. The second makes a number of requests while appending a number to it. Solution: the string cache is useful, but there's little point caching large strings.
Comment 4 vilks56 2006-11-29 06:36:18 UTC
(In reply to comment #3) > Ok, I understand the script I think. Yes, the script works exactly how You said. I created it just to isolate the problem from my real app where I need to POST many large Strings (could be megabytes). First, I don't understand why there is a need to cache request parameters for a longer duration than it takes to process the request. It would be useful only in very rare cases: if clients would make the exact same request many times. Second, I could not confgure the cahce in a useful way. These are default StringCache settings in catalina.properties: # String cache configuration. tomcat.util.buf.StringCache.byte.enabled=true #tomcat.util.buf.StringCache.char.enabled=true #tomcat.util.buf.StringCache.trainThreshold=500000 #tomcat.util.buf.StringCache.cacheSize=5000 The byte cahce is enabled by default and the memrory leak occurs. I tried to avoid the leak by trying different configurations and here is what I discovered: 1) The leak does not appear if the byte cache is disabled (...StringCache.byte.enabled=false). But disabled cache can't be useful at all. 2) I don't know what is the purpose of trainTreshold property, but that is the only property that can help: setting it low enough (even extremely low) avoids the leak. For example, lowering trainTreshold to 100 for 10kB requests avoids the leak, but isn't that the same as disabling the cache? 3) Limiting the cache size does not work as expected - for example, with cacheSize=1 the large requests still lead to exhaustion of heap memory. Digging with Memory Leak Detector shows that _all_ the request parameters still stay in heap, referenced by a static field in org.apache.tomcat.util.buf.StringCache.
Comment 5 Remy Maucherat 2006-11-29 08:21:41 UTC
This cache caches the result of the conversion of byte to String. This is an expensive operation since it includes many allocations and B2C operations(which makes the cache lookup cost insignificant as long as no sync occurs), and it's a bit of an experiment. The way the cache works is that it looks for a while at the strings the webapp is using, build usage data (for example, the attributes names, etc, are going to be used oftem), and then generates a static cache out of that. The cache is supposed to be relatively small, however, so if it can get too big it's bug. I added a limit on the String size in the code along with a new system property, which should be all that is needed.
Comment 6 Mark Thomas 2006-12-02 12:12:26 UTC
This has been fixed by Remy in SVN for 6.0.x and I have ported his fix to 5.5.x. The fix will be included in 5.5.21 and 6.0.3 onwards.