Bug 65661 - 'OutOfMemoryError: Direct buffer memory' in DiskFileItem.get()
Summary: 'OutOfMemoryError: Direct buffer memory' in DiskFileItem.get()
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 9
Classification: Unclassified
Component: Util (show other bugs)
Version: 9.0.54
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: -----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-10-28 09:36 UTC by Peter Kovacs
Modified: 2021-11-02 14:20 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Peter Kovacs 2021-10-28 09:36:09 UTC
DiskFileItem.get() loads a whole multipart part into memory. Since the commit 6650205974619771f9ffe19d1b7a5490ce468e9d it uses Files.newInputStream(...) to read the file contents. This creates a ChannelInputStream behind the scenes which might use direct memory instead of heap.

This should work correctly except that usually direct memory limit is set much lower for applications than regular heap size. Thus this may result in 'OutOfMemoryError: Direct buffer memory' for large or multiple concurrent multipart uploads, when  -XX:MaxDirectMemorySize is not set high enough. (in our case it was 128M).

Not sure how to fix it to keep the performance the same, but I think at least it should be documented somewhere that applications need at least multipart size limit * max. concurrent uploads bytes of direct memory.



Suspected change: https://github.com/apache/tomcat/commit/6650205974619771f9ffe19d1b7a5490ce468e9d
Location:
https://github.com/apache/tomcat/blame/main/java/org/apache/tomcat/util/http/fileupload/disk/DiskFileItem.java#L304


JVM: openjdk 11.0.13 2021-10-19 LTS


Ok with 9.0.48, fails on 9.0.54

Stacktrace:
java.lang.OutOfMemoryError: Direct buffer memory
	at java.base/java.nio.Bits.reserveMemory(Unknown Source)
	at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)
	at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)
	at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
	at java.base/sun.nio.ch.FileChannelImpl.read(Unknown Source)
	at java.base/sun.nio.ch.ChannelInputStream.read(Unknown Source)
	at java.base/sun.nio.ch.ChannelInputStream.read(Unknown Source)
	at java.base/sun.nio.ch.ChannelInputStream.read(Unknown Source)
	at org.apache.tomcat.util.http.fileupload.IOUtils.read(IOUtils.java:199)
	at org.apache.tomcat.util.http.fileupload.IOUtils.readFully(IOUtils.java:226)
	at org.apache.tomcat.util.http.fileupload.IOUtils.readFully(IOUtils.java:247)
	at org.apache.tomcat.util.http.fileupload.disk.DiskFileItem.get(DiskFileItem.java:305)
	at org.apache.tomcat.util.http.fileupload.disk.DiskFileItem.getString(DiskFileItem.java:327)
	at org.apache.catalina.core.ApplicationPart.getString(ApplicationPart.java:127)
	at org.apache.catalina.connector.Request.parseParts(Request.java:2948)
	at org.apache.catalina.connector.Request.getParts(Request.java:2823)
Comment 1 Mark Thomas 2021-11-02 14:20:24 UTC
Thanks for the report.

I have added a note to the change log for each of the affected versions and to the migration guide.