Bug 62081 - java.lang.OutOfMemoryError: Map failed occurs when processing a bunch of msg files
Summary: java.lang.OutOfMemoryError: Map failed occurs when processing a bunch of msg ...
Status: RESOLVED LATER
Alias: None
Product: POI
Classification: Unclassified
Component: POI Overall (show other bugs)
Version: 3.16-FINAL
Hardware: PC Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: POI Developers List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-02-06 15:36 UTC by Sathish Kumar V
Modified: 2018-04-02 21:11 UTC (History)
0 users


Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Sathish Kumar V 2018-02-06 15:36:52 UTC 
I am trying to read and extract contents of msg file using POI 3.16.  While trying to process a bunch of msg files (not large, in the <500kb) and the following error occurs.  
==========================================================================
org.apache.poi.EmptyFileException: The supplied file was empty (zero bytes long)
        at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:218)
        at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:168)
        at com.testcode.MSGWriter.MapiMessage.addNamedPropertiesToMessage(MapiMessage.java:236)
        at com.testcode.MSGReader.MessageReader.parseAttachment(MessageReader.java:1352)
        at com.testcode.MSGReader.MessageReader.parseDirectory(MessageReader.java:520)
        at com.testcode.MSGReader.MessageReader.parseMessage(MessageReader.java:405)
        at com.testcode.MSGReader.MessageReader.readMessageFile(MessageReader.java:314)
        at txtGeneration.txtGeneration.Task.run(MTMsgTotxtConvertor.java:83)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
java.lang.RuntimeException: java.io.IOException: Map failed
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBufferIterator.next(NPOIFSStream.java:173)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBufferIterator.next(NPOIFSStream.java:142)
        at org.apache.poi.poifs.filesystem.NPOIFSMiniStore.getBlockAt(NPOIFSMiniStore.java:69)
        at org.apache.poi.poifs.filesystem.NPOIFSMiniStore.createBlockIfNeeded(NPOIFSMiniStore.java:97)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBuffer.createBlockIfNeeded(NPOIFSStream.java:226)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBuffer.write(NPOIFSStream.java:246)
        at org.apache.poi.poifs.filesystem.NPOIFSDocument.store(NPOIFSDocument.java:143)
        at org.apache.poi.poifs.filesystem.NPOIFSDocument.<init>(NPOIFSDocument.java:84)
        at org.apache.poi.poifs.filesystem.DirectoryNode.createDocument(DirectoryNode.java:422)
        at com.testcode.MSGWriter.MessageWriterHelper.writeNamedPropertyDocuments(MessageWriterHelper.java:938)
        at com.testcode.MSGWriter.MapiMessage.addNamedPropertiesToMessage(MapiMessage.java:242)
        at com.testcode.MSGReader.MessageReader.parseAttachment(MessageReader.java:1352)
        at com.testcode.MSGReader.MessageReader.parseDirectory(MessageReader.java:520)
        at com.testcode.MSGReader.MessageReader.parseMessage(MessageReader.java:405)
        at com.testcode.MSGReader.MessageReader.readMessageFile(MessageReader.java:314)
        at txtGeneration.txtGeneration.Task.run(MTMsgTotxtConvertor.java:83)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
        at org.apache.poi.poifs.nio.FileBackedDataSource.read(FileBackedDataSource.java:99)
        at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.getBlockAt(NPOIFSFileSystem.java:484)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBufferIterator.next(NPOIFSStream.java:169)
        ... 18 more
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
        ... 21 more
java.lang.RuntimeException: java.io.IOException: Map failed
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBufferIterator.next(NPOIFSStream.java:173)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBufferIterator.next(NPOIFSStream.java:142)
        at org.apache.poi.poifs.filesystem.NPOIFSMiniStore.getBlockAt(NPOIFSMiniStore.java:69)
        at org.apache.poi.poifs.filesystem.NPOIFSMiniStore.createBlockIfNeeded(NPOIFSMiniStore.java:97)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBuffer.createBlockIfNeeded(NPOIFSStream.java:226)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBuffer.write(NPOIFSStream.java:246)
        at org.apache.poi.poifs.filesystem.NPOIFSDocument.store(NPOIFSDocument.java:143)
        at org.apache.poi.poifs.filesystem.NPOIFSDocument.<init>(NPOIFSDocument.java:84)
        at org.apache.poi.poifs.filesystem.DirectoryNode.createDocument(DirectoryNode.java:422)
        at com.testcode.MSGWriter.MessageWriterHelper.writeNamedPropertyDocuments(MessageWriterHelper.java:938)
        at com.testcode.MSGWriter.MapiMessage.addNamedPropertiesToMessage(MapiMessage.java:242)
        at com.testcode.MSGReader.MessageReader.parseAttachment(MessageReader.java:1352)
        at com.testcode.MSGReader.MessageReader.parseDirectory(MessageReader.java:520)
        at com.testcode.MSGReader.MessageReader.parseMessage(MessageReader.java:405)
        at com.testcode.MSGReader.MessageReader.readMessageFile(MessageReader.java:314)
        at txtGeneration.txtGeneration.Task.run(MTMsgTotxtConvertor.java:83)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
        at org.apache.poi.poifs.nio.FileBackedDataSource.read(FileBackedDataSource.java:99)
        at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.getBlockAt(NPOIFSFileSystem.java:484)
        at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBufferIterator.next(NPOIFSStream.java:169)
        ... 18 more
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
        ... 21 more
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fd7c8a1a000, 262144, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 262144 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/testuser/CVLibTest/hs_err_pid6684.log
[thread 140564401960704 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fd7c93f0000, 65536, 1) failed; error='Cannot allocate memory' (errno=12)
[thread 140564395820800 also had an error]
==========================================================================
I see the following bug has been addressed in 3.15 already, but still seeing the issue.  for now sysctl -w vm.max_map_count to a larger value seems to be working, but can this be fixed?

Bug 60140 - OOM caused by Memory Leak in FileBackedDataSource
Comment 1 Dominik Stadler 2018-02-07 17:04:18 UTC 
Can you share some sample files which reproduce this?
Comment 2 Dominik Stadler 2018-04-02 21:11:24 UTC 
We cannot reproduce the problem with the information that was provided. No sample file was provided, so we cannot do much here unless we get more information. Thus closing this as LATER for now, please reopen with more information if this is still a problem for you.