We are observing the regression in the title with java version "1.8.0_201" Java(TM) SE Runtime Environment (build 1.8.0_201-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode) the relevant JVM flags are "-Xmx3584m -Xms256m" on Linux [...] 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux the same regression does NOT happen on Windows, under a variety of java versions and stack/heap space (all java 8). ------------- Detailed description: When using `WorkbookFactory.create()` on this XLS file: https://drive.google.com/file/d/10XOpyz2imxJ6Q3cItqhboab8EWdUEQR_/view?usp=sharing the entire JVM crashes because it's trying to request too much memory from the OS, with the following message: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory([...], [...], 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map [...] bytes for committing reserved memory. # An error report file with more information is saved as: # [...] this does not happen in v3.13 and lower, but starts happening v3.14-beta1, I've tested several versions, including 3.14, 3.17, 4.1.0 and others I've probably forgotten. Note that this is different from the usual OOM errors that happen while reading large XLS files, as those do not cause the whole JVM to crash. The files that trigger this error are rare, but I've seen more than one example. Please let me know if there is any other test of information I can do/give.
Could you move this to the dev list? Reading xlsx files using XSSFWorkbook takes a lot of memory. In my mind you have options like * stay with POI 3.13 * increase your JVM heap * rewrite your code to use a streaming xlsx parser (eg https://svn.apache.org/repos/asf/poi/trunk/src/examples/src/org/apache/poi/xssf/eventusermodel/XLSX2CSV.java) * try to patch POI 4.1 to use less memory
Do note that this is an XLS file, I expect WorkbookFactory.create to create a HSSFWorkbook under the hood. XLSX files are much less memory heavy, and there are effective streaming parsers for it, but no such options exist for XLS files. I imagine something in the 4.14-beta dev cycle raised by a lot the memory needed for an XLS file, I hope there is a way to fix it or disable it from the user side.
Any chance you could take the lead on investigating the memory issue by using a java profiler and testing with POI 3.13 and POI 4.1 (best to use POI 4.1 because if we fix something it will be in that version)?
I'm not exactly sure how exactly to investigate it, the JVM straight crashes, it's not a regular OOM. I also don't have a linux dev environment right now, but I can acquire it. If you can suggest a testing procedure, I can try it.
If I don't know where and when an exception / crash happens, I usually add log / system.out statements to the source. So you can iteratively narrow the error corridor. A different strategy is https://git-scm.com/docs/git-bisect, to find out which commit caused the trouble.
Please note that a crash with "Cannot allocate memory" indicates that your system runs out of physical available memory overall, not just for the single Java process. So it seems your Xmx is higher than the amount of memory available on your system. You might simply test on a different machine with more available physical memory or try stopping some memory-hungry processes on this machine to see if that makes it work. Also reducing the Xmx might show the usual OutOfMemory exception again. In general, if the JVM crashes, the problem is usually outside of the Java program itself.
I understand that giving the program more space would solve the issue, but what I wanted to point out is that it does not happen in 3.13, at all. I guess I can try to bisect the 3.13 - 3.14-beta1 cycle and see what commit is the culprit, if you guys cannot do so.
What I was trying to say was that the error you see actually indicates that Java itself did not use too much memory yet, but allocating the memory from the operating system failed, i.e. like if you have a system with only 2GB or RAM, but give Java -Xmx3g, at some point it would not be able to aquire more memory from the operating system.
The system has plenty of memory leftover, I easily have 5 free GBs over the max size of java's heap. I am trying to bisect the 3.13 - 3.14 cycle to find the problematic commit, but I'm having some issues. I can easily replicate the issue on a separate project with the two artifacts from maven, but I can't get the library junit tests to help me. I've added a junit test doing exactly what I do in my standalone program, but whatever tag I checkout, `ant test-main` always fails at my test with an OOM (either heap space or GM limit, NOT the horrible crash a standalone program does), no matter what heap space I give with ANT_OPTS. I could bisect by building the jars each time and importing them in my toy project, but that seems very very laborious, do you have better ideas?
You could use git repo and use git-biset to do the tedious work for you if you can come up with a script which does the full building-jars-and-run-application cycle with a result, see https://git-scm.com/docs/git-bisect for details. I have used this a number of times successfully in the past.