After upgrading from 5.1.0 to 5.2.0 I can't open the same XLS file because I am getting the following error: org.apache.poi.util.RecordFormatException: Tried to allocate an array of length 2,000,001, but the maximum length for this record type is 2,000,000. If the file is not corrupt, please open an issue on bugzilla to request increasing the maximum allowable size for this record type. As a temporary workaround, consider setting a higher override value with IOUtils.setByteArrayMaxOverride() at org.apache.poi.util.IOUtils.throwRFE(IOUtils.java:535) ~[lisy-starter-4.7.3-SNAPSHOT-fat.jar:?] at org.apache.poi.util.IOUtils.checkLength(IOUtils.java:212) ~[lisy-starter-4.7.3-SNAPSHOT-fat.jar:?] at org.apache.poi.util.IOUtils.safelyAllocateCheck(IOUtils.java:497) ~[lisy-starter-4.7.3-SNAPSHOT-fat.jar:?] at org.apache.poi.hssf.record.RecordFactory.createRecords(RecordFactory.java:180) ~[lisy-starter-4.7.3-SNAPSHOT-fat.jar:?] at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:385) ~[lisy-starter-4.7.3-SNAPSHOT-fat.jar:?] at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:445) ~[lisy-starter-4.7.3-SNAPSHOT-fat.jar:?] Downgrading works. The production file was larger than my test file, so the error was seen first in production.
After using IOUtils.setByteArrayMaxOverride(2_000_001) to increase the limit, the following error occurs: org.apache.poi.util.RecordFormatException: Tried to allocate an array of length 38,142,464, but the maximum length for this record type is 2,000,001. If the file is not corrupt, please open an issue on bugzilla to request increasing the maximum allowable size for this record type. As a temporary workaround, consider setting a higher override value with IOUtils.setByteArrayMaxOverride() at org.apache.poi.util.IOUtils.throwRFE(IOUtils.java:535) ~[poi-5.2.0.jar:5.2.0] at org.apache.poi.util.IOUtils.checkLength(IOUtils.java:209) ~[poi-5.2.0.jar:5.2.0] at org.apache.poi.util.IOUtils.safelyAllocateCheck(IOUtils.java:497) ~[poi-5.2.0.jar:5.2.0] at org.apache.poi.poifs.filesystem.POIFSFileSystem.<init>(POIFSFileSystem.java:341) ~[poi-5.2.0.jar:5.2.0] at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init>(HSSFWorkbook.java:445) ~[poi-5.2.0.jar:5.2.0]
can you try a bigger number - one bigger than 38,142,464? if you want us to investigate why this is kicking in, you will need to provide an xls file that reproduces the issue so we can investigate if one of the limits is set too small for a common use case
> can you try a bigger number - one bigger than 38,142,464? Yes, this works. > if you want us to investigate why this is kicking in, you will need to > provide an xls file that reproduces the issue so we can investigate if one > of the limits is set too small for a common use case It's a company-related file, so it's difficult to provide it. I'm wondering why it happens in 5.2.0. I currently also don't have an idea how to create a reproducible example because I only have this one file I am loading.
org.apache.poi.hssf.record.RecordFactory was hardened in POI 5.2.0 to ensure that a max number of records could be created for 1 file - to avoid issues where malicious actors send crafted xls files that are designed to cause POI to create so many records that OutOfMemoryException can occur
I increased the default max number of records in trunk branch to be 5 million (up from 2 million). r1898149
Thanks for the increase, but for the case of the ~38m records I still have to increase the max number? I see the point why this was added. I just need to know if the usage of IOUtils.setByteArrayMaxOverride is the now recommended and correct way of fixing this?
try IOUtils.setByteArrayMaxOverride - note this setting is used for a few limits so setting it to a not too high, not too low value is tricky. Set it as high as you comfortable with - especially if you are not accepting files from unknown parties who could send you malicious files
https://poi.apache.org/components/configuration.html has some other settings that you might want to try out
Ok thanks, I think my current max size should be fine.
Resolving again as it is not clear why it was reopened. Please comment why you think this is not resolved or create a new issue for other topics.
For the record, with Apache POI 5.2.5 you can use RecordFactory#setMaxNumberOfRecords() to define a higher limit for this value so that you do not need to increase the general max allocation limit via IOUtils.setByteArrayMaxOverride().