Bug 64001 - Need increased record type size for hdgf.streams.StreamStore
Summary: Need increased record type size for hdgf.streams.StreamStore
Alias: None
Product: POI
Classification: Unclassified
Component: HDGF (show other bugs)
Version: 4.1.1-FINAL
Hardware: Macintosh All
: P2 enhancement (vote)
Target Milestone: ---
Assignee: POI Developers List
: 64027 (view as bug list)
Depends on: 63569
  Show dependency tree
Reported: 2019-12-13 13:42 UTC by David Shifflett
Modified: 2022-04-03 20:28 UTC (History)
1 user (show)

stack trace for Visio error (4.68 KB, text/plain)
2019-12-13 13:42 UTC, David Shifflett

Note You need to log in before you can comment on or make changes to this bug.
Description David Shifflett 2019-12-13 13:42:21 UTC
Created attachment 36916 [details]
stack trace for Visio error

The Visio file is larger than the 1 MB limit, so I can't attach it, but I can share it if needed.

Caused by: org.apache.poi.util.RecordFormatException: Tried to allocate an array of length 27907952, but 10000000 is the maximum for this record type.
If the file is not corrupt, please open an issue on bugzilla to request
increasing the maximum allowable size for this record type.
As a temporary workaround, consider setting a higher override value with IOUtils.setByteArrayMaxOverride()

See attached stack trace, which has calls to my code snipped out.
Comment 1 Dominik Stadler 2019-12-14 07:56:35 UTC
This is mostly caused/fixed by bug 63569 which prevents to set a custom limit currently.

The default limits should be set so memory allocations are guarded and still almost all files can be processed. There always will be possibility to encounter files with bigger sizes. 

So we do not want to set the default to the largest file out there as it would defeat the purpose of this functionality, i.e. to avoid allocating unexpected high amounts of main memory.
Comment 2 PJ Fanning 2019-12-24 09:29:24 UTC
Some changes have been made on trunk branch and anyone who wants to try them out could download the nightly build (eg https://builds.apache.org/view/P/view/POI/job/POI-DSL-1.8/837/artifact/build/dist/)
Comment 3 PJ Fanning 2019-12-24 09:31:00 UTC
*** Bug 64027 has been marked as a duplicate of this bug. ***
Comment 4 Dominik Stadler 2022-04-03 20:28:10 UTC
I think we should not try to increase these limits more and more but rather ask users to use the override-functionality if the used file is exceptionally large.

So closing this as WONTFIX, we can improve the error-message somewhat to indicate that large files require the user to set an override.