Bug 64001

Summary: Need increased record type size for hdgf.streams.StreamStore
Product: POI Reporter: David Shifflett <shifflett_david>
Component: HDGFAssignee: POI Developers List <dev>
Status: NEW ---    
Severity: enhancement CC: snag
Priority: P2    
Version: 4.1.1-FINAL   
Target Milestone: ---   
Hardware: Macintosh   
OS: All   
Bug Depends on: 63569    
Bug Blocks:    
Attachments: stack trace for Visio error

Description David Shifflett 2019-12-13 13:42:21 UTC
Created attachment 36916 [details]
stack trace for Visio error

The Visio file is larger than the 1 MB limit, so I can't attach it, but I can share it if needed.

Caused by: org.apache.poi.util.RecordFormatException: Tried to allocate an array of length 27907952, but 10000000 is the maximum for this record type.
If the file is not corrupt, please open an issue on bugzilla to request
increasing the maximum allowable size for this record type.
As a temporary workaround, consider setting a higher override value with IOUtils.setByteArrayMaxOverride()

See attached stack trace, which has calls to my code snipped out.
Comment 1 Dominik Stadler 2019-12-14 07:56:35 UTC
This is mostly caused/fixed by bug 63569 which prevents to set a custom limit currently.

The default limits should be set so memory allocations are guarded and still almost all files can be processed. There always will be possibility to encounter files with bigger sizes. 

So we do not want to set the default to the largest file out there as it would defeat the purpose of this functionality, i.e. to avoid allocating unexpected high amounts of main memory.
Comment 2 PJ Fanning 2019-12-24 09:29:24 UTC
Some changes have been made on trunk branch and anyone who wants to try them out could download the nightly build (eg https://builds.apache.org/view/P/view/POI/job/POI-DSL-1.8/837/artifact/build/dist/)
Comment 3 PJ Fanning 2019-12-24 09:31:00 UTC
*** Bug 64027 has been marked as a duplicate of this bug. ***