Bug 60670 - Fails to create document nodes in MSI files larger than 2 GB
Summary: Fails to create document nodes in MSI files larger than 2 GB
Status: NEEDINFO
Alias: None
Product: POI
Classification: Unclassified
Component: POIFS (show other bugs)
Version: 3.15-FINAL
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: POI Developers List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-01-31 08:03 UTC by Marcus Lundblad
Modified: 2017-03-01 09:29 UTC (History)
1 user (show)



Attachments
Stand-alone program to expose bug (736 bytes, text/x-java)
2017-01-31 14:59 UTC, Marcus Lundblad
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marcus Lundblad 2017-01-31 08:03:06 UTC
I have generated 2 MSI files 3 GB in size (one with a single large document entry and one with three smaller 1 GB entries).
Trying to add an additional document entry in the file system, I get:

Caused by: java.lang.IllegalArgumentException: Negative position
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:755) [rt.jar:1.8.0_121]
	at org.apache.poi.poifs.nio.FileBackedDataSource.write(FileBackedDataSource.java:120) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.createBlockIfNeeded(NPOIFSFileSystem.java:505) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBuffer.createBlockIfNeeded(NPOIFSStream.java:226) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.NPOIFSStream$StreamBlockByteBuffer.write(NPOIFSStream.java:246) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.NPOIFSDocument.store(NPOIFSDocument.java:143) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.NPOIFSDocument.<init>(NPOIFSDocument.java:84) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.DirectoryNode.createDocument(DirectoryNode.java:422) [poi-3.15.jar:3.15]
	at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.createDocument(NPOIFSFileSystem.java:689) [poi-3.15.jar:3.15]
	at org.signserver.module.msauthcode.signer.MSAuthCodeSigner.signMSI(MSAuthCodeSigner.java:570) [SignServer-Module-MSAuthCode-4.1.0-SNAPSHOT.jar:]

I think the culprit here is the prevBlock and nextBlock index "pointers" in StreamBlockByteBuffer being ints causing an overflow.
Comment 1 Nick Burch 2017-01-31 08:16:06 UTC
I can't seem to find the source code for MSAuthCodeSigner, so it's hard to tell if this is a POI bug or a mistake in the calling code. (It seems to be missing from the module list in https://svn.cesecore.eu/svn/signserver/trunk/signserver/modules/ )

Are you able to share the POI-related lines of code from there, so we can take a look? (If it is a bug, we'll need to write a unit test, which'll require the code steps taken to reproduce)
Comment 2 Marcus Lundblad 2017-01-31 09:08:25 UTC
Right, missed that, sorry about that.

This is in a module in-development.
The code looks something like this:

final NPOIFSFileSystem fsOut =
                    new NPOIFSFileSystem(responseData.getAsFile(), false)) {
                // Add the signature file
fsOut.createDocument(new ByteArrayInputStream(signedbytes), "\05DigitalSignature");

where the getAsFile() call would return a File instance pointing to the resulting file (which is just a copy of the original file).
and the signedbytes is a byte array with the signature data (I don't think the content should really matter in this case.

I could try to write a simple stand-alone program to expose the issue as well.
Comment 3 Andreas Beeker 2017-01-31 09:52:22 UTC
Btw. I've received a contribution from Sebastian S. a while ago, which I eventually plan to integrate ... maybe it's a help to you ...

http://people.apache.org/~kiwiwings/20140908_MSIAPI_JUnitTests.zip
Comment 4 Nick Burch 2017-01-31 10:07:57 UTC
If you could knock up a failing unit test based on a temp File and some huge empty starting entries, that'd be great!

If not, I can try later
Comment 5 Marcus Lundblad 2017-01-31 14:59:27 UTC
Created attachment 34697 [details]
Stand-alone program to expose bug

The supplied program can be run with an input file as commandline argument.
The file is overwritten (if successful).
Comment 6 Marcus Lundblad 2017-01-31 15:06:22 UTC
(In reply to Marcus Lundblad from comment #5)
> Created attachment 34697 [details]
> Stand-alone program to expose bug
> 
> The supplied program can be run with an input file as commandline argument.
> The file is overwritten (if successful).

https://www.update.uu.se/~ml/large.zip

An archive containing a very large MSI file.
NOTE: the file will decompress to a very large file.
The MSI file was created with msibuilder (from msitools) and only contains document streams filled with zeros. It will not actually work as a real installer in Windows.
Comment 7 Nick Burch 2017-02-10 13:26:58 UTC
I've just tried with your program and your 3gb msi file, but I think the file might be corrupt? When I run the program, it fails with:

Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 8, Size: 8
	at java.util.ArrayList.rangeCheck(ArrayList.java:635)
	at java.util.ArrayList.get(ArrayList.java:411)
	at org.apache.poi.poifs.property.PropertyTableBase.populatePropertyTree(PropertyTableBase.java:128)
	at org.apache.poi.poifs.property.PropertyTableBase.<init>(PropertyTableBase.java:63)
	at org.apache.poi.poifs.property.NPropertyTable.<init>(NPropertyTable.java:66)
	at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.readCoreContents(NPOIFSFileSystem.java:440)
	at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:235)
	at org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.java:168)
	at Main.main(Main.java:18)

I've done a fix for PropertyTableBase to skip broken entries. That then lets me run your program without error
Comment 8 Marcus Lundblad 2017-03-01 09:29:47 UTC
(In reply to Nick Burch from comment #7)
> I've just tried with your program and your 3gb msi file, but I think the
> file might be corrupt? When I run the program, it fails with:
> 
> Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 8,
> Size: 8
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> 	at java.util.ArrayList.get(ArrayList.java:411)
> 	at
> org.apache.poi.poifs.property.PropertyTableBase.
> populatePropertyTree(PropertyTableBase.java:128)
> 	at
> org.apache.poi.poifs.property.PropertyTableBase.<init>(PropertyTableBase.
> java:63)
> 	at
> org.apache.poi.poifs.property.NPropertyTable.<init>(NPropertyTable.java:66)
> 	at
> org.apache.poi.poifs.filesystem.NPOIFSFileSystem.
> readCoreContents(NPOIFSFileSystem.java:440)
> 	at
> org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.
> java:235)
> 	at
> org.apache.poi.poifs.filesystem.NPOIFSFileSystem.<init>(NPOIFSFileSystem.
> java:168)
> 	at Main.main(Main.java:18)
> 
> I've done a fix for PropertyTableBase to skip broken entries. That then lets
> me run your program without error

I can still reproduce the "Negative position" exception problem with a file generated by the WiX toolset.
I only managed to generate a very large file with random data (so it won't compress, will see if I can still provide a link to download it).
Also, it did actually work with my sample test program, but when I increased the size of the byte array to add as an extra document to 10000 bytes I got the same error again, so I guess it might be due to padding (when a new block needs to be created).