Using POI version 2.5 final. When given an XLS file with a drop-down list, the constructor call to HSSFWorkbook throws the following exception: java.lang.reflect.InvocationTargetException: java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.poi.hssf.record.UnknownRecord.<init> (UnknownRecord.java:62) at org.apache.poi.hssf.record.SubRecord.createSubRecord (SubRecord.java:57) at org.apache.poi.hssf.record.ObjRecord.fillFields(ObjRecord.java:99) at org.apache.poi.hssf.record.Record.fillFields(Record.java:90) at org.apache.poi.hssf.record.Record.<init>(Record.java:55) at org.apache.poi.hssf.record.ObjRecord.<init>(ObjRecord.java:61) at java.lang.reflect.Constructor.newInstance(Native Method) at org.apache.poi.hssf.record.RecordFactory.createRecord (RecordFactory.java:224) at org.apache.poi.hssf.record.RecordFactory.createRecords (RecordFactory.java:160) at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init> (HSSFWorkbook.java:165) at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init> (HSSFWorkbook.java:132) at nas.components.nruf.poi25test.main(poi25test.java:14) Exception in thread "main" org.apache.poi.hssf.record.RecordFormatException: Unable to construct record instance, the fo llowing exception occured: null at org.apache.poi.hssf.record.RecordFactory.createRecord (RecordFactory.java:237) at org.apache.poi.hssf.record.RecordFactory.createRecords (RecordFactory.java:160) at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init> (HSSFWorkbook.java:165) at org.apache.poi.hssf.usermodel.HSSFWorkbook.<init> (HSSFWorkbook.java:132) at nas.components.nruf.poi25test.main(poi25test.java:14) I will attach the simple Java program used and an offending XLS file.
Created attachment 10968 [details] Java program that reproduces the bug
Created attachment 10969 [details] XLS file (with drop down) that causes bug
I was able to track down this bug even for an 'empty' workbook (that is, nothing is visible there). The error occurs if in a call to static method SubRecord.createSubRecord( short subRecordSid, short size, byte[] data, int offset ) the 'size' parameters exceeds the available space in the 'data' buffer. As a consequence. I implemented a workaround that works for me by doing some checks in method ObjRecord::fillFields() before the thread of control proceeds to SubRecord.createSubRecords() there. I'm well aware this patch might be dangerous. Find the diff of file ObjRecord.java below, the entire new file is placed into a separate attachment (that still contains some useful trace messages). 99c99,107 < Record subRecord = SubRecord.createSubRecord(subRecordSid, subRecordSize, data, pos + 4); --- > Record subRecord = null; > if (size - (pos+4) < subRecordSize) > { > subRecord = SubRecord.createSubRecord(subRecordSid,size-(pos+4)), data, pos+4); > } > else > { > subRecord = SubRecord.createSubRecord(subRecordSid, subRecordSize, data, pos+4); > } 103d110 <
Created attachment 11371 [details] ObjRecord-patch27929.java, workaround for bug 27929
I can reproduce this bug with 2.5 FINAL.
Here is an alternate patch, with JUnit Testcases. I came up with almost the same solution as Roland except 1) I adjusted the size in the static method SubRecord.createSubRecord(). I also set the sub record size to 0 if it is ever negative. 2) I adjusted the size of the subrecord to 4 less than the total size of the OBJ record. The last 4 bytes of an OBJ record should always be an ftEnd subrecord, and this appears to be the case with Autofilters and dropdown boxes inserted via the Forms toolbar. 3) I adjusted Obj.java to advance the offset by SubRecord.getRecordSize() (instead of 4 + subRecordSize). The value of subRecordSize may not be reliable. The patch includes new JUnit Testcases for the modified SubRecord.java and ObjRecord.java. These tests fail with the 2.5 as pulled yesterday, but work with the changes in the patch file.
Created attachment 11735 [details] CVS DIFF and new TestCase classes
Hi Michael, can you provide a unified diff? I'd rather merge the diff once for either Release 2 or HEAD. Thanks.
Created attachment 11954 [details] diff -u against HEAD
Michael, I'm just trying to understand your test case. The following test case is failing: public void testParseEnd() { Record r = SubRecord.createSubRecord( (short) 0x0015, (short) 0x0012, dataAutoFilter, 0x0000 ); assertEquals( "ftCmo is 22 bytes", 22, r.getRecordSize() ); assertEquals( "ftEnd is a EndSubRecord" , "org.apache.poi.hssf.record.EndSubRecord" , r.getClass().getName() ); } The sid you're passing in is for the CommonObjectDataSubRecord record yet your test is expecting it to be a EndSubRecord. Why is this the case?
Looks like I goofed. That last testcase should be public void testParseEnd() { Record r = SubRecord.createSubRecord( (short)0x0000, (short)0x0000, dataAutoFilter, 0x0046 ); assertEquals( "ftEnd is 4 bytes", 4, r.getRecordSize()); assertEquals( "ftEnd is a EndSubRecord" , "org.apache.poi.hssf.record.EndSubRecord" , r.getClass().getName() ); } Nothing fancy about this testcase. It merely takes the actual bytes from an OBJ record that describes a dropdown, which was causing a problem in this report. Since I needed to change that method (SubRecord.createSubRecord), and verifies that the subrecords are all parsed and created properly.
Created attachment 11963 [details] Revised TestSubRecord.java JUnit Testcase
Tried the new patch but get the following error now: java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method) at org.apache.poi.hssf.record.UnknownRecord.<init>(UnknownRecord.java:62) at org.apache.poi.hssf.record.SubRecord.createSubRecord(SubRecord.java:57) at org.apache.poi.hssf.record.TestSubRecord.testParseAutoFilterLbsData(TestSubRecord.java:82)
I'm guessing you did not apply the changes to SubRecord.java and ObjRecord.java http://issues.apache.org/bugzilla/showattachment.cgi?attach_id=11954 The ArrayIndexOutOfBounds is exactly what is happening inside the InvocationTargetException, whenever there is a dropdown list on the spreadsheet There is also a JUnit testcase TestSubRecord.java http://issues.apache.org/bugzilla/showattachment.cgi?attach_id=11963 BTW, there is also another JUnit testcase TestObjRecord.java, inside the zip file at http://issues.apache.org/bugzilla/showattachment.cgi?attach_id=11735 Let me see if I can put all this together in a single ZIP file Revised ObjRecord.java Revised SubRecord.java New TestObjRecord.java New TestSubRecord.java
as a sidenote, Eclipse's [Team - Create Patch] patch creation includes new files in one single patch. A lot easier to submit to bugzilla.
I applied them as far as I can tell. Here's a diff with the changes applied to you can compare for yourself. Index: java/org/apache/poi/hssf/record/ObjRecord.java =================================================================== RCS file: /home/cvs/jakarta-poi/src/java/org/apache/poi/hssf/record/ObjRecord.java,v retrieving revision 1.1.2.2 diff -u -r1.1.2.2 ObjRecord.java --- java/org/apache/poi/hssf/record/ObjRecord.java 22 Feb 2004 11:54:47 -0000 1.1.2.2 +++ java/org/apache/poi/hssf/record/ObjRecord.java 3 Jul 2004 11:47:12 -0000 @@ -98,7 +98,7 @@ short subRecordSize = LittleEndian.getShort(data, pos + 2); Record subRecord = SubRecord.createSubRecord(subRecordSid, subRecordSize, data, pos + 4); subrecords.add(subRecord); - pos += 4 + subRecordSize; + pos += subRecord.getRecordSize(); } } Index: java/org/apache/poi/hssf/record/SubRecord.java =================================================================== RCS file: /home/cvs/jakarta-poi/src/java/org/apache/poi/hssf/record/SubRecord.java,v retrieving revision 1.1.2.2 diff -u -r1.1.2.2 SubRecord.java --- java/org/apache/poi/hssf/record/SubRecord.java 22 Feb 2004 11:54:47 -0000 1.1.2.2 +++ java/org/apache/poi/hssf/record/SubRecord.java 3 Jul 2004 11:47:12 -0000 @@ -42,19 +42,28 @@ { Record r = null; + short adjustedSize = size; + if( size < 0) { + adjustedSize = 0; + } else if( offset + size > data.length) { + adjustedSize = (short) (data.length - offset); + if( adjustedSize > 4) { + adjustedSize -= 4; + } + } switch ( subRecordSid ) { case CommonObjectDataSubRecord.sid: - r = new CommonObjectDataSubRecord( subRecordSid, size, data, offset ); + r = new CommonObjectDataSubRecord( subRecordSid, adjustedSize, data, offset ); break; case GroupMarkerSubRecord.sid: - r = new GroupMarkerSubRecord( subRecordSid, size, data, offset ); + r = new GroupMarkerSubRecord( subRecordSid, adjustedSize, data, offset ); break; case EndSubRecord.sid: - r = new EndSubRecord( subRecordSid, size, data, offset ); + r = new EndSubRecord( subRecordSid, adjustedSize, data, offset ); break; default: - r = new UnknownRecord( subRecordSid, size, data, offset ); + r = new UnknownRecord( subRecordSid, adjustedSize, data, offset ); } return r; Index: testcases/org/apache/poi/hssf/record/TestObjRecord.java =================================================================== RCS file: testcases/org/apache/poi/hssf/record/TestObjRecord.java diff -N testcases/org/apache/poi/hssf/record/TestObjRecord.java --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ testcases/org/apache/poi/hssf/record/TestObjRecord.java 3 Jul 2004 11:47:18 -0000 @@ -0,0 +1,94 @@ +/* ==================================================================== + Copyright 2002-2004 Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +==================================================================== */ + +package org.apache.poi.hssf.record; + +import junit.framework.*; + +import java.util.List; + +/** + * Tests for OBJ record. Test data taken directly + * from a real Excel file. + * + * @author Michael Zalewski (zalewski at optonline.net) + */ +public class TestObjRecord extends TestCase +{ + + public TestObjRecord(String name) + { + super(name); + } + + +/* + The following is a dump of the OBJ record corresponding to an auto-filter + drop-down list. The 3rd subrecord beginning at offset 0x002e (type=0x0013) + does not conform to the documentation, because the length field is 0x1fee, + which is longer than the entire OBJ record. + + 00000000 15 00 12 00 14 00 01 00 01 21 00 00 00 00 3C 13 .........!....<. Type=0x15 Len=0x0012 ftCmo + 00000010 F4 03 00 00 00 00 + 0C 00 14 00 00 00 00 00 00 00 ................ Type=0x0c Len=0x0014 ftSbs + 00000020 00 00 00 00 01 00 08 00 00 00 10 00 00 00 + 13 00 ................ Type=0x13 Len=0x1FEE ftLbsData + 00000030 EE 1F 00 00 08 00 08 00 01 03 00 00 0A 00 14 00 ................ + 00000040 6C 00 + 00 00 00 00 l..... Type=0x00 Len=0x0000 ftEnd +*/ + + byte[] dataAutoFilter = new byte[] { + // ftCmo + (byte)0x15,(byte)0x00,(byte)0x12,(byte)0x00,(byte)0x14,(byte)0x00,(byte)0x01,(byte)0x00 + ,(byte)0x01,(byte)0x00,(byte)0x01,(byte)0x21,(byte)0x00,(byte)0x00,(byte)0x3c,(byte)0x13 + ,(byte)0xf4,(byte)0x03,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00 + + // ftSbs (currently UnknownSubrecord) + ,(byte)0x0c,(byte)0x00 + ,(byte)0x14,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00 + ,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x01,(byte)0x00,(byte)0x08,(byte)0x00 + ,(byte)0x00,(byte)0x00,(byte)0x10,(byte)0x00,(byte)0x00,(byte)0x00 + + // ftLbsData (currently UnknownSubrecord) + ,(byte)0x13,(byte)0x00 + ,(byte)0xee,(byte)0x1f,(byte)0x00,(byte)0x00,(byte)0x08,(byte)0x00,(byte)0x08,(byte)0x00 + ,(byte)0x01,(byte)0x03,(byte)0x00,(byte)0x00,(byte)0x0a,(byte)0x00,(byte)0x14,(byte)0x00 + ,(byte)0x6c,(byte)0x00 + + // ftEnd + ,(byte)0x00,(byte)0x00,(byte)0x00,(byte)0x00 + }; + + public void testAutoFilter() { + ObjRecord or = new ObjRecord( ObjRecord.sid, (short) dataAutoFilter.length, dataAutoFilter); + assertEquals( "Obj record size", 74, or.getRecordSize()); + List lstSubRecords = or.getSubRecords(); + assertEquals( "Subrecord count", 4, lstSubRecords.size()); + Object oSubRecord = lstSubRecords.get( 0); + assertEquals( + "First subrecord class" + , "org.apache.poi.hssf.record.CommonObjectDataSubRecord" + , oSubRecord.getClass().getName() + ); + oSubRecord = lstSubRecords.get( 3); + assertEquals( + "Last subrecord class" + , "org.apache.poi.hssf.record.EndSubRecord" + , oSubRecord.getClass().getName() + ); + } +} \ No newline at end of file Index: testcases/org/apache/poi/hssf/record/TestSubRecord.java =================================================================== RCS file: testcases/org/apache/poi/hssf/record/TestSubRecord.java diff -N testcases/org/apache/poi/hssf/record/TestSubRecord.java --- /dev/null 1 Jan 1970 00:00:00 -0000 +++ testcases/org/apache/poi/hssf/record/TestSubRecord.java 3 Jul 2004 11:47:19 -0000 @@ -0,0 +1,95 @@ +/* ==================================================================== + Copyright 2002-2004 Apache Software Foundation + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +==================================================================== */ + +package org.apache.poi.hssf.record; + +import junit.framework.*; + +/** + * Tests Subrecord components of an OBJ record. Test data taken directly + * from a real Excel file. + * + * @author Michael Zalewski (zalewski at optonline.net) + */ +public class TestSubRecord extends TestCase +{ + /* + The following is a dump of the OBJ record corresponding to an auto-filter + drop-down list. The 3rd subrecord beginning at offset 0x002e (type=0x0013) + does not conform to the documentation, because the length field is 0x1fee, + which is longer than the entire OBJ record. + + 00000000 15 00 12 00 14 00 01 00 01 21 00 00 00 00 3C 13 .........!....<. Type=0x15 Len=0x0012 ftCmo + 00000010 F4 03 00 00 00 00 + 0C 00 14 00 00 00 00 00 00 00 ................ Type=0x0c Len=0x0014 ftSbs + 00000020 00 00 00 00 01 00 08 00 00 00 10 00 00 00 + 13 00 ................ Type=0x13 Len=0x1FEE ftLbsData + 00000030 EE 1F 00 00 08 00 08 00 01 03 00 00 0A 00 14 00 ................ + 00000040 6C 00 + 00 00 00 00 l..... Type=0x00 Len=0x0000 ftEnd + */ + + byte[] dataAutoFilter = new byte[]{ + // ftCmo + (byte) 0x15, (byte) 0x00, (byte) 0x12, (byte) 0x00, (byte) 0x14, (byte) 0x00, (byte) 0x01, (byte) 0x00 + , (byte) 0x01, (byte) 0x00, (byte) 0x01, (byte) 0x21, (byte) 0x00, (byte) 0x00, (byte) 0x3c, (byte) 0x13 + , (byte) 0xf4, (byte) 0x03, (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00 + + // ftSbs (currently UnknownSubrecord) + , (byte) 0x0c, (byte) 0x00 + , (byte) 0x14, (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00 + , (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x01, (byte) 0x00, (byte) 0x08, (byte) 0x00 + , (byte) 0x00, (byte) 0x00, (byte) 0x10, (byte) 0x00, (byte) 0x00, (byte) 0x00 + + // ftLbsData (currently UnknownSubrecord) + , (byte) 0x13, (byte) 0x00 + , (byte) 0xee, (byte) 0x1f, (byte) 0x00, (byte) 0x00, (byte) 0x08, (byte) 0x00, (byte) 0x08, (byte) 0x00 + , (byte) 0x01, (byte) 0x03, (byte) 0x00, (byte) 0x00, (byte) 0x0a, (byte) 0x00, (byte) 0x14, (byte) 0x00 + , (byte) 0x6c, (byte) 0x00 + + // ftEnd + , (byte) 0x00, (byte) 0x00, (byte) 0x00, (byte) 0x00 + }; + + public TestSubRecord( String name ) + { + super( name ); + } + + public void testParseCmo() + { + Record r = SubRecord.createSubRecord( (short) 0x0015, (short) 0x0012, dataAutoFilter, 0x0000 ); + assertEquals( "ftCmo is 22 bytes", 22, r.getRecordSize() ); + assertEquals( "ftCmo is a CommonObjectDataSubRecord" + , "org.apache.poi.hssf.record.CommonObjectDataSubRecord" + , r.getClass().getName() ); + } + + public void testParseAutoFilterLbsData() + { + Record r = SubRecord.createSubRecord( (short) 0x0013, (short) 0x1fee, dataAutoFilter, 0x0032 ); + assertEquals( "ftLbsData is 20 bytes", 20, r.getRecordSize() ); + } + + public void testParseEnd() + { + Record r = SubRecord.createSubRecord( (short) 0x0000, (short) 0x0000, dataAutoFilter, 0x0046 ); + assertEquals( "ftEnd is 4 bytes", 4, r.getRecordSize() ); + assertEquals( "ftEnd is a EndSubRecord" + , "org.apache.poi.hssf.record.EndSubRecord" + , r.getClass().getName() ); + } +} \ No newline at end of file
Stuff like this is extremely ugly on more than one level (copy'n'paste from org. apache.poi.hssf.record.RecordFactory.createRecord): --------8<--------------------------------------------------- catch (Exception introspectionException) { introspectionException.printStackTrace(); throw new RecordFormatException("..."); } -------->8--------------------------------------------------- Instead you should do it like this: --------8<--------------------------------------------------- catch (Exception introspectionException) { RecordFormatException rfe = new RecordFormatException("..."); rfe.initCause(introspectionException); throw rfe; } -------->8--------------------------------------------------- Or even better, make a new constructor in RecordFormatException: --------8<--------------------------------------------------- public RecordFormatException(String message, Throwable cause) { super(exception, cause); } [and then] catch (Exception introspectionException) { throw new RecordFormatException("...", introspectionException); } -------->8---------------------------------------------------
I was going to report that using the Data > Filter > Autofilter feature causes the worksheet to fail to load, but it appears to be just one facet of this bug, and a major headache. Is there any way to get around this?
OK, maybe that was a stupid question. What I meant was "Is there any way to work around this without having to actually edit the source, learn how to use Ant, etc." Meantime I managed to edit the source, learn how to use Ant, and get it fixed. Works great, MacOS X, Java 1.4.2, Hooray. Still, I think there are enough people out there who aren't going to get much farther than the limitations page. Maybe this bug (and a workaround, if there is one) should be addressed there? POI rocks. Jakarta rocks. Java rocks.
Using POI 2.5.1 final The error occurs when reading an xls file with a pivot table. I suppose it happens because of the drop down in the pivot table. The ObjRecord-patch27929.java works well for me!!!
Okay... finally applied this one to HEAD. Sorry to be such a slow bastard.
The patch file produces errors if an attempt is made to write to the same excel file as the solution in the patch adds null sub records. A work around is possible by putting null checks toString and serialize methods. The methods are given bellow. public String toString() { StringBuffer buffer = new StringBuffer(); buffer.append("[OBJ]\n"); for ( Iterator iterator = subrecords.iterator(); iterator.hasNext(); ) { Record record = (Record) iterator.next(); if(record != null) { buffer.append("SUBRECORD: " + record.toString()); } } buffer.append("[/OBJ]\n"); return buffer.toString(); } public int serialize(int offset, byte[] data) { int pos = 0; LittleEndian.putShort(data, 0 + offset, sid); LittleEndian.putShort(data, 2 + offset, (short)(getRecordSize() - 4)); pos = offset + 4; for ( Iterator iterator = subrecords.iterator(); iterator.hasNext(); ) { Record record = (Record) iterator.next(); if(record != null) { pos += record.serialize(pos, data); } } return getRecordSize(); }