Issue 109940 - DataPilot´s result is restricted to 2^14 rows
Summary: DataPilot´s result is restricted to 2^14 rows
Alias: None
Product: Calc
Classification: Application
Component: code (show other issues)
Version: OOo 3.2
Hardware: PC All
: P3 Trivial (vote)
Target Milestone: ---
Assignee: AOO issues mailing list
QA Contact:
Depends on:
Reported: 2010-03-08 15:46 UTC by ohallot
Modified: 2013-01-29 21:52 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---

Data pilot defect sample (145.96 KB, application/vnd.oasis.opendocument.spreadsheet)
2010-03-08 15:48 UTC, ohallot
no flags Details
testcase (592.88 KB, text/plain)
2010-03-30 16:43 UTC, raal
no flags Details

Note You need to log in before you can comment on or make changes to this issue.
Description ohallot 2010-03-08 15:46:16 UTC
The attached data pilot table (pivot table), contains a extreme situation that
leads to error in with no warning.

The source sheet contains 20305 entries, in column A (header is BNAME)

The data pilot is built in the following manner

1)select all cells in BNAME
2)Start the data pilot
3)Drag BNAME in the line field (on the left)
4)drag BNAME in the data field (center)
5)Select COUNT in Options
6)Click in More... and assign a new sheet for the data pilot.
7)Click OK

The resulting data pilot, will be placed in a new sheet.

We are not concerned on the results of the data pilot (column B). It does count
the number of similar occurences of BNAME entries.

We think there is an error because the data pilot size is 16387 lines and some
entries of the source sheet in BNAME are missing, with no warning (look at the
last ones in the source, they do not show in the data pilot).

Please advise if this is wornking as expected. We think we have a DEFECT.

We must emphasize that the data pilot came from an EXCEL 2003 spreadsheet, and
Excel pivot table does show all entries in BNAME.
Comment 1 ohallot 2010-03-08 15:48:03 UTC
Created attachment 68220 [details]
Data pilot defect sample
Comment 2 gibi 2010-03-29 17:44:36 UTC
Perhaps I didn't understand the issue but imha the result is ok as far as there
are duplicate values of BNAME (for example, 2 x N1AW, 2 x N1B1, and so on).

So what your Data Pilot is asked for is to give one line per item.
You can verify that the number is correct by comparison with a Standard filter
on your source with:
- column BNAME = 'non empty'
- More Options: 
         - 'no duplicate'
         - 'Copy result to ...' <free cells>

=> CountA on the filtered result in the status bar give 16385 lines (including
title 'BNAME')
Comment 3 ohallot 2010-03-29 18:24:33 UTC
"duplicate values of BNAME (for example, 2 x N1AW, 2 x N1B1, and so on)."

No. A more in-depth look is needed here.

If you browse the last lines of the source you will see entries that are not
listed in the data pilot.

for example:

SOG1	1

while the data pilot goes only up to 

SJD1	1
SJK2	2
SK01	1

circa line 16387.


Comment 4 raal 2010-03-30 16:42:04 UTC
Reproducible with version 3.2
I will attach another test case. Seems that datapilot has some limit 16386 rows.
Dataloss without warning, please consider target 3.2.1.

Attachment - sheet List1, column D "reproduce" contains unique records.
Sheet "Průvodce daty_List1_2" is data pilot from column List1.D
There is 27722 rows on sheet List1.
DataPilot takes only 16384 rows.
Comment 5 raal 2010-03-30 16:43:17 UTC
Created attachment 68640 [details]
Comment 6 Stefan Weigel 2010-04-14 08:44:12 UTC
Confirming restriction of DataPilot´s result to 2^14 rows. Result ist cut off
without notice.

DEFECT: You get a wrong result without warning.

However, this is *not* a data loss.

Suggested solutions:
(1) Give warning, when DataPilot reaches maximum number of possible rows/columns.
(2) Extend restriction
Comment 7 niklas.nebel 2010-04-14 10:10:20 UTC
This is no longer a problem in DEV300m76, where ScDPTableDataCache handles the
entries in a different way. But that change is too big for 3.2.1.
Comment 8 gibi 2010-04-14 10:11:20 UTC
There is a loss of *information* and a *wrong* result so how could we trust Data
Pilot in a business use?
It's a critical blocker.

Tested with 3.1.1 isolating possible issue with the "DataPilot-defect.ods" file
itself by doing this:
- opening DataPilot-defect.ods with OOo 3.1.1 Win XP
- in source tab, selecting cells with CTRL * and copy
- creating a new spreadsheet then paste
- following the procedure explained by ohallot 

Result is: same error than in 3.2