Issue 55266 - DataPilot uses large amounts of memory when selecting multiple "row fields"
Summary: DataPilot uses large amounts of memory when selecting multiple "row fields"
Status: CLOSED FIXED
Alias: None
Product: Calc
Classification: Application
Component: open-import (show other issues)
Version: OOO 2.0 Beta2
Hardware: PC All
: P3 Trivial with 11 votes (vote)
Target Milestone: ---
Assignee: oc
QA Contact: issues@sc
URL:
Keywords: crash, oooqa
: 69302 77341 79948 (view as issue list)
Depends on:
Blocks:
 
Reported: 2005-09-30 00:39 UTC by cpinney
Modified: 2017-05-20 10:33 UTC (History)
7 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---


Attachments
DataPilot sample, try changing zip code, Windows memory usage increases over 600MB (80.08 KB, application/vnd.sun.xml.calc)
2005-09-30 00:42 UTC, cpinney
no flags Details
Example file with grouped items, which makes problems (19.74 KB, application/vnd.oasis.opendocument.spreadsheet)
2009-11-02 16:01 UTC, niklas.nebel
no flags Details
This file contains two DataPilot tables with numeric groups (12.89 KB, application/vnd.oasis.opendocument.spreadsheet)
2009-11-11 13:59 UTC, niklas.nebel
no flags Details
Example with date grouping (12.29 KB, application/vnd.oasis.opendocument.spreadsheet)
2009-11-16 19:02 UTC, niklas.nebel
no flags Details
Example file with hidden details for some items (20.17 KB, application/vnd.oasis.opendocument.spreadsheet)
2009-11-25 18:30 UTC, niklas.nebel
no flags Details

Note You need to log in before you can comment on or make changes to this issue.
Description cpinney 2005-09-30 00:39:50 UTC
Configured a 5000 row spreadsheet placing zip code in "Page fields", Customer
name, number and city in "Row fields". Windows XP Home ran out of memory after
2GB and closed Open Office. System is a Dell Dimmension 3000 P4, 2.8GHZ, 1GB
Ram. Reduced file to 3000 rows, and I am able to use DataPilot after about 1
minute. Windows reports soffice.bin as using 696,932K memory (the spreadsheet is
only 81Kb).
Comment 1 cpinney 2005-09-30 00:42:32 UTC
Created attachment 30016 [details]
DataPilot sample, try changing zip code, Windows memory usage increases over 600MB
Comment 2 frank 2005-10-17 14:24:50 UTC
Hi Niklas,

have a look at this one please. Re-target as needed.

Frank
Comment 3 niklas.nebel 2006-09-07 14:24:17 UTC
*** Issue 69302 has been marked as a duplicate of this issue. ***
Comment 4 kpalagin 2006-09-28 13:17:52 UTC
Confirming with m180 and 2.0.4 RC2 on Windows - just clicking arrow in cell B2 
of "DataPilot_Sample.ods" makes soffice.bin start consuming RAM at rate of 
several MB/s. Once it hits 800MB I have to kill the process to regain control 
over the machine.
Any chance of targeting this for 2.1?
Comment 5 frank 2006-09-28 13:26:09 UTC
hi kpalagin,

please don't change the version field of any Issue except your are asked for it
by a developer. This field defines the first occurens of the defect and is
therefore an important information. If an Issue is assigned to a developer, he
has it on his schedule, so if you can give valuable information, just add a
comment but do not change any fields.

Thanks for your understanding.

Frank 
Comment 6 frank 2007-07-06 14:54:46 UTC
*** Issue 77341 has been marked as a duplicate of this issue. ***
Comment 7 niklas.nebel 2007-07-25 15:23:09 UTC
*** Issue 79948 has been marked as a duplicate of this issue. ***
Comment 8 kpalagin 2008-02-27 13:14:09 UTC
Any chance for 3.0?
Comment 9 canis 2008-04-05 11:48:22 UTC
Hm, I was bet on high priority of such issues. Because of this Calc CRASHES
working with pivot tables. Is not this the sufficient cause for solving this issue?
Comment 10 oopla 2008-09-11 18:44:07 UTC
hello,

any chances this issue gets fixed anytime soon? 2.4.1 is still unusable because 
of this bug (~10krows -> datapilot -> boom: oom a gogo, may even bring down 
the whole multi-user machine).

Pls note that pivot-table/datapilot is a critical feature for any serious calc 
in biz (and elsewhere). I had high hopes for 2.0.x which seemed to work where 
1.x failed on a few 100s rows, but since then all the pivot stuff is left to
externals running MS-X (or else, for that matter).

Please rise P to no less than P2, this issue is a showstopper for many users,
and 3+ years old.
Comment 11 oopla 2008-09-15 17:54:08 UTC
tried OOO300m5 build 9350:

- calc starts as a 130MB (vm) process
- after recovering a 18000 rows by 6 cols + datapilot sheet .ods the vm usage
  is ~173 MB
- refresh datapilot table ... vm skyrockets quickly consuming the 1GB RAM, then
  swapper kicks in till the whole thing hangs at ~98% cpu, 1600MB vm.
- killed after a few mins before going into troubles.

That's on Debian Etch (Linux x86 2.6.26.5, libc 2.3.6)
Comment 12 Rainer Bielefeld 2008-10-29 11:57:39 UTC
Modify OS due to comments from oopla Mon Sep 15.

Still a problem with "Ooo 3.0.0 RC3 Multilingual version German UI WIN XP:
[OOO300m8 (Build9357)]" 

Issue 95509 seems to have the same roots like this one.
I do not believe that "Later" is an appropriate TM for this issue
Comment 13 kpalagin 2008-12-18 08:33:17 UTC
FWIW, technically it is not a crash as Calc is able to complete operation once 
it allocates more than 1GB of RAM.
Nonetheless, I do not think that such behavior is acceptable.

Niklas,
please see if you can make Calc use less RAM for this trivial task.

Thanks a lot.
Comment 14 entityseven3 2009-01-25 16:23:33 UTC
I have encountered a similar problem with DataPilot in OOo 3.0.0 (Windows XP
SP3) when trying to group a small amount of data (4124 rows at 7 columns per
row). I took two rows for the filter (page fields) and placed two rows into the
row fields. No rows in any other fields. When clicking OK the CPU load is going
up to 100% and OOo is starting consuming memory up to 1.5GB. This takes only a
few seconds. Then a system sound is played and one time a error message was
displayed from DataPilot, but only once. The other times I reproduced the issue
no message box was displayed, only the sound. The only way to close Calc now is
to kill the process. The hardware is a Toshiba Tecra with 2GB of RAM and an
1.8GHz Intel Centrino.  As oopla wrote, this is a no-go for using OOo Calc in
any serious data processing, where PivotTables are daily business. I can supply
a sample file if needed.
Comment 15 niklas.nebel 2009-11-02 16:01:06 UTC
Created attachment 65869 [details]
Example file with grouped items, which makes problems
Comment 16 niklas.nebel 2009-11-11 13:59:57 UTC
Created attachment 66065 [details]
This file contains two DataPilot tables with numeric groups
Comment 17 niklas.nebel 2009-11-16 19:02:35 UTC
Created attachment 66142 [details]
Example with date grouping
Comment 18 niklas.nebel 2009-11-25 18:30:27 UTC
Created attachment 66345 [details]
Example file with hidden details for some items
Comment 19 niklas.nebel 2010-02-09 14:45:40 UTC
Fixed in CWS "datapilotperf". See also
http://wiki.services.openoffice.org/wiki/Calc/Proposal_DataPilot_byIBM
Comment 20 niklas.nebel 2010-02-09 14:46:35 UTC
Reassigning to QA for verification.
Comment 21 oc 2010-03-16 15:28:31 UTC
verified in internal build cws_datapilotperf
Comment 22 wope 2010-05-01 17:48:17 UTC
ok in DEV300_m76, issue closing
Comment 23 compkupleiplet 2010-11-10 16:33:09 UTC
Created attachment 73218