Bug 123872 - Reduce memory consumption for huge projects
Summary: Reduce memory consumption for huge projects
Alias: None
Product: cnd
Classification: Unclassified
Component: Code Model (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker (vote)
Assignee: Vladimir Voskresensky
: 120036 139352 141303 186765 (view as bug list)
Depends on: 89648 120042 120168 122313 135585 197297 215226
Blocks: 108501
  Show dependency tree
Reported: 2007-12-12 11:34 UTC by Alexander Simon
Modified: 2013-01-15 11:26 UTC (History)
2 users (show)

See Also:
Issue Type: TASK
Exception Reporter:


Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Simon 2007-12-12 11:34:21 UTC
Target is opening OpenSolaris priject in 1 Gb Xmx memory
Comment 1 Alexander Simon 2007-12-12 14:18:01 UTC
-use one instance of equal strings at project reading. It save 1% of memory.

CVS log:
Checking in ConfigurationXMLCodec.java;
 <--  ConfigurationXMLCodec.java
new revision: 1.14; previous revision: 1.13
Comment 2 Alexander Simon 2007-12-12 15:38:07 UTC
- make lazy cookies (CppEditorSupport, BinaryExecSupport) in CndDataObject. It save 3,6% of memory.

CVS log:

Checking in highlight/src/org/netbeans/modules/cnd/highlight/error/HighlightProvider.java;
/shared/data/ccvs/repository/cnd/highlight/src/org/netbeans/modules/cnd/highlight/error/HighlightProvider.java,v  <-- 
new revision: 1.10; previous revision: 1.9
Checking in highlight/nbproject/project.xml;
/shared/data/ccvs/repository/cnd/highlight/nbproject/project.xml,v  <--  project.xml
new revision: 1.19; previous revision: 1.18
Checking in core/src/org/netbeans/modules/cnd/loaders/HDataObject.java;
/shared/data/ccvs/repository/cnd/core/src/org/netbeans/modules/cnd/loaders/HDataObject.java,v  <--  HDataObject.java
new revision: 1.6; previous revision: 1.5
Checking in core/src/org/netbeans/modules/cnd/loaders/CndDataObject.java;
/shared/data/ccvs/repository/cnd/core/src/org/netbeans/modules/cnd/loaders/CndDataObject.java,v  <--  CndDataObject.java
new revision: 1.8; previous revision: 1.7
Comment 3 Alexander Simon 2007-12-13 07:52:34 UTC
- make lazy ItemComfiguration initialization. It saves 3,4% of memory.

CVS log:
Checking in ItemConfiguration.java;
 <--  ItemConfiguration.java
new revision: 1.12; previous revision: 1.11
Comment 4 Alexander Simon 2007-12-14 11:44:24 UTC
-own unique name key, cache compound names in renderer, cache canonical names. It save 8% of memory.

CVS log:
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/AstUtil.java;
/shared/data/ccvs/repository/cnd/modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/AstUtil.java,v  <--  AstUtil.java
new revision: 1.12; previous revision: 1.11
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/DeclarationContainer.java;
<--  DeclarationContainer.java
new revision: 1.10; previous revision: 1.9
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/FileContainer.java;
/shared/data/ccvs/repository/cnd/modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/FileContainer.java,v  <-- 
new revision: 1.18; previous revision: 1.17
RCS file: /shared/data/ccvs/repository/cnd/modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/FixedString.java,v
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/FixedString.java;
/shared/data/ccvs/repository/cnd/modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/FixedString.java,v  <-- 
initial revision: 1.1
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/AstRenderer.java;
/shared/data/ccvs/repository/cnd/modelimpl/src/org/netbeans/modules/cnd/modelimpl/csm/core/AstRenderer.java,v  <-- 
new revision: 1.27; previous revision: 1.26
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/repository/OffsetableDeclarationKey.java;
 <--  OffsetableDeclarationKey.java
new revision: 1.5; previous revision: 1.4
Checking in repository/src/org/netbeans/modules/cnd/repository/sfs/SimpleFileIndex.java;
/shared/data/ccvs/repository/cnd/repository/src/org/netbeans/modules/cnd/repository/sfs/SimpleFileIndex.java,v  <-- 
new revision: 1.7; previous revision: 1.6
Checking in modelimpl/src/org/netbeans/modules/cnd/modelimpl/uid/UIDObjectFactory.java;
/shared/data/ccvs/repository/cnd/modelimpl/src/org/netbeans/modules/cnd/modelimpl/uid/UIDObjectFactory.java,v  <-- 
new revision: 1.10; previous revision: 1.9
Comment 5 Alexander Simon 2008-03-13 18:23:33 UTC
In NB 6.1 memory was reduced on 16%.
Next memory reducing can be achieved by:
- use char sequence in ANTLR & APT
- keep part of repository keys in memory
- use number for repository keys
Comment 6 Alexander Simon 2008-06-27 10:47:44 UTC
Save 2 bytes on each object:
Comment 7 Vladimir Kvashin 2008-07-23 11:15:45 UTC
*** Issue 141303 has been marked as a duplicate of this issue. ***
Comment 8 Vladimir Kvashin 2008-07-23 11:16:16 UTC
*** Issue 120036 has been marked as a duplicate of this issue. ***
Comment 9 Vladimir Kvashin 2008-07-23 11:34:38 UTC
*** Issue 139352 has been marked as a duplicate of this issue. ***
Comment 10 Vladimir Kvashin 2008-07-23 11:34:42 UTC
Comments from another issue by philbyjohn@netbeans.org
that is made a dup pf this one

OS: Fedora8 with Kernel
System: AMD Turion on Toshiba Satellite A215

Problem Also noted on 
OS: Windows 2000 Professional
System: 2.8GHz IBM Think Centre 8434
RAM: 777,200KB

I tried to setup a project giving it a Makefile from the latest Linux kernel I had downloaded. But NetBeans6.1 just
hangs. NetBeans takes up huge memory and it parses the project every time you open it. NetBeans hangs while showing
"Parsing LinuxKernel ..." at 11% on status bar and the Memory shows 335.2/337.4MB.

Indeed I was able to resolve the hanging issue on both Linux Fedora8 and Windows2000 with the -J-Xmx700m and
-J-XX:PermSize=96m set in 
etc/netbeans.conf. But the IDE is non responsive to mouse clicks and is very sluggish.

Compared to an agile editor such as Source Insight, NetBeans is way off when it comes to speed. It took exactly 47
minutes for the parsing to 
complete on Linux and 32 minutes on a Windows system. Whereas Source Insight does the job in 8 minutes flat on Linux
(using wine) and 5 
minutes on Windows.

Outline of the steps that I had followed.
1)Go to http://www.kernel.org/
2)Download the latest Linux Kernel (as I type 2.6.26) by clicking on the "F" link which stands for "full source".
3)Extract and create a new project by specifying the "Makefile" in the directory "linux-2.6.26"

Also, I am Using NetBeans IDE 6.1, JDK Update 5 all contained in the free installation DVD available on request.

VK: I'd like to note that the project is large: it has 6,608,795 lines in 19,800 files (both header and source)
VK: With -Xmx1563m I got it parsed within 7 minutes (on multy-CPU machine, but with one parsing thread). NB memory
indicator shows 870/1200 Mb after parsing is finished.

Comment 11 Vladimir Voskresensky 2012-07-31 12:23:35 UTC
Nb 7.2 needs 1.7Gb of memory to parse Mozilla sources. 
7.3 should use less than 1Gb
Comment 12 Vladimir Voskresensky 2012-07-31 12:33:28 UTC
*** Bug 186765 has been marked as a duplicate of this bug. ***
Comment 13 Vladimir Voskresensky 2012-09-09 18:29:55 UTC
parse Mozilla sources - 700mb
parse Linux sources - 1470mb