Hi,

I have created some patches which fix some 
bugs and improve the performance of the vcscore. 

These patches reduce the frequency of cache 
updates to and from the disk, and keep the memory 
footprint of cache small.  The number of file 
system refreshes is also reduced.

We are working on a VCS profile for vcsgeneric 
(for our proprietary code management system (CMS) )
and these patches are required. 

If at all possible, we'd like to be able to have 
these patches reviewed and incorporated into 3.6.  
Could someone review them ASAP and get back to us?

Related nbdev link :
http://www.netbeans.org/servlets/ReadMsg?msgId=698985&listName=nbdev

Thanks

/Isao Yanagimahci
Hewlett-Packard

Created attachment 13810 [details]
Patch file for CacheDir.java

Created attachment 13811 [details]
Patch file for CacheReference.java

Created attachment 13812 [details]
Patch file for ExcuteCommand.java

Created attachment 13813 [details]
Patch file for VcsCacheDir.java

Created attachment 13814 [details]
Patch file for VcsCacheFile.java

Created attachment 13815 [details]
Patch file for VcsFileSystem.java

Thanks Isao, starting to look at it...

I've reviewed the patch. It looks good, especially the optimized
refreshing in ExecuteCommand and cache reading in CacheDir.

I've made some minor correction in CacheFile.getFiles() and removed
all methods which manipulated with siblings from VcsCacheFile and
VcsCacheDir. I'm a little worried of the change from WeakReference to
SoftReference, but when I test both options I didn't encounter a
significant difference in memory consumption. Since SoftReference
should the cache performance, I'll leave there SoftReference.

I've tested the patch with CVS profile and I didn't encounter any
functional problems. I didn't measure the performance, but during the
tests it seemed to be faster.

I've committed the patch into trunk and I'll try to get it into 3.6.

/cvs/vcscore/src/org/netbeans/modules/vcscore/VcsFileSystem.java,v 
<--  VcsFileSystem.java
new revision: 1.250; previous revision: 1.249
done
Checking in cache/CacheDir.java;
/cvs/vcscore/src/org/netbeans/modules/vcscore/cache/CacheDir.java,v 
<--  CacheDir.java
new revision: 1.35; previous revision: 1.34
done
Checking in cache/CacheReference.java;
/cvs/vcscore/src/org/netbeans/modules/vcscore/cache/CacheReference.java,v
 <--  CacheReference.java
new revision: 1.9; previous revision: 1.8
done
Checking in caching/VcsCacheDir.java;
/cvs/vcscore/src/org/netbeans/modules/vcscore/caching/VcsCacheDir.java,v
 <--  VcsCacheDir.java
new revision: 1.49; previous revision: 1.48
done
Checking in caching/VcsCacheFile.java;
/cvs/vcscore/src/org/netbeans/modules/vcscore/caching/VcsCacheFile.java,v
 <--  VcsCacheFile.java
new revision: 1.18; previous revision: 1.17
done
Checking in cmdline/ExecuteCommand.java;
/cvs/vcscore/src/org/netbeans/modules/vcscore/cmdline/ExecuteCommand.java,v
 <--  ExecuteCommand.java
new revision: 1.74; previous revision: 1.73

Oops, Target Milestone must be "promo-D" until it's merged into
release36 branch.

Created attachment 13827 [details]
The final contextual diff.

thanks Isao.

Do we have any measurable case where the improvement can be demonstrated?

Ad Soft vs. Weak references: soft refs are cleared when there is a
demand for more memory so it will not reclaim objects so easily. When
the weak refs are freed and how soft refs help? Do we now?

I am eager to see as many performance enhancements in the IDE as
possible but would like to see some hard numbers.

We can give you some performance numbers for our generic profile for 
CMS.  The difference is astonishing really.  Isao will post them 
this afternoon.

Thanks for consideration.

Meg

>Do we have any measurable case where the improvement can be 
>demonstrated

yes.

Environment : P4(2.4 2CPU) 1G RAM machine with XP. The files were  
              under the network mounted share. 

* With all the vcscore patches applied :

     Refreshing 1500 files took about 5-6 sec. Updating 750 files 
     from scratch took about 34 sec.

* Without patches being applied :

     Refreshing 1500 files caused out of memory exception from 
     NetBeans after a while. Updating 750 files from scratch took 
     about 45  sec.

Basically I observed more performance gain as the number of files 
increased.

>When the weak refs are freed and how soft refs help? Do we now?

With just the Refernece patch alone, and refreshing about 800 files,
it took at least 4 more secs to refresh using WeakReference versus 
SoftReference. I think this is because refreshing in vcscore is 
expensive in that once the file reference is cleared, it requires 
buffered file IO to recreate the cache.  And when refreshing many 
number of files in vcscore , many of the file references are cleared 
and reclaimed. So IMHO at least in the case of vcscore where 
reclaiming file references is expensive, I think SoftReference has 
advantage.


Isao

QA, please, verify the fix in trunk build. Thanks.

I verified with trunk builds that the improvement for Refresh is
significant. Having a local CVS repository with 1500 simple java files
in a single directory I did refresh. With a build prior to this patch
I ran into OOME and after 51 seconds the refresh command failed
completely. With a build with the patch the refresh finished
successfully within 15 seconds.

Thanks for the verification.
The fix is merged into release36 branch:

Checking in src/org/netbeans/modules/vcscore/VcsFileSystem.java;
/shared/data/helm/cvs/repository/vcscore/src/org/netbeans/modules/vcscore/VcsFileSystem.java,v
 <--  VcsFileSystem.java
new revision: 1.249.2.1; previous revision: 1.249
done
Checking in src/org/netbeans/modules/vcscore/cache/CacheDir.java;
/shared/data/helm/cvs/repository/vcscore/src/org/netbeans/modules/vcscore/cache/CacheDir.java,v
 <--  CacheDir.java
new revision: 1.34.18.1; previous revision: 1.34
done
Checking in src/org/netbeans/modules/vcscore/cache/CacheReference.java;
/shared/data/helm/cvs/repository/vcscore/src/org/netbeans/modules/vcscore/cache/CacheReference.java,v
 <--  CacheReference.java
new revision: 1.8.150.1; previous revision: 1.8
done
Checking in src/org/netbeans/modules/vcscore/caching/VcsCacheDir.java;
/shared/data/helm/cvs/repository/vcscore/src/org/netbeans/modules/vcscore/caching/VcsCacheDir.java,v
 <--  VcsCacheDir.java
new revision: 1.48.2.1; previous revision: 1.48
done
Checking in src/org/netbeans/modules/vcscore/caching/VcsCacheFile.java;
/shared/data/helm/cvs/repository/vcscore/src/org/netbeans/modules/vcscore/caching/VcsCacheFile.java,v
 <--  VcsCacheFile.java
new revision: 1.17.2.1; previous revision: 1.17
done
Checking in src/org/netbeans/modules/vcscore/cmdline/ExecuteCommand.java;
/shared/data/helm/cvs/repository/vcscore/src/org/netbeans/modules/vcscore/cmdline/ExecuteCommand.java,v
 <--  ExecuteCommand.java
new revision: 1.73.2.1; previous revision: 1.73
done