This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Measure the recognition process with ~1000 folders and ~1000 files. The recognition must be fast enough so that the user is not blocked. Some faster recognition process could be introduced based on pattern matching, etc.
Created attachment 10593 [details] OptimizeIt output - expanded node in explorer, that contained 1000 of folders
I've made some rough measurment (just by hand without any performance tool) on the projects build and with masterfs implementation on a 2.53GHz Intel machine. I've tested expansion of a folder with: a) 1000 empty folders, b) 1000 folders with CVS directories and c) 1000 folders with CVS directories with Repository == directory name a) was under 1 second, no CVS filesystem was recognize, IMHO there is no significand performance problem, but I have to compare it with VCS modules turned off. b) was about 3-4 seconds, the time varied (it was also ~7 seconds sometimes) due to intermittent GC and system caching of files. This is an artificial case, because one will unlikely have 1000 different CVS working directories. c) was about 10 seconds and more, the time varied (it was also ~20 seconds sometimes) due to intermittent GC (this is a memory-intensive operation), I had to increase -J-Xmx option to prevent OOME. This mimics the nb_all approach (you do not have CVS folders in nb_all, but CVS folders are in every sub-folder). There's a problem, that after the CVS fileystem is explicitly mounted to nb_all folder, all sub-folders are inspected once more. I will do some more tests, the implementation of c) needs to be changed, or the explicit mount will be removed if the performance can not be made satisfactory. This will mean, that nb_all will not be recognized :-(((.
I've made some more tests of the recognition process. The point is, that the recognition process as such does not bring a significant overhead. What makes the overhead is the VCS filesystem. The tests concentrated mainly on case 'c)', because this step has the worst performance. This step first creates the VCS filesystem (about 1 sec), mount it to masterfs (9-12 sec, this has caused refreshing of the VCS FS, which takes long time), run recognizers for all 1000 sub-folders (~ 1sec) then the folder content is shown in the Explorer and some final refreshing of VCS FS lasts for a few seconds. From this it's clear, that the biggest performance problem is in VcsFileSystem.children() call, which does extensive operations on cache files and cause the mount to masterfs to take a significant amount of time. Another case was also tested: d) one folder with CVS directory (recognized and mounted as CVS filesystem) with 1000 sub-directories. Expansion of this folder took ~ 6-7 seconds. The autorecognition process is very fast in this case and most of the time is consumed by the VCS filesystem itself. I've made some minor improvements to the auto-recognition process: CVS_DIR + File.separator + CVS_ROOT is pre-computed and stored in a static field: /cvs/vcsgeneric/profiles/cvsprofiles/src/org/netbeans/modules/vcs/profiles/cvsprofiles/commands/CvsFSAutorecognizer.java,v <-- CvsFSAutorecognizer.java new revision: 1.2.50.1; previous revision: 1.2 The mounting process is not spawned in RequestProcessor, but executed directly in "isRootOfFileSystem()" call. This has caused deadlock on old localfs implementation, but seems to work O.K. in masterfs implementation. This improves the performance, because the recognition process is not run twice for the sub-folders: /cvs/vcscore/src/org/netbeans/modules/vcscore/registry/Attic/VcsFSProvider.java,v <-- VcsFSProvider.java new revision: 1.1.2.11; previous revision: 1.1.2.10 IMHO the auto-recognition process as such can not be much further improved. The biggest performance problems are caused by the implementation of VcsFilesystem and it's file status caching system. Several thread dumps are going to be attached, that demonstrate this.
Created attachment 10838 [details] Several thread dumps, made while VCS filesystem was being mounted, also some refreshes after the mounting was finished are catched.
Really leads to deadlock (see #34698): /cvs/vcscore/src/org/netbeans/modules/vcscore/registry/Attic/VcsFSProvider.java,v <-- VcsFSProvider.java new revision: 1.1.2.11; previous revision: 1.1.2.10 There is not intended to call MountSupport.mount in isRootOfFileSystem. You should wait until method createFileSystem is called. But I you think, that this bring radically improved performance, then I'll strive to fix somehow deadlock on my side first.
Adding dependency on issue #35512. findExistingResource() method will speed up FileObjects refreshing.
There will be no auto-recognition in promotion D. Scheduling for future.
No autorecognition => INVALID.