This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 208185 - Scanning Projects takes more than 2-3 minutes to complete
Summary: Scanning Projects takes more than 2-3 minutes to complete
Status: RESOLVED FIXED
Alias: None
Product: versioncontrol
Classification: Unclassified
Component: Subversion (show other bugs)
Version: 7.1
Hardware: PC Windows 7
: P2 normal (vote)
Assignee: Ondrej Vrabec
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-09 09:00 UTC by stefanp_safilo
Modified: 2013-03-07 01:57 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
My messages.log (34.12 KB, text/plain)
2012-07-13 07:58 UTC, stefanp_safilo
Details
Self profiling snapshot (266.17 KB, application/octet-stream)
2012-07-13 08:25 UTC, stefanp_safilo
Details
Messages log file (70.24 KB, text/plain)
2012-07-13 09:00 UTC, stefanp_safilo
Details
Zipped messages.log file (with higher logging level) (1.03 MB, application/octet-stream)
2012-07-23 13:47 UTC, stefanp_safilo
Details
Self profiling snapshot (2.57 MB, application/octet-stream)
2013-03-05 11:16 UTC, stefanp_safilo
Details

Note You need to log in before you can comment on or make changes to this bug.
Description stefanp_safilo 2012-02-09 09:00:44 UTC
At startup, the "scanning projects" task takes a very long and annoying time to complete. 
At times it takes more than 3 minutes on a Windows7/2.66 Ghz Quad core/4Gb ram PC.
And until it's finished, code completion and file upload won't work.
Comment 1 stefanp_safilo 2012-02-09 09:04:20 UTC
On a typical scenario, the IDE has from 4 to 7 projects open, all of them on the C:\ hard drive.
Comment 2 Marian Mirilovic 2012-02-15 12:45:13 UTC
Could you share the projects with us ?
Comment 3 stefanp_safilo 2012-02-15 12:47:28 UTC
(In reply to comment #2)
> Could you share the projects with us ?

I'm afraid I can't because they are owned by the company I work for, but if you need other info, please ask me and (if I can) I'll answer.
Comment 4 Petr Cyhelsky 2012-02-15 12:53:03 UTC
Following will be great:
1. your messages.log file (to know the details of your setup)
http://wiki.netbeans.org/FaqLogMessagesFile
2. Self-profile snapshot of scanning of the project(s)
create it vua those steps: start the ide with empty userdir(and cachedir if you are using the newest dev builds) activate all the clusters needed for your projects. Start self-profiling, open your projects, wait till all the scanning is finished, stop the self-profiling.
http://wiki.netbeans.org/FitnessViaPartnership
Comment 5 stefanp_safilo 2012-07-13 07:58:49 UTC
Created attachment 121995 [details]
My messages.log
Comment 6 stefanp_safilo 2012-07-13 08:02:27 UTC
How can I do self-profiling? 
If I select Profile->Attach profiler, there is no way (nor any reference) to "self profile" the IDE.
I wish it could be simpler, both for users and for developers, to track bugs in Netbeans...
Comment 7 Tomas Hurka 2012-07-13 08:04:09 UTC
(In reply to comment #6)
> How can I do self-profiling? 
See http://wiki.netbeans.org/FitnessViaPartnership - it is very simple.
Comment 8 stefanp_safilo 2012-07-13 08:25:07 UTC
Created attachment 121997 [details]
Self profiling snapshot
Comment 9 Petr Cyhelsky 2012-07-13 08:58:28 UTC
~260s of the time spent in scanning is spent in native filesystem methods like
java.io.WinNTFileSystem.getBooleanAttributes
java.io.WinNTFileSystem.list
java.io.WinNTFileSystem.getLastModifiedTime
there are some possibilities what could have caused this (generally slow disc, antivirus, very fragmented filesystem, very very large project)
can you please attach new messages log (it has to be from the time of the long scanning slowness otherwise there is nothing relevant in the log)
Comment 10 stefanp_safilo 2012-07-13 09:00:10 UTC
Created attachment 121998 [details]
Messages log file
Comment 11 Petr Cyhelsky 2012-07-16 11:28:49 UTC
there seems to be disc access by parsing and subversion simultaneously, subversion guys please evaluate
Comment 12 bht 2012-07-21 21:43:32 UTC
I found another case of possible simultaneous disk access in bug 215792
Comment 13 Ondrej Vrabec 2012-07-23 11:46:50 UTC
Please run the IDE with -J-Dorg.netbeans.modules.subversion.level=500 and attach the messages log once the scanning finishes.
Comment 14 stefanp_safilo 2012-07-23 11:50:25 UTC
(In reply to comment #13)
> Please run the IDE with -J-Dorg.netbeans.modules.subversion.level=500 and
> attach the messages log once the scanning finishes.

Where am I supposed to write -J-Dorg.netbeans.modules.subversion.level=500 ? Should I run it from the command line?

Again, should I delete messages.log first?

May I ask what this option does to both the IDE and the message log?
Comment 15 Ondrej Vrabec 2012-07-23 12:08:29 UTC
(In reply to comment #14)
> Where am I supposed to write -J-Dorg.netbeans.modules.subversion.level=500 ?
> Should I run it from the command line?
Yes, in commandline, go to the install dir/bin and run 'netbeans -J-Dorg.netbeans.modules.subversion.level=500'
> Again, should I delete messages.log first?
no need, it will be overwritten in the var/log folder. Simply attach the latest one messages.log file.
> May I ask what this option does to both the IDE and the message log?
it enables more logging. The message log will be bigger. And we will see if subversion waits for the parsing to finish. Generally, subversion waits the first 180s of the initial parsing/scanning process and does not get in the way of the scanning. After 180 seconds it starts its own scanning no matter if the parsing of sources finishes or not.
Comment 16 stefanp_safilo 2012-07-23 13:47:51 UTC
Created attachment 122271 [details]
Zipped messages.log file (with higher logging level)
Comment 17 Ondrej Vrabec 2012-07-23 14:08:40 UTC
hmm, the messages log does not indicate that subversion refresh has been delayed. Will try to tweak it.
Comment 18 Ondrej Vrabec 2012-07-23 14:21:36 UTC
fix: http://hg.netbeans.org/core-main/rev/fc1e8d83a60b - fixes also bug #211634
Comment 19 stefanp_safilo 2012-07-23 14:30:08 UTC
(In reply to comment #18)
> fix: http://hg.netbeans.org/core-main/rev/fc1e8d83a60b - fixes also bug #211634

Sorry about my ignorance. Does that mean that if I download v7.1.2 and install it, this issue should disappear?
Comment 20 Ondrej Vrabec 2012-07-23 14:44:06 UTC
> Sorry about my ignorance. Does that mean that if I download v7.1.2 and install
> it, this issue should disappear?
No, the Target Milestone field for this issue is set for 7.3. That means the fix will be delivered in the 7.3 release and available in daily builds soon.
Comment 21 stefanp_safilo 2013-03-05 11:16:10 UTC
Created attachment 132197 [details]
Self profiling snapshot

Just got the same problem as before. Same machine, same OS.
Comment 22 stefanp_safilo 2013-03-05 11:17:15 UTC
And the installed version is the new 7.3

(In reply to comment #21)
> Created attachment 132197 [details]
> Self profiling snapshot
> 
> Just got the same problem as before. Same machine, same OS.
Comment 23 Ondrej Vrabec 2013-03-05 11:50:45 UTC
looking at the snapshot i came to the following conclusion:
for the first ~150s the only running RP is RepositoryUpdater.Worker and Git is successfully blocked. After that time the VCS subsyetem cannot wait any longer (there must be a time limit after which the VCS starts anyway) and the two tasks start competing for resources. So if i understand the snapshot right then it's probably indexing fault that it cannot finish in 2-3 minutes.
Comment 24 Tomas Zezula 2013-03-05 11:59:00 UTC
>if i understand the snapshot right thenit's probably indexing fault that it cannot finish in 2-3 minutes.
This is wrong evaluation. According to snapshot most of the time is spent in the collecting the files to be scanned (FileObject.getChildren and checking for symlinks in fact File.listFiles) these operations are IO bound and are slowed down by the versioning. The 2-3 minutes arguments is a non sense as the time depends on projects size.
Comment 25 Ondrej Vrabec 2013-03-05 13:10:51 UTC
(In reply to comment #24)
> The 2-3 minutes arguments is a non sense as the time depends on projects size.
Then i guess i can introduce a switch for those having such big repositories and clock versioning indefinitely while the parsing is running.
Comment 26 Ondrej Vrabec 2013-03-06 07:59:38 UTC
fixed: core-main #84a819690420
Comment 27 Ondrej Vrabec 2013-03-06 08:05:18 UTC
Start with -J-Dversioning.delayscan.nolimit=true and versioning system status scan won't ever block the parsing. The drawback is that till parsing is running file vcs statuses won't be refreshed.
Comment 28 Quality Engineering 2013-03-07 01:57:00 UTC
Integrated into 'main-golden', will be available in build *201303062300* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/84a819690420
User: Ondrej Vrabec <ovrabec@netbeans.org>
Log: Issue #208185 - Scanning Projects takes more than 2-3 minutes to complete
introducing a switch blocking status scanning indefinitely until parsing finishes