This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 206443 - [71cat] Very slow scanning
Summary: [71cat] Very slow scanning
Status: RESOLVED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 7.1
Hardware: All All
: P2 normal with 3 votes (vote)
Assignee: Svata Dedic
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-12-15 15:10 UTC by jmborer
Modified: 2013-01-07 09:21 UTC (History)
20 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter: 183812


Attachments
stacktrace (1.89 KB, text/plain)
2011-12-15 15:10 UTC, jmborer
Details
stacktrace (3.44 KB, text/plain)
2012-01-12 18:56 UTC, pwasson
Details
stacktrace (3.46 KB, text/plain)
2012-03-08 19:59 UTC, jmborer
Details
stacktrace (1.89 KB, text/plain)
2012-10-12 19:41 UTC, augcampos
Details

Note You need to log in before you can comment on or make changes to this bug.
Description jmborer 2011-12-15 15:10:35 UTC
Build: NetBeans IDE 7.1 (Build 201112051121)
VM: Java HotSpot(TM) Client VM, 20.1-b02, Java(TM) SE Runtime Environment, 1.6.0_26-b03
OS: Windows XP

User Comments:
jmborer: Runs after an SVN update. Now it starts scanning all projects an the IDE becomes very slow because a buid is running in parralell.




Stacktrace: 
java.lang.Exception: Scan canceled.
   at java.lang.Thread.getStackTrace(Thread.java:1479)
   at org.netbeans.modules.parsing.impl.indexing.LogContext.create(LogContext.java:74)
   at org.netbeans.modules.parsing.impl.indexing.LogContext.create(LogContext.java:67)
   at org.netbeans.modules.parsing.api.indexing.IndexingManager.refreshAll(IndexingManager.java:414)
   at org.netbeans.modules.parsing.api.indexing.IndexingManager.refreshAllIndices(IndexingManager.java:373)
   at org.netbeans.modules.versioning.indexingbridge.Bridge$1.call(Bridge.java:63)
Comment 1 jmborer 2011-12-15 15:10:37 UTC
Created attachment 114228 [details]
stacktrace
Comment 2 Tomas Zezula 2011-12-15 15:23:05 UTC
Seems that the VCS triggers scan of everything even small number of files was changed. :-(

at org.netbeans.modules.parsing.api.indexing.IndexingManager.refreshAllIndices(IndexingManager.java:373)
   at org.netbeans.modules.versioning.indexingbridge.Bridge$1.call(Bridge.java:63)
   at org.netbeans.modules.parsing.api.indexing.IndexingManager.runProtected(IndexingManager.java:390)
   at org.netbeans.modules.versioning.indexingbridge.Bridge.runWithoutIndexing(Bridge.java:59)
   at org.netbeans.modules.versioning.util.IndexingBridge.runWithoutIndexing(IndexingBridge.java:139)

Needs to be verified.
Comment 3 David Strupl 2011-12-22 10:12:21 UTC
Jardo, can you please have a look and potentially re-assign? Thanks.
Comment 4 J Bachorik 2011-12-23 09:09:09 UTC
Well, the call is "org.netbeans.modules.parsing.api.indexing.IndexingManager.refreshAllIndices(false, false, files)" where "files" contains the files/folders a user selected to update. According to javadoc the "refreshAllIndices" should scan only the provided files in this case. And, in fact, on my machine it does. I am still not able to trigger all projects full rescan just by doing svn update.

jmborer, could you provide more detailed description how to reproduce this problem?
Comment 5 David Strupl 2012-01-08 09:28:01 UTC
If I read the very first comment correctly, the reporter was doing SVN update while some build was running. jmborer, can you please tell us what build was running (I mean java, maven, php)? And also was the build running from the IDE or externally? Or do you mean that the IDE was doing the build (scan) automatically (and you were not building anything yourself)?
Comment 6 jmborer 2012-01-09 11:50:29 UTC
Sorry responding so late. I just returned from vacation. 

I only use Maven projects. Indexing works most of the time correctly. it is very hard to figure out a scenario to reproduce scanning problems. What I noticed:

* file change detection has improved in NB 7.1. Side effect: scanning now triggers very frequently
* when scanning becomes very slow, most of the time the native windows file change daemon seems involved. I am also running out of memory then...
* I use a code generator in some of my Maven projects that is called from the Maven Ant plugin. I use <sourceRoot>${generated-sources}</sourceRoot> to make them visible to Maven. The scanning somehow looses sync with those generated files after a while even when I invoke "Scan for external changes". All generated files used in my code appear then red underlined in NB. The only solution is to delete the index cache. It occurs more often when a do an external Maven clean than from within the IDE, but it does also occur.

Globally: file change detection has improved, but scanning is becoming slow after a while on my PC (maybe memory issue). Because it triggers very often now, all operation requiring scanning to be finished are blocked and my IDE is very often stuck until scanning completion. Scanning seems not to work perfectly with Maven "third party" code generation declared in <sourceRoot>${generated-sources}</sourceRoot>.

Product Version: NetBeans IDE 7.1 (Build 201112071828)
Java: 1.6.0_30; Java HotSpot(TM) Client VM 20.5-b03
System: Windows XP version 5.1 running on x86; Cp1252; en_US (nb)
User directory: H:\projects\.netbeans\7.1
Cache directory: H:\projects\.netbeans\7.1\var\cache 

Hope it gives some insights.

Jean-Marc
Comment 7 pwasson 2012-01-12 18:56:40 UTC
Created attachment 114843 [details]
stacktrace

I am editing Entity classes (using JPA annotations). I have about 6 projects open. "Scanning Projects" happens frequently, uses 100% of one CPU core, and makes the IDE unusably unresponsive until it finishes.
Comment 8 J Bachorik 2012-01-13 16:02:52 UTC
Seems rather suspicious that rescanning a project with 0 new and 0 modified files can take up to 30s according to the message.log.

Tomasi, do you have insight in this?
Comment 9 jmborer 2012-01-13 17:44:32 UTC
Not much additional information yet. As soon as I get more info I'll provide it.
Comment 10 jcisio 2012-01-18 15:08:59 UTC
I have the same problem. NetBeans 7.1 on Linux, 4 opened projects (about 2 GB, but mostly binary files (JPG...)). I run a "svn update" in the command line, a dozens of files change, and NetBeans use 50-100% iowait and cpu to rescan the whole project, make it unusable in 5 minutes.
Comment 11 jcisio 2012-01-18 15:12:19 UTC
(In reply to comment #10)
> I have the same problem. NetBeans 7.1 on Linux, 4 opened projects (about 2 GB,
> but mostly binary files (JPG...)). I run a "svn update" in the command line, a
> dozens of files change, and NetBeans use 50-100% iowait and cpu to rescan the
> whole project, make it unusable in 5 minutes.

I forgot to add that I even have Tools -> Options -> Miscellaneous -> Files -> "Enable auto-scanning of sources" disabled.
Comment 12 Tomas Zezula 2012-01-18 15:16:06 UTC
The problem is in cmd line SVN update.
In the case of cmd line SVN update the IDE gets huge amount of unsorted file events which are causing lots of redundant scans. If you use the IDE SVN support the events are fired after the SVN update finished and the events are correctly ordered.

As far as I know the Tools -> Options -> Miscellaneous -> Files -> "Enable auto-scanning of sources" has no effect in case of OS with native file listening support (Win, OSX, Linux, Solaris).
Comment 13 saden1 2012-02-08 20:35:23 UTC
I have a bunch of maven based projects and modules and every time I start netbeans or switch to a different Project Group netbeans scans my entire ~/.m2/repository. This didn't seem to be an issue with prior netbeans 7 releases. The scan time is brutal and I was wondering if this issue is similar be bug #162706?

Netbeans: 7.1
Java Runtime: 1.6.0_27-b07
Java HotSpot: 20.2-b06, mixed mode
OS: Linux Mint 12 (3.0.0-15-generic-pae #26-Ubuntu SMP)
Comment 14 Tomas Zezula 2012-02-09 07:52:05 UTC
In reply to comment #13:
The jar indexing (up to date problem) in NB 7.1 is caused by SpringBinaryIndexer which does (even on up to date check) expensive operations. This is already fixed in 7.2. Here are the issues:
1st) Issue #205575 - allow SBI to use cheaper predicate
2nd) Issue #206059 - solving the problem that some j2ee indexers did not do up to date check. They do indexing even when nothing has changed. For indexers using the new registration the infrastructure does not rather call these indexers when jar file is up to date.
Comment 15 jmborer 2012-03-08 19:59:15 UTC
Created attachment 116490 [details]
stacktrace
Comment 16 Marian Mirilovic 2012-09-20 13:16:10 UTC
159 dups ... at least P2
Comment 17 Tomas Zezula 2012-09-20 13:28:05 UTC
QA should split the reports and close the wrong reports.
Comment 18 Marian Mirilovic 2012-09-20 13:31:21 UTC
Sorry Tomas, we can't do that ... there are thousands of reports like this, if you find particular pattern we will add it into the duplication mechanism.
Comment 19 augcampos 2012-10-12 19:41:25 UTC
Created attachment 125884 [details]
stacktrace
Comment 20 Svata Dedic 2012-12-18 20:30:22 UTC
I went through the 7.3 reports. Some of them were truncated or cancelled too early (-> invalid). An issue was raised against VCS (issue #223997) since refreshAll was called at rapid rate. I attributed some reports to issue #223171.

The rest of reports relate to previous versions and are likely either fixed in 7.3, or already dispatched based on other reports I've processed throughout the year. I recommend to close this defect.
Comment 21 Svata Dedic 2013-01-07 09:21:41 UTC
Cleared next 7.3 reports (vcs, html, cancelled early). Closing as the exc. report does not report 7.3 issues at all.