I'm reproducing below the e-mail messages exchanged about this topic in NetCat mail list:

---------------------------------------------
Edson Richter - SimKorp
      16 de agosto
      
    
    Hi, NB Developers,

I'm wondering if it is possible to allow developers "choose" which libraries 
will be indexed - instead indexing everything...
My index folder has currently > 186Mb and 13.593 files distributed in 7.223 
folders!!!

And when I'm working, NetBeans offer things I'll never use at all. Example (attached image 1)


NONE of the options above are of my interest... I don't use iText neither Corba 
in my project (but it is required to be in Tomcat path - and therefore, NetBeans 
believes I want it in my code completion) the options I would like to see at far 
below the presented options...

The fact is that I need some libraries that are dependencies of libraries I use 
- for example, JasperReports 5 has hundred of dependencies I'll never user - I 
even don't know if it is useful or not for my project, but they throw an error 
if not available at runtime.

I know I can filter by package name... but very time consuming and I'll spend 
more time discovering which package names I don't want, and even then they are 
still indexed and analyzed/maintained without need.

So, I would like to be able to check only libraries (or JAR files) I really need 
for code completion, and nothing else.
I've filled a enhancement request with this idea in mind few months ago, but no 
comments so far.
Thanks for your attention, and all comments and criticism will be appreciated.


------------------------------------------------------------------------------
      Mark Wilmoth
      16 de agosto
      
    
Edson,

I don't know if this would stop indexing or not or if this is any help, I was 
just looking at it to see what it does now, but if I add 
a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z to the exclude list, it 
excludes everything (Ctrl+Space two times) except it still shows all static members.
Is there a good reason it is showing static members always? Is this a bug?

Mark  


------------------------------------------------------------------------------
      Edson Richter - SimKorp
      16 de agosto
      
    
Hi, Mark!

Yes, I think this is one good point.
I've filtered "com.lowagie.*", and the static members are still visible.
I think you detected a new bug and worth to create a new ticket issue for that; 
probably a bug introduced recently, but would be good to test in 7.3.1 to know 
if it was happening or not.

But I'm still centered in reducing size of indexes.
Another idea that just occurred to me is to have per project indexes - and would 
be difficult to switch from one index to another due fluid nature of NB projects.

Edson Richter
  
  
------------------------------------------------------------------------------
      Mark Wilmoth
      17 de agosto
      
    
I checked 7.3.1 and it doesn't suggest an static members before a dot is typed. 
It was added in 7.4 and is discussed in "New and Noteworthy" [1].

[1] http://wiki.netbeans.org/NetBeans_74_NewAndNoteworthy#Editor 
<http://wiki.netbeans.org/NetBeans_74_NewAndNoteworthy#Editor>

Mark


------------------------------------------------------------------------------
      Edson Richter - SimKorp
      segunda-feira
      
    
Is there any way to tell NB to not index JAR libraries that are in tomcat/lib 
folder? I'm volunteer to test this and report the results, IMHO this would be a 
huge improvement.

Regards,

Edson


------------------------------------------------------------------------------  
      Mark Wilmoth
      segunda-feira
      
    
    Some of that stuff that comes up now is "internal proprietary" and it isn't even supposed to be used. :)

Mark


------------------------------------------------------------------------------  
      Petr Jiricka
      terça-feira
      
    
Hi Edson, 

I found that some time ago you filed issue https://netbeans.org/bugzilla/show_bug.cgi?id=227558, is that what you are referring to? Can you please elaborate more on what specific UI you envisage? In this issue you are requesting to disable code completion "at library level", but for example if you go to project properties, the truth is that libraries present in tomcat/lib are not presented in the project properties UI - they are not on project classpath. Assuming you use Ant-based projects, these jars are added to the index/code completion via the "server platform classpath" (which you'll see if you expand the Libraries -> Apache Tomcat node in the project), not through "project classpath". The other thing is that it's not possible to remove ALL jar files in tomcat/lib from classpath/code completion - for example servlet-api.jar file needs to stay there, it is a part of the Java EE platform. 

I also believe that there are legitimate situations when you may want to compile against some of the classes in tomcat/lib. So, there would need to be some UI which would control if the non-API jars from tomcat/lib should be included on classpath or not (and possibly which ones). How specifically would you like to control this, and at which level? Per server instance? Per project? Global in the IDE? Or…?

Thanks,
Petr
  

------------------------------------------------------------------------------  
      Thomas Wolf
      terça-feira
      
    
Hi Petr,
I don't want to speak for Edson, but his bug looks very similar to one from 2009 that I voted for (https://netbeans.org/bugzilla/show_bug.cgi?id=162471) - that one requests the same thing - but for the "Go To Type" window.  Can the two be fixed together somehow?  I'm afraid that otherwise it will languish another 4 years.

thnx,
tom
  

------------------------------------------------------------------------------  
      Edson Richter - SimKorp
      terça-feira
      
    
1) Sometimes for different projects I have to keep different Tomcat 
configurations with different libraries set. For example, for one project, I do 
have all "JasperReports 3.x" in lib folders (yeah, It has to be in Tomcat lib to 
avoid OOME during project reload). For another project, I do have JasperReports 
5.x. Thinking about other project types also I believe that per project would 
give better granularity.

2) Selection must be at JAR level for fine granularity (and not per library). 
For example, again using JasperReports (but applies for almost every library in 
the world), I do want to have auto complete for JasperReports main library jar 
file, but not for batik, iText and everything else.

3) I don't know internals of NetBeans indexes, and I don't know if they have any 
other information than auto-completion. So I can't say if all JAR files can be 
completely eliminated

4) For a first time implementation, it would be a "blacklisted jar files" 
configured outside the UI (in nbproject.properties?). Advanced developers would 
go into project properties file and define which JAR files are blacklisted and 
should not be present in code completion.
After this feature proves to be stable and useful, NetBeans would start 
collecting JAR files (also per project) and add them into a list, and then users 
can select which ones will be indexed or not. I can imagine something similar to

(see image attachment 2 [details])

Of course, after changing JAR selection, developer will be forced to "Rebuild 
Indexes", otherwise it will not be effective.

Thanks for sharing your thoughts and questions, they are really valuable and 
helped me to develop a more concise idea of this feature.

Edson


------------------------------------------------------------------------------  
      Petr Jiricka
      quarta-feira
      
    
    Thanks for the ideas, this is a pretty concrete proposal. Can you please record these thoughts in bugzilla (e.g. as a comment to issue 227558), so they are not lost?

Thanks,
Petr
  

  
------------------------------------------------------------------------------  
      Edson Richter - SimKorp
      quarta-feira
      
    
    Yes, for sure. I'm in traveling, but as soon as I reach the office, I'll 
record this in the issue.
I've started this mail because I wanted to open a broad discussion: we 
have a rich group of developers from all over the world, and I would 
love to hear their experiences as well.

Regards,

Edson

Created attachment 139214 [details]
Image 1 - undesired options shown

Created attachment 139215 [details]
Image 2 - proposal for project properties jar indexing UI

Reassigning to web/project based on off-line conversation with Tomas Zezula. Per Tomas:

Internally the web project has to use runtime classpath, the CP returned by ClassPath.getClassPath(webProjectDir, CP.EXECUTE)
otherwise the debugging will not work.

So it's up to web project to provide such a UI and return correct ClassPaths.
The CP.COMPILE may not be equal not subset of the CP.EXECUTE.
If the Web project reports correct CP.COMPILE and CP.EXECUTE everything will work.
Neither new API nor changes to project infrastructure are needed.

The easiest fix is to split j2ee.platform.classpath to j2ee.platform.compile.classpath and j2ee.platform.runtime.classpath.
By default:
j2ee.platform.runtime.classpath=${j2ee.platform.compile.classpath}
But user can move references among these two.

The ClassPath.EXECUTE will contain: j2ee.platform.runtime.classpath
The ClassPath.COMPILE will contain: j2ee.platform.compile.classpath

No UI needed, user can move the references in project.properties

Thanks for your comments and the analysis.

But I don't want to reduce/change classpath, but the index and code completion options offered to developer while programming.

By today, "Filtering" options is a bit buggy, and even when you apply the filter, by issueing double "Ctrl+Space" you will get all that trash back (by trash I mean all those classes necessary for runtime, but not for "editor/code completion" time).

And this apply to either Java Web, J2SE, J2ME and Maven projects.

When creating a project or adding a library, it NetBeans would offer the option to automatically include it into the index process (and/or background scanning), and the developer would decide to remove it. Later, developer would add/remove libraries, and this would trigger a complete reindex task if necessary.

I've projects that I don't touch libraries for months, but all those classes are still being served as options for code completion (even if I never use them).

One example is when I type BigDecimal| (Ctrl+Space), NetBeans always offer dozen or more options for it... that I never used. (see image 1)

If I use "subword completion", and type ALL| (Ctrl+Space), I get even more options I'll never use in my life! (see image 2)

Ops, I mean Image 3 and Image 4... those are the new I've uploaded into the ticket.

Created attachment 147893 [details]
Image 3 - BigDecimal example

Created attachment 147894 [details]
Image 4 - ALL example

PetrJ, what you suggest makes sense to me. For application servers it sort of works as application server plugin should always return only classpath representing the server APIs. For example GF returns just EE APIs. Tomcat unfortunately returns bunch or irrelevant internal jars. But the moment user is using a J2SE library there is no other way than adding the library and all its jars into compilation classpath. That could be amended by your proposal of adding "Run" classpath into J2EE project properties and it would help.

But what I'm hearing from Edson is slightly different - add ability to exclude jar (or its subset?) from IDE features like Code Completion or Go To Type (issue 162471). Which makes lots of sense to me as code completion often contain lots of "garbage". I see this as UI only filter. The indexing still has to process all files but results are filtered out. It is similar to "Debugging Sources" view where user can check/uncheck jars they want to step into or not during debugging. Such feature should work in all project types. And would cover also lack of separate Run classpath in J2EE projects. Does that make sense to you TomasZ?

Hi David!
I believe that the implementation or internal jars of the app server do not need to be scanned. It's similar to the compiled classpath vs runtime classpath in the J2SE project. The implementations like jndi providers, jdbc drivers, etc belong to the runtime classpath and just the APIs belong to the compile classpath. The compile classpath (not the runtime classpath) is scanned and used for code completion, Go To Type (Symbol) and compilation.

Splitting the app server classpath to compile and runtime app server classpath will change the behaviour to the same as in the j2se project. Jars on the app server runtime classpath will not be scanned and will not be offered in code completion, go to...

Cau Tomasi,

Re. "Splitting the app server classpath" - as a said that is a bug in the Tomcat server plugin and there is a way to fix it. GlassFish does not have that problem as it exposes only API jars.

But try this: create J2SE project and add "Struts 1.x" library to it. That will "pollute" the compilation classpath with bunch of implementation libraries like antlr, apache commons, etc. How should that be address? Perhaps there should be two libraries: "Struts 1.x API" and "Struts 1.x Runtime Impl" and user should add first one to Compilation classpath and second one to Run classpath?? That would make sense to me.

Hi Davide.
Yes it makes sense.
We can also simplify it for user by changing the J2SE Library to contain both classpath and runtime classpath volumes. Adding such a library to project will populate both project's compile classpath and runtime classpath.
Should I do such a change in project.libraries?

> that is a bug in the Tomcat server plugin

Cc'ing Petr H.

Thank you for all effort so far.
Two question remains:
1) how to deal with libraries that doesn't offer separate jars for API and implementation?

2) And if this is a bug in Tomcat plugin, how to deal with J2SE suffering of very same problem (I mean the pollution)?

Thanks,

Edson

(In reply to Tomas Zezula from comment #12)
> We can also simplify it for user by changing the J2SE Library to contain
> both classpath and runtime classpath volumes. Adding such a library to
> project will populate both project's compile classpath and runtime classpath.

This makes sense to me.

I would even think that we should have had such feature in place for a very long time. But if it was not requested earlier maybe it is not such a big issue?? Just thinking out loud here.

I filed issue 245652 for the Tomcat problem.

(In reply to ecerichter from comment #14)
> 1) how to deal with libraries that doesn't offer separate jars for API and
> implementation?

Manually create them is the only workaround for now I think.

> 2) And if this is a bug in Tomcat plugin, how to deal with J2SE suffering of
> very same problem (I mean the pollution)?

Would Tomas's suggested solution above work for you? that Ant Library Manager would have "Runtime Classpath" classpath in addition to "Classpath"? That would mean that for example JasperReports library would keep in "Classpath" only JasperReports API jars and all dependencies would be moved to the "Runtime Classpath". As a result code completion would show only JasperReports APIs jars. Would that solve your problem? Again, it should be testable even today with a bit of manual setup in the J2SE project - duplicate JasperReports and separate API jars from impl ones.

> Would Tomas's suggested solution above work for you? that Ant Library
> Manager would have "Runtime Classpath" classpath in addition to "Classpath"?
> That would mean that for example JasperReports library would keep in
> "Classpath" only JasperReports API jars and all dependencies would be moved
> to the "Runtime Classpath". As a result code completion would show only
> JasperReports APIs jars. Would that solve your problem? Again, it should be
> testable even today with a bit of manual setup in the J2SE project -
> duplicate JasperReports and separate API jars from impl ones.

Worth to try, please let me know when/how can I make a test?

Regards,

Edson