This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 111294 - Editor should run annotation processors and be aware of generated classes
Summary: Editor should run annotation processors and be aware of generated classes
Status: RESOLVED FIXED
Alias: None
Product: java
Classification: Unclassified
Component: Editor (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker with 5 votes (vote)
Assignee: Jan Lahoda
URL:
Keywords:
: 111080 (view as bug list)
Depends on:
Blocks: 141246 111065
  Show dependency tree
 
Reported: 2007-07-28 09:53 UTC by brucechapman
Modified: 2011-10-19 15:27 UTC (History)
5 users (show)

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments
Sample test project (642.21 KB, application/zip)
2010-06-08 09:54 UTC, brucechapman
Details

Note You need to log in before you can comment on or make changes to this bug.
Description brucechapman 2007-07-28 09:53:59 UTC
JSR-269 Annotation processors can generate source (*.java) and binary classes.

The editor should run any processors associated with annotations in the source file being edited, in order generate
these files so the generated classes are known to the editor in terms of resolving unknown types (so they don't show
spurious errors in the editor which wont occur when built), and so that the editor can show code completions for uses of
these generated classes.

One strategy to achieve this is to run the annotation processors against a custom Filer
(javax.annotation.processingFiler ) implementation which would keep the "files" in memory with no persistence. 

Note however that a processor may also read files. The primary motivation for this read functionality is as follows:

A) Imaging a processor which generates a single class as a result of processing all annotations (of a certain type) in a
given package.

B) In order for the processor to work correctly *in the presence of incremental builds* (such as when being called from
the editor) the processor should persist state for all the annotations present in the package into its own data file.

C) Each time the processor is run, it should first read the persisted state (if there is any), update the state for the
current set of compilation units (RoundEnvironment.getRootElements() ) based on the annotations present, persist that
new state, and perform generation based on all that state. The state is normally persisted to the SOURCE_OUTPUT location
so that it does not make its way into the jar files being built.


For an in memory file system, care should be taken support this scenario by processors. It might be that if the file
being read does not exist in the in-memory file system, then it should be obtained from the actual project file
structure, and thereafter maintained in memory. The project file system would contain state resulting from builds
(compiling project, package, and individual files, and sequences of these), the editor's version would initialise to the
project's version, but thereafter update as changes are made to the source code. It might me that the in-memory copy
should be dropped when a build is performed - or it might be best done by looking at the last modification times, and if
the project's is later, then that should be used to "freshen the cache" of the in memory version.
Comment 1 Jesse Glick 2007-07-28 16:00:37 UTC
Regarding step C, I was under the impression that a processor was forbidden to overwrite an existing file (i.e. its
store its write-once). Is this restriction present only for binary output and not for source output, perhaps?
Comment 2 Jan Lahoda 2007-07-29 19:17:45 UTC
While I would like to see better support of AnnotationProcessors in NetBeans, I think this is more an enhancement than a
defect.
Comment 3 brucechapman 2007-07-29 22:54:49 UTC
An annotation processor may only write to a file once. The compilation units being compiled are considered to be written
in zeroth round and so cannot be written at all. Otherwise it is permitted to overwrite a file once. The expert group
were careful to permit the scenario described above. jglick's comment is almost correct, *.java source code cannot be
overwritten, but generated *.java, *.class and other files can be overwritten at most once.

javax.annotation.processing.Filer class docs say

"During each run of an annotation processing tool, a file with a given pathname may be created only once. If that file
already exists before the first attempt to create it, the old contents will be deleted. Any subsequent attempt to create
the same file during a run will throw a FilerException, as will attempting to create both a class file and source file
for the same type name or same package name. The initial inputs to the tool are considered to be created by the zeroth
round; therefore, attempting to create a source or class file corresponding to one of those inputs will result in a
FilerException."
Comment 4 jessholle 2009-10-23 14:09:33 UTC
We're starting to make serious use of annotation processors which generate additional Java sources (e.g. base classes)
from annotations in our sources.

Some naive, crude coverage is possible in free-form projects by simply doing

      <java-data xmlns="http://www.netbeans.org/ns/freeform-project-java/2">
         <compilation-unit>
            <package-root>src</package-root>
            <package-root>src_gen</package-root>

where src_gen is the generated source tree.

Unfortunately, src_gen is only populated upon an actual build or compile.  Thus the editor is crippled when one is
working on files one has not yet compiled (showing lots of bogus errors, not having code completion, etc).

Also, src_gen is not checked into source control (it would be silly to do so), so it starts out empty.  Even if one
starts will a full build of src and src_gen in the output jar this is ignored since NetBeans always prefers sources and
ignores jars when it believes it has equivalent sources.  Thus one has to fully populate src_gen prior to NetBeans
making any sense of things.  This is likely to become a real issue for those working on downstream modules which could
care less whether src_gen is populated and will reasonably expect this information to be garnered from the module's
output jar.

Since Eclipse's scanner provides annotation processor coverage (which is working), I believe this is going to cause more
of our developers (we have hundreds) to move to Eclipse.  NetBeans' other scanning issues have already pushed a lot of
developers in that direction.  Overall I'd rank top-notch scanning as job #1 for NetBeans, where "top-notch" includes
always having full code navigation, code completion, attach debugger, etc, functionalities available without
pauses/delays/waits during re-scans as well as handling of annotation processors.
Comment 5 jessholle 2009-10-23 14:12:21 UTC
How far does http://kenai.com/projects/pelmel/pages/AnnotationProcessors go in this regard?

What are the chances of a near term solution to this issue?
Comment 6 Jesse Glick 2009-10-23 16:57:14 UTC
Pelmel does not currently implementing anything like this. It runs APs, but it does so as an editor hint (not as part of
the scan), and just to check for error messages from the AP; any generated files are sent to /dev/null. For now the only
way to deal with processors which generate classes (needed by primary sources to compile) is to send them to disk and
include that in your sourcepath, as in your src_gen example.

You are correct that src will show errors until the project has been built. As to the issue with downstream modules, I
am not quite sure what you are getting at but this should probably be a separate issue report. It is possible for a
custom SourceForBinaryQueryImplementation2 in global lookup to instruct the scanner to use the JAR from an upstream
module instead of the source root, so if this would help matters then you have a potential workaround (say handled by an
experimental module with some GUI TBD).
Comment 7 jessholle 2009-12-14 09:27:31 UTC
Any motion on this?  This is becoming a huge gap in NetBeans for us.

Eclipse provides built in support in this area and is definitely outdoing Sun on its own technology in this regard.

If this situation remains unchanged we'll likely see most if not all of our dozens of NetBeans users move to Eclipse.
Comment 8 jessholle 2009-12-17 08:59:05 UTC
Hmmm.... Unless something changes I'd bet that all of our many dozens of NetBeans users will have to move to Eclipse over the next month or two due to this.

This is a real shame, but maybe their other code scanning behaviors won't drive me insane -- this could be a good thing for my sanity, if a bad thing for NetBeans.
Comment 9 Jesse Glick 2009-12-17 11:54:44 UTC
Note that even without special support from the editor, it is possible to work with annotation processors that generate classes to some extent - you need to run a regular build first, and then error badges will go away, code completion will work, etc.

For example, as of 6.9 (bug #178426), NBM projects can make use of classes generated by APs without any special configuration (you need only have the AP registered in your classpath), and the editor will be OK once you have built the project. With Java SE projects, there is no special treatment for AP by default, but as of 6.7 (bug #105645) it is not hard to add: http://wiki.netbeans.org/FaqApt

Of course it is irritating if the project is as yet unbuilt; and code changes which would affect the AP can only be verified by doing a clean build, which is slower than relying on the usual background compiler to catch mistakes. But if the generated classes do not change frequently, it can be practical to work this way.
Comment 10 jessholle 2009-12-17 12:03:24 UTC
Are you saying that as of 6.9 NetBeans' scanner will notice the generated classes in my output jars without the sources being generated or the source directory being part of the project?  If so, that would *help*.  In particular we have hundreds of projects, some of them very large.  Any given developer will work in projects depending on many other projects.  They should not have to build all of these projects.  Rather they work against an installed copy of our software containing all the output jars.  If the scanner would recognize classes found in the output jars even when a corresponding source directory exists (but does not contain those classes -- source should be preferred to jars, but jars should not be ignored), that would help.

On the other hand, having the scanner understand annotation processor source generation *before* the project is built is really a key in cases where one is starting to write code using such processors to generate base classes, for instance.

Also note that (1) I'm specifically referring to free-form projects for our cases and (2) it really would be best to have separate compilation and annotation processor paths like javac does.
Comment 11 Jesse Glick 2009-12-17 12:29:24 UTC
No, currently the sources must be generated before the NB scanner will recognize them. That is why I mention this as a workaround that may be adequate for many users, not as a fix. Read FaqApt for a discussion of what works and what does not.

"source should be preferred to jars, but jars should not be ignored" - the scanner does not work this way: if sources are preferred then the JAR is ignored entirely. Changing this would be tricky, because you do want to show error badges in case a source file in use by other classes is deleted (even though its classfile may still exist in a JAR). Probably better to leave this behavior the way it is, and run APs during the scan.

Freeform projects vs. IDE-managed Ant projects vs. Maven projects does not matter much for this discussion. The scanner does not know or care what kind of project you have; it just looks at what source roots you have registered and their classpaths etc.

Optionally separating an annotation processor path is trivial. In current IDE releases, since the scanner does not run APs at all, your build script can do whatever it likes. When we do start to run APs in the scanner, the project will just need to report the processorpath using a new API in 6.8 (bug #174202), and I imagine there would be some GUI for customizing this for j2seproject's.

Somewhat off topic, but in your case things might work better - and in other respects than just AP - if your module-to-module dependencies were against published JARs (optionally associated with source ZIPs) rather than against subprojects. Then a project would just scan the JARs it uses and never have to refer to sources of upstream projects. This is especially convenient if you use Maven, since you need merely specify release dependencies like "1.3" rather than snapshot dependencies like "1.3-SNAPSHOT". In this case any changes to intermodule APIs necessitate publishing a new version of the API JAR, which could be a blessing or a curse depending on your perspective.
Comment 12 jessholle 2009-12-17 16:09:11 UTC
My module-to-module dependencies are against jars -- as I'm using free-form projects.  These are the same jars that the other projects denote as output jars as the other projects will, of course, build to these jars.  [No, this is not via Maven.]

Having associated source zips is a non-starter as I really want associated source directories, i.e. those from my other projects.
Comment 13 Jesse Glick 2009-12-17 16:29:39 UTC
Still off-topic, but to clarify:

(In reply to comment #12)
> My module-to-module dependencies are against jars -- as I'm using free-form
> projects.  These are the same jars that the other projects denote as output
> jars

That counts as a subproject dependency, not a binary dependency, because the JAR is associated with a source project.

> Having associated source zips is a non-starter as I really want associated
> source directories, i.e. those from my other projects.

When using Maven you can freely switch between snapshot dependencies, which map to a source project (when available); and release dependencies, which do not (but may have Javadoc or non-preferred source associations). This is logical enough: a snapshot JAR is essentially versionless, so the IDE has to track ongoing source changes in the subproject in order to offer correct code completion and so on; whereas a release JAR is associated with a particular immutable version of the other module, so recent changes in that source project are irrelevant. You can switch a given dependency from one mode to the other just by using -SNAPSHOT or not.

You can get a similar effect with freeform projects if you like. For example, have each module compile to build/classes, declared as an output of that project; and, upon explicit request, also deploy a versioned JAR to a shared area, *not* declared as an output. You can include ../other/build/classes in your classpath to make a subproject dependency if you are actively making parallel changes to both projects, or include ../shared/other-1.5.jar if you just want to use the last-released copy (the normal case). Besides the extra build infrastructure it is less attractive because there is no automated way to associate a source ZIP with a deployed JAR (though writing such a custom SourceForBinaryQueryImplementation would be trivial). I imagine Ivy could be used to get a Maven-like effect here from Ant scripts, though I don't know much about it.
Comment 14 Dusan Balek 2010-02-04 09:58:09 UTC
Implemented for j2se and apisupport projects. See:

Changeset: http://hg.netbeans.org/main/rev/d59a1df3507f
Log: Issue #111065: Support Annotation Processors in Java Editor added.
Comment 15 jessholle 2010-02-04 10:13:25 UTC
To be clear I really need this to work for free-form projects (specifically free-form web projects in this case, though that shouldn't make much difference as the "web" bit is just to get a bit of JSP support).
Comment 16 Jesse Glick 2010-02-04 13:07:22 UTC
(In reply to comment #15)
> I really need this to work for free-form projects

I think AnnotationProcessingQuery.EMPTY.annotationProcessingEnabled should return true since this is javac's default behavior. Then there would be reasonable default behavior for any Java-based project type; with more specific control available for project types which define a source output location, permit customization of processors, etc.
Comment 17 jessholle 2010-02-27 09:15:59 UTC
Jesse, I'm not sure what you last comment means.

I see annotation processor support noted in the new and noteworthy document for 6.9 M1, which is great.  Is this only for standard projects or does this apply to free-form projects (including web ones)?

Although seeing the annotation processor settings in the free-form project properties UI would be nice, this is just icing on the cake from my perspective.  What I mainly need is an extension to compilation-unit in project.xml to allow addition of a processorpath child element containing the annotation processor path -- and for the editor to use this properly.  I also need an ability to specify -s <generatedSrcDir> and -Axxx=yyy arguments.  [Yes, we are doing complex generation of many additional sources, including base classes for the non-generated sources, that we need NetBeans to comprehend.]  I don't necessarily see the need for any flag to enable/disable editor usage of this processorpath, as in the context of a free-form project I'd not have informed NetBeans of this path if I didn't want it to use it.

Allowing the new elements in project.xml should be fairly trivial.  The meat of the matter is proper invocation/treatment of annotation processors by the IDE.  The sooner such a capability is available the sooner we can try it out and raise any issues.

One other note: I see "Track Java Dependencies" in the standard project screenshot.  When one changes an annotation on a high-level interface Eclipse currently knows all the downstream generated classes to regenerate on this basis and automatically regenerates just the right classes.  This sort of tracing would be nice in NetBeans as well...
Comment 18 Jesse Glick 2010-02-27 09:26:58 UTC
(In reply to comment #17)
> I'm not sure what you[r] last comment means.

You need to follow the API to make sense of it.

> Is this only for standard projects or does this apply
> to free-form projects (including web ones)?

Currently only for j2seprojects and NBM projects that I know of.

> What I mainly need is an extension to compilation-unit in
> project.xml to allow addition of a processorpath child element containing the
> annotation processor path -- and for the editor to use this properly.

This would be a separate enhancement request for freeform projects, since it would require a schema change in http://www.netbeans.org/ns/freeform-project-java/2, plus a query impl in that module.

> I don't necessarily see the need for any flag to enable/disable editor usage of
> this processorpath, as in the context of a free-form project I'd not have
> informed NetBeans of this path if I didn't want it to use it.

If my suggestion were followed, AP would be enabled by default (for any project type that does not specify otherwise), using the compile classpath as the processor path. This is what javac does: you do not need to ask to invoke annotation processing if processors are registered in your classpath, it is fully automatic. Compare: https://bugs.eclipse.org/bugs/show_bug.cgi?id=280542

> I see "Track Java Dependencies" in the standard project screenshot.

This refers to usage of <depend> in the Ant script. It has nothing to do with the Java editor.

> When one changes an annotation on a high-level interface Eclipse
> currently knows all the downstream generated classes to regenerate on this
> basis and automatically regenerates just the right classes.  This sort of
> tracing would be nice in NetBeans as well...

It would. I have no idea how difficult it would be to make this work reliably and efficiently in NetBeans. This would be in the Java editor component - no involvement from the project type.
Comment 19 jessholle 2010-02-27 09:35:52 UTC
Jesse, you state:

"If my suggestion were followed, AP would be enabled by default (for any project
type that does not specify otherwise), using the compile classpath as the
processor path. This is what javac does: you do not need to ask to invoke
annotation processing if processors are registered in your classpath, it is
fully automatic. Compare: https://bugs.eclipse.org/bugs/show_bug.cgi?id=280542"

That's one way to use javac.  I could rearrange things to work that way by simply concatenating my processorpath to the end of my compilation classpath, but there are reasons to keep these separate and so we have kept them separate at the javac level.  Also, while we have registered our annotation processor via META-INF/services, we generally do need to pass -s and -A arguments.  I suppose that if the editor parsing generated sources just go into "limbo" (e.g. temporary files or memory buffers) rather than their normal locations perhaps we could forgo these arguments, though I'd have to look into that.

Thus if we had the basic "always use annotation processors in the classpath" bit that you suggest I could at least see how far that gets me and possibly hack around the lack of -s and -A capability.  It may still turn out that I really need to pass these arguments, but I could at least find out.
Comment 20 jessholle 2010-02-27 09:56:49 UTC
As to why one would have the compilation and annotation processor paths separate, a processor may have many dependencies that one does not wish to allow as compilation (or runtime) dependencies.  Keeping these paths separate can thus be quite important.  Merging these together just for editor/parser purposes would be alright as an interim tactic, though.
Comment 21 Jesse Glick 2010-02-27 10:33:03 UTC
(In reply to comment #19)
> I could rearrange things to work that way by
> simply concatenating my processorpath to the end of my compilation classpath,
> but there are reasons to keep these separate and so we have kept them separate
> at the javac level.

That's fine; I'm just arguing what the default behavior should be unless otherwise configured.

> we generally do need to pass -s and -A arguments

-s is supported by the current API. -A is not; this looks like an oversight: I guess a Map<String,String> processorOptions should be one of the things the query should specify.

> if the editor parsing generated sources just go into "limbo" (e.g.
> temporary files or memory buffers) rather than their normal locations

If the project indicates via the query what -s would be set to then the editor's parser will generate to that location.
Comment 22 jessholle 2010-02-27 10:53:03 UTC
I'm not sure what this means:

"If the project indicates via the query what -s would be set to then the
editor's parser will generate to that location."

How would a free-form project indicate a -s setting?  [For src, we have src_gen and for src_xxx, src_xxx_gen.]
Comment 23 Jesse Glick 2010-02-27 11:10:20 UTC
(In reply to comment #22)
> How would a free-form project indicate a -s setting?

It would need to be specified somewhere in project.xml.

(For http://wiki.netbeans.org/AutomaticProjects I hope to make this and other AP-related properties sniffed automatically from the build.)
Comment 24 jessholle 2010-02-27 17:01:57 UTC
By "It would need to be specified somewhere in project.xml", do you mean that there already is a place for this or that a place would need to be added for this?
Comment 25 Jesse Glick 2010-02-27 18:38:26 UTC
(In reply to comment #24)
> By "It would need to be specified somewhere in project.xml", do you mean that
> there already is a place for this or that a place would need to be added for
> this?

The latter.
Comment 26 Jesse Glick 2010-03-02 07:27:07 UTC
*** Bug 111080 has been marked as a duplicate of this bug. ***
Comment 27 Jesse Glick 2010-03-02 07:34:25 UTC
(In reply to comment #21)
> -A is not; this looks like an oversight

bug #181421
Comment 28 Jesse Glick 2010-03-02 07:39:16 UTC
(In reply to comment #16)
> I think AnnotationProcessingQuery.EMPTY.annotationProcessingEnabled should
> return true

bug #179749
Comment 29 Jesse Glick 2010-03-02 07:47:18 UTC
Currently apisupport.project includes build/classes-generated in the ClassPath.SOURCE for src. When this is implemented, assuming its AnnotationProcessingQueryImplementation implements sourceOutputDirectory for src to be build/classes-generated, will I be able to remove these entries, or will SOURCE still also need to include the generated classes?

(Since these dirs are usually empty, it slows down classpath scanning to include them, but if I don't, modules which do generate sources to these dirs show parser errors. And of course I want users to be able to open these sources in the editor for review.)
Comment 30 Jesse Glick 2010-03-02 09:27:47 UTC
(In reply to comment #18)
>> What I mainly need is an extension to compilation-unit in
>> project.xml to allow addition of a processorpath child element containing the
>> annotation processor path -- and for the editor to use this properly.
> 
> This would be a separate enhancement request for freeform projects

bug #181439
Comment 31 Jesse Glick 2010-03-02 09:29:33 UTC
(In reply to comment #23)
> (For http://wiki.netbeans.org/AutomaticProjects I hope to make this and other
> AP-related properties sniffed automatically from the build.)

bug #181441
Comment 32 brucechapman 2010-06-08 09:54:18 UTC
Created attachment 99889 [details]
Sample test project

Sample project useful for testing autocompletion against a class generated from multiple source files.
Comment 33 brucechapman 2010-06-08 10:26:37 UTC
Have just checked NB6.9 RC2 against an annotation processor which has a many-to-one relationship between source files and generated files, and designed to be safe in the face of incremental compiles.

It is simple, but a good exercise for editor support for annotation processors.

For background on this processor see http://weblogs.java.net/blog/brucechapman/archive/2008/03/anouncement_no.html


The source code is here
https://rapt.dev.java.net/source/browse/rapt/src269/net/java/dev/rapt/anon/

and uses
https://hickory.dev.java.net/


COS is turned on.

Compiling either source file with F9 then viewing the generated source (Ctrl-B with cursor on Runnables) works fine.

But the syntax highlighting and code completion seems to be using a a different and incorrect version of the generated class.


To demonstrate, unzip the attached project and open in netbeans.

clean and build to confirm that it builds OK.


Rename one of the instance methods in either class, as well as the invocation of the corresponding method on Runnables (on the line that makes and starts a new Thread). compile either the class only or whole project and notice how the ant build is still correct. Do same for other class and single compile. all OK as far as the ant build script is concerned.

But notice how the syntax highlighting is giving errors. Attempt to autocomplete a static method on Runnables and it will either show the methods generated from one class or the other, but not both at the same time. This is the primary remaining problem that I am aware of.

It seems that often (but not always) the autocompletion database contains the methods generated from the source file that was not the most recently compiled via COS (least recently saved methods are present). 

Viewing the source of Runnables (Ctrl-B with cursor on "Runnables") seems to mostly but not always show the correct generated source. 

I will attempt to create another version of this project with an instrumented copy of the annotation processor so we can maybe get a better picture of what is happening (in particular outputting the URLs of the various Filer objects so we can see what is being written and read).  

I hope all is clear. Just play with the attached project renaming the methods and the ragged edges of this otherwise good feature will become apparent.
Comment 34 zolta 2010-12-06 16:20:11 UTC
http://wiki.apidesign.org/wiki/LiveDB
Could be an other sample test project
Comment 35 Jesse Glick 2011-03-16 15:51:29 UTC
Shouldn't this be FIXED (other than any bugs turned up in comment #33)?
Comment 36 brucechapman 2011-03-18 04:24:20 UTC
This stuff is now 'good enough' in 7. Close these bugs and we can create new ones for the minor nits that still exist.

Bruce
Comment 37 jessholle 2011-03-18 10:16:56 UTC
I think this stuff was good enough for most cases in NB 6.9.1, so I'm assuming it still is in NB 7.  If there are specific issues then specific bugs should be filed on those.
Comment 38 Jesse Glick 2011-10-19 15:27:17 UTC
Closing; essentially implemented.