Bug 185139

Created attachment 98165 [details]
stacktrace

I experienced the same problem when trying to create a JAXB binding using the local XSD file.

Same problem - I was trying to create a JAXB binding.

Needs to be fixed.

Not immediately obvious what the Maven support should do when asked to add javax.xml.bind.* to the "endorsed" classpath, which is a nebulous concept in Java.

Currently queries on ClassPath.ENDORSED just return elements in maven-compiler-plugin's <bootclasspath>...</> which are not in JavaPlatformManager.defaultPlatform.bootstrapLibraries, which seems rather peculiar; see bug #172952 for background. ProjectClassPathModifier is unimplemented for this classpath type, hence this exception.

Probably what is expected is for some JAXB artifact (mapped from the NB library using the usual Maven volume identifier) to be added with <scope>provided</>. This seems similar to bug #186221, which defined an ad-hoc constant classpath/compile_only to be interpreted by Maven as provided scope.

Should these hacks be unified into one hack? So that ClassPath.ENDORSED can be used to both read and write provided-scope dependencies in Maven projects? It is not entirely clear to me what the ramifications would be.

One possible issue is that provided-scope dependencies presumably cannot supersede JRE interfaces in case of conflict; for example, since javax.xml.bind is already in Java 6, adding a provided-scope dependency on these interfaces would have no effect. Which again gets back to the fact that endorsed libs were a bad idea to begin with and we are all eagerly awaiting some Java module system which would save us from this mess.

I think that 'provided' classpath and endorsed are two different things, no?

Re. provided classpath - concrete usecase is to add a jar to web project classpath but exclude that jar from being packaged into resulting WAR file (for example Java EE API jars - they are provided by application server container). Variation on this usecase is a need to package something to WAR without placing it on compilation classpath (for example Derby JDBC driver which is not available in runtime on server but should not be available for compilation neither). Ant based web project solved this by introducing WebProjectLibrariesModifier (a friend SPI). Just yesterday I had to implement and use this SPI also in Enterprise Application Project (should have been done about 5 years ago!). Perhaps Maven should just implement this SPI and any client who needs to add provided classpath should just use it. Would that work?

Re. endorsed classpath - concrete usecase here is that Java EE 6 is build on top of Java SE 6 with exception of two or three SE APIs which are required in newer version then available in Java SE 6. These new APIs are needed for project compilation, unit test running but also for project execution (Application Client is Java EE module which runs in local JVM) and has to supersede ones coming from Java SE 6. When it comes to deployment to server endorsed classpath does behave similarly to provided classpath because it is not needed - server container must provide them.

I may just have rephrased what you said Jesse, but I wanted to make sure we are on the same page. Is it clear what I'm saying?

Right - provided and endorsed are two different things. Endorsed classpath in Maven is not implemented via provided scope, rather, it is implemented by prepending the revised API jar to the bootclasspath argument of the compiler plugin. See e.g. the Maven Java EE 6 web project, which does this directly in the archetype.

So, I think the fix should either be to add this hacky solution to the user's pom file when adding JAXB, or to use an older version of JAXB, which is compatible with JDK 6 and does not require the endorsed classpath. I am cc'ing Martin G as the JAXB expert and owner for comment.

(In reply to comment #6)
> package something to WAR without placing
> it on compilation classpath (for example Derby JDBC driver which is not
> available in runtime on server but should not be available for compilation
> neither)

This is <scope>runtime</> ~ ClassPath.EXECUTE, I think.

> Ant based web project solved this by introducing
> WebProjectLibrariesModifier (a friend SPI).
> Perhaps Maven should just implement this SPI and any
> client who needs to add provided classpath should just use it.

Feel free to implement that as a @ProjectServiceProvider(projectType="org-netbeans-modules-maven/war") to work with <scope>provided</>. The API should be moved to a more neutral location (web.api?) since web.project is a concrete project type and this does not belong there. Cleaner would be to define an official ClassPath type PROVIDED and use the existing ClassPathModifier API. But since you say that provided != endorsed, this sounds like an independent issue to me.

> Re. endorsed classpath - concrete usecase here is that Java EE 6 is built on
> top of Java SE 6 with exception of two or three SE APIs which are required in
> newer version then available in Java SE 6. These new APIs are needed for
> project compilation, unit test running but also for project execution
> (Application Client is Java EE module which runs in local JVM) and has to
> supersede ones coming from Java SE 6.

Is there a concrete example of a class which exists in Java SE 6 in one version but has an enhanced API in a publicly downloadable official update? That would be useful for testing <scope>provided</>, <scope>system</>, bootclasspath, etc.

(I could find various examples of packages which seemed incomplete in the JRE but more complete in a glassfish3/modules/*.jar, e.g. javax.transaction, but this is not helpful since you could just add the more complete javax.transaction.jar as a regular dependency to the module - it need not override any particular JRE class.)

> When it comes to deployment to server
> endorsed classpath does behave similarly to provided classpath because it is
> not needed - server container must provide them.

Right. So is the only difference that endorsed libs must be (somehow) prepended to boot CP for purposes of compilation and test execution, whereas provided libs can be passed in the regular tool -classpath args?

(In reply to comment #7)
> implemented by
> prepending the revised API jar to the bootclasspath argument of the compiler
> plugin. See e.g. the Maven Java EE 6 web project, which does this directly in
> the archetype.

I guess you mean

<compilerArguments>
  <bootclasspath>${settings.localRepository}/javax/javaee-endorsed-api/6.0/javaee-endorsed-api-6.0.jar${path.separator}${sun.boot.class.path}</bootclasspath>
</compilerArguments>
<dependency>
  <groupId>javax</groupId>
  <artifactId>javaee-endorsed-api</artifactId>
  <version>6.0</version>
</dependency>

Wow that is clumsy. Perhaps Maven ought to define a supported <scope>endorsed</> to improve this situation, as http://www.mail-archive.com/dev@maven.apache.org/msg82720.html suggests. http://maven.apache.org/guides/mini/guide-using-toolchains.html sounds somewhat related but I don't think it addresses this issue.

http://www.mindbug.org/2009/02/adding-endorsements-to-mavens-plugins.html seems like a cleaner approach (-Djava.endorsed.dirs, dependency:copy) than our current hack, or http://osdir.com/ml/dev-geronimo-apache/2010-04/msg00172.html (-endorseddirs) is similar and maybe even better. There is the drawback that some possibly large JARs are copied into the target dir, but at least each endorsed dep is specified just once (rather than in two places as in our current hack). EndorsedClassPathImpl would need to be modified to understand these idioms as well as the current hack.

Note #1: if I delete javaee-endorsed-api-6.0.jar from my local repo and try to build the project, Maven fails to find it for download; what repo is this supposed to come from? I don't know how this got into my local repo to begin with, and now I can't get it back.

Note #2: there is javaee-web-api-6.0.jar in provided scope as well, which looks to be a superset of javaee-endorsed-api-6.0.jar, which is confusing.

> I think the fix should either be to add this hacky solution to the user's
> pom file when adding JAXB

Possible. Would have to search for //profile/[id='endorsed'] and update it if found, else create such a profile from scratch. Or do the same for the alternate idioms given above.

> or to use an older version of JAXB, which is
> compatible with JDK 6 and does not require the endorsed classpath.

Probably xml.jaxb should at least catch this UOE and try to behave gracefully, especially if ClassPath.BOOT does already contain some version of javax.xml.bind. This would be useful for various project types - it cannot expect that every one knows how to deal with ENDORSED.

(In reply to comment #8)
> Note #1: if I delete javaee-endorsed-api-6.0.jar from my local repo and try to
> build the project, Maven fails to find it for download; what repo is this
> supposed to come from? I don't know how this got into my local repo to begin
> with, and now I can't get it back.

Seems it is in the java.net repo, but did not get downloaded merely be being present as a dep of maven-compiler-plugin. That dep placement is really wrong since it affects the CP *of the plugin impl* which is surely not what you meant. Adding this artifact (with provided scope) under profile/dependencies works better.

Created attachment 101248 [details]
Apparently functional POM not relying on bootclasspath hacks

As previously mentioned, not sure how to test that Java SE 6 classes are actually being superseded in this case. Class.protectionDomain.codeSource.location in a project unit test on e.g. javax.xml.bind.Binder throws an NPE, I hope because endorsed libs like boot libs define no code source.

(In reply to comment #8)
> Perhaps Maven ought to define a supported <scope>endorsed</>

I filed http://jira.codehaus.org/browse/MNG-4752 since this issue may affect lots of people.

(In reply to comment #8)
> > package something to WAR without placing
> > it on compilation classpath (for example Derby JDBC driver which is not
> > available in runtime on server but should not be available for compilation
> > neither)
> 
> This is <scope>runtime</> ~ ClassPath.EXECUTE, I think.

I would think so. Though author of WebProjectLibrariesModifier decided to not handle it that way and created a dedicated API methods [add|remove]Package[Libraries|Artifacts|Roots] instead.

> > Ant based web project solved this by introducing
> > WebProjectLibrariesModifier (a friend SPI).
> > Perhaps Maven should just implement this SPI and any
> > client who needs to add provided classpath should just use it.
> 
> Feel free to implement that as a
> @ProjectServiceProvider(projectType="org-netbeans-modules-maven/war") to work
> with <scope>provided</>. The API should be moved to a more neutral location
> (web.api?) since web.project is a concrete project type and this does not
> belong there. Cleaner would be to define an official ClassPath type PROVIDED
> and use the existing ClassPathModifier API.

I would prefer to get rid of WebProjectLibrariesModifier in favor of ClassPath.PROVIDED and ClassPath.EXECUTE. By 'prefer' I mean it would be nice to do that but right now it is extra work I do not want to spend time on. 

> Is there a concrete example of a class which exists in Java SE 6 in one version
> but has an enhanced API in a publicly downloadable official update? That would
> be useful for testing <scope>provided</>, <scope>system</>, bootclasspath, etc.

The one I was using is described by Ludo at the beginning of issue 172952. SE 6 contains Common Annotations 1.0 but EE 6 requires Common Annotations 1.1. The changelog of CA1.1 can be found here: http://jcp.org/aboutJava/communityprocess/maintenance/jsr250/jsr250ChangeLog.html and Ludo's usecase is #4 - new lookup attribute. Other SE APIs which are required in newer versions are jaxb and webservices.

> Right. So is the only difference that endorsed libs must be (somehow) prepended
> to boot CP for purposes of compilation and test execution, whereas provided
> libs can be passed in the regular tool -classpath args?

Yes.

And in case of Enterprise Application Client endorsed libs must be prepended to boot CP even for project execution.

> Note #2: there is javaee-web-api-6.0.jar in provided scope as well, which looks
> to be a superset of javaee-endorsed-api-6.0.jar, which is confusing.

Name of "javaee-endorsed-api-6.0.jar" is probably confusing or not self descriptive. It contains all classes which upgrade SE 6 to level required by EE 6, that is union of all changes in Common Annotations 1.1, jaxb and webservices APIs is stored in this jar so that it can be preppend to bootclasspath and simplify compilation of EE 6 applications. Probably better name would be javase-endorsed-api-for-javaee6.0.jar.

<unrelated-to-this-discussion>
Jesse, (FYI) I'm not sure if you are aware of this but EE 6 APIs (which are on java.net) do not contain full Java class files. Bytecode was stripped down and what remains in these jars are just method signatures. I think this was motivated by some legal issues. The result is that you can compile against these jars and in runtime server provides full EE 6 APIs so in theory everything works and everybody is happy until you decide to run your unit tests. That's why for example javaee.api/src/org/netbeans/modules/javaee/api/javaee-endorsed-api-6.0.xml library does not contain javaee-endorsed-api-6.0.jar but instead have three individual jars with real classes: javax.annotation.jar, jaxb-api-osgi.jar and webservices-api-osgi.jar. If Maven users want to unit test a code which requires any EE 6 APIs they will have to add to their POM's real EE 6 jars from a server implementation they use. It is quite messy and I'm not sure how IDE could help them but I'm giving up. For GlassFish a solution would be easy as it is open source and it publish its artifacts in java.net repo, but once users start using Weblogic or Websphere then not much can be done I think.
</unrelated-to-this-discussion>

(In reply to comment #12)
> I would prefer to get rid of WebProjectLibrariesModifier in favor of
> ClassPath.PROVIDED and ClassPath.EXECUTE. By 'prefer' I mean it would be nice
> to do that but right now it is extra work I do not want to spend time on. 

Maybe file a separate issue for it. I don't think it blocks this issue, but it may need to be fixed to provide feature parity for Maven projects.

>> Note #2: there is javaee-web-api-6.0.jar in provided scope as well, which looks
>> to be a superset of javaee-endorsed-api-6.0.jar, which is confusing.
> 
> Name of "javaee-endorsed-api-6.0.jar" is probably confusing or not self
> descriptive.

What I found confusing was not the name but the fact that there are classes in both of the abovementioned JARs, i.e. they overlap. Is this intentional? Shouldn't -web-api.jar contain only the additional (EE-specific) classes?

> EE 6 APIs (which are on java.net) do not contain full Java class files.

Ugh, what a mess. Does this apply to the -endorsed.jar, the -web-api.jar, or both?

> If Maven users want to unit test a code which
> requires any EE 6 APIs they will have to add to their POM's real EE 6 jars from
> a server implementation they use.

Is this what Maven EE devs actually do - what is the custom? Do people normally just run in-server tests using e.g. htmlunit? Do these considerations also apply to users of mocking frameworks? What about people running a CAR project?


Anyway, I think the action item for this issue is to

1. Change the web archetype to use the endorsed dirs trick, unless and until MNG-4752 is implemented.

2. Change ClassPathModifierImpl to read and write POMs using the same trick, with some sort of backward compatibility for existing projects using the sun.boot.class.path hack.

3. Make xml.jaxb fail more gracefully in case it cannot alter the endorsed CP of a project.

(In reply to comment #13)
> Maybe file a separate issue for it. I don't think it blocks this issue, but it
> may need to be fixed to provide feature parity for Maven projects.

ok, bug 189331


> > Name of "javaee-endorsed-api-6.0.jar" is probably confusing or not self
> > descriptive.
> 
> What I found confusing was not the name but the fact that there are classes in
> both of the abovementioned JARs, i.e. they overlap. Is this intentional?
> Shouldn't -web-api.jar contain only the additional (EE-specific) classes?

That's weird. I would believe that -web-api.jar should contain only extra EE classes and nothing from SE. I will ask Ludo.

> > EE 6 APIs (which are on java.net) do not contain full Java class files.
> 
> Ugh, what a mess. Does this apply to the -endorsed.jar, the -web-api.jar, or
> both?

Both. Actually it is three files: javaee-api.jar, javaee-api-web.jar and endorsed one. EE 6 introduced concept called Profiles and javaee-api-web is "EE 6 Web Profile" - that is an API subset of Java EE 6 aimed for servers like Tomcat.

> > If Maven users want to unit test a code which
> > requires any EE 6 APIs they will have to add to their POM's real EE 6 jars from
> > a server implementation they use.
> 
> Is this what Maven EE devs actually do - what is the custom? Do people normally
> just run in-server tests using e.g. htmlunit? Do these considerations also
> apply to users of mocking frameworks? What about people running a CAR project?

That's something I do not know. The bank I used to work for used mockup objects and it worked fine. For some tests htmlunit is good idea too. They migrated recently to Maven from Ant and as far as I know they have their own internal repo so you can place there whatever you want. But it is hard for IDE to provide some out of the box setup.

CAR is I believe very rarely used.
 
> Anyway, I think the action item for this issue is to

Your plan sounds good.

(In reply to comment #8)
> Is there a concrete example of a class which exists in Java SE 6 in one version
> but has an enhanced API in a publicly downloadable official update?

Apparently: javax.annotation.Resource.lookup (exists in EE endorsement but not in SE 6)

The use of

<systemProperties>
    <property>
        <name>java.endorsed.dirs</name>
        <value>${endorsed.dir}</value>
    </property>
</systemProperties>

in the sample POM I attached has no real effect; Surefire seems to set system properties after starting the JVM, so it is too late (though System.getProperty reports the var as set). <systemPropertyVariables> which is now recommended seems to do nothing at all. useSystemClassLoader and useManifestOnlyJar do not help either. The only thing which does make the JVM interpret the endorsed JAR is

<argLine>-Djava.endorsed.dirs="${endorsed.dir}"</argLine>

Unfortunately...

(In reply to comment #12)
> EE 6 APIs (which are on java.net) do not contain full Java class files.

when you use this you get a segmentation fault at runtime. So punting on fixing the unit test runtime classpath (test compile classpath works fine by configuring maven-compiler-plugin).

Test case:

@Resource(lookup="hello")
public class C {}

public class CTest {
    @Test public void x() {
        System.err.println(System.getProperty("java.endorsed.dirs"));
        assertEquals("hello", C.class.getAnnotation(Resource.class).lookup());
    }
}

(In reply to comment #13)
> 1. Change the web archetype to use the endorsed dirs trick, unless and until
> MNG-4752 is implemented.

http://jira.codehaus.org/browse/MARCHETYPES-35

(In reply to comment #13)
> 2. Change ClassPathModifierImpl to read and write POMs using the same trick

Reading: core-main #035abf7e679d

(In reply to comment #13)
> 3. Make xml.jaxb fail more gracefully in case it cannot alter the endorsed CP
> of a project.

core-main #c06f19d020e6 but note that xml.jaxb seems to be expecting Ant-based projects so adding a JAXB binding merely adds the schema to the project but does nothing to the build. To really work, besides affecting the endorsed CP the module would need to actually configure the right Maven plugin. This means making some kind of JAXB + Maven module - a task for mgrebac, I guess.

> 2. Change ClassPathModifierImpl to read and write POMs using the same trick

Deferring the writing part until there is a use case. Any Maven-specific JAXB support could just as easily fix the dependencies of the project as its plugin list, so it does not make much sense to have generic logic for this.

> xml.jaxb...merely adds the schema to the project but does nothing to the build

This would be a task for Milan Kuchtiak, cc'ed. Milan, can you please investigate and file a new bug for this? Thanks.

> xml.jaxb...merely adds the schema to the project but does nothing to the build
I've reported an issue 189596.

Integrated into 'main-golden', will be available in build *201008180001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/035abf7e679d
User: Jesse Glick <jglick@netbeans.org>
Log: Part of #185139: treat target/endorsed/*.jar as part of ClassPath.ENDORSED.