Bug 64858 - Allow to deploy a WAR whose dependencies are on a Maven repository
Summary: Allow to deploy a WAR whose dependencies are on a Maven repository
Status: NEW
Alias: None
Product: Tomcat 10
Classification: Unclassified
Component: Catalina (show other bugs)
Version: unspecified
Hardware: PC All
: P2 enhancement (vote)
Target Milestone: ------
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-10-28 20:09 UTC by Gael Lalire
Modified: 2020-11-26 00:07 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Gael Lalire 2020-10-28 20:09:19 UTC
The war in the Maven repository will not have any lib in WEB-INF/lib, Tomcat should use Maven dependency resolving to get the dependencies.

Thanks to the missing WEB-INF/lib of the war its size will allow it to be publish in a Maven repository without a disk usage explosion.

Tomcat could use directly the Maven resolver project (https://github.com/apache/maven-resolver), however there is a lot of work on the classloader construction.

I created Vestige (https://gaellalire.fr/vestige/) which is able to manage classloader very precisely: you can group interdependent jar in same classloader, you can tell which jar should not be shared because of its use of static variable ...

I decided to create an API (https://github.com/vestige-java/vestige.spi.resolver.maven) that dynamic application can use to create and share classloader.

Then I modified some Tomcat to be sure my API is enough and the POC 
(https://github.com/vestige-java-app/tomcat_vestige) is able to load a this WAR (https://github.com/vestige-java-app/mywar) using it.

This POC was done on Tomcat 8.0.32, you could copy and correct the code in Tomcat master or maybe better create an extension point API which will allow to enable this functionality without overloading any Tomcat class.
Comment 1 Michael Osipov 2020-10-28 21:59:37 UTC
Why don't you do this yourself with Resolver Ant Tasks? The likelihood to have this in zTomcsg is almost zero.
Comment 2 Gael Lalire 2020-10-28 23:21:02 UTC
Yes, it is also possible to rebuild the WAR locally with maven resolver. However you will have multiple copies of same lib in your disk (more disk usage) and you lose the possibility to share lib's classes between WAR (more memory usage).
In addition classloader created by Vestige are faster to load classes, because it does not try each lib jar, it evaluates a minimized DFA to know in which jar the class should be loaded.

Also I made not many changes to only 6 classes (https://github.com/vestige-java-app/tomcat_vestige/commit/67dea6054c9da30047ebba3e9a376fa44b544f13#diff-e9240c9af00e44b55b8036434d325bea52d194dc43615ef31478c531d24751cc) in Tomcat and one (WebappClassLoaderBase.java) does not count because it is only removing illegal operations on JVM > 9 which newer version of Tomcat will not have.

With an extension point all the code to create classloader in another way will not be in Tomcat.

But I need someone, who find reduced disk and memory usage useful, to work on how to create this extension point.
Comment 3 Remy Maucherat 2020-10-29 00:09:30 UTC
It's great and all, but such a proprietary extension with extra dependencies has almost zero chances to be integrated with Tomcat. Anyway, this is an enhancement.
Comment 4 Gael Lalire 2020-10-29 07:44:18 UTC
There is nothing proprietary, all my code is open source.

And by extension point I mean a Tomcat extension point.

So for example in Tomcat :

interface MainResourceSetLoader {
  String getExtension();
  DirResourceSet load(File file);
}

in Tomcat-vestige :

class VestigeMainResourceSetLoader implements MainResourceSetLoader {
  String getExtension() { return ".vwar" }
  DirResourceSet load(File file) { /* read the vwar and create the resourceset */ }
}


So in StandardRoot.java you can do

private MainResourceSetLoader externalMainResourceSetLoader;
...
  if (externalMainResourceSetLoader != null && 
    file.endsWith(externalMainResourceSetLoader.getExtension())) {
    mainResourceSet = externalMainResourceSetLoader.load(file);
  }

That is what I mean by extension point.
Comment 5 Mark Thomas 2020-10-29 19:32:56 UTC
I have always viewed WebResource and friends as the extension point for things like this. Easy extensibility rather took a back seat as those interfaces evolved but I'd be happy to consider changes to them to make it easier to provide custom extensions (or complete replacement) of Tomcat's resources implementation.

I think it highly unlikely the necessary extension points to add additional deployment types identified by file extension will be added.
Comment 6 Gael Lalire 2020-10-31 14:18:58 UTC
Yes, WebResource are a terrific extension point. However it's a pity that scanners don't use it and rely on constructed jar: URL.

Anyway I already use WebResource but I need an entry point.

As long as JPMS named module are not activated I can overload Tomcat classes by a classpath hack.

But with JPMS named module activated I cannot use an already defined package that why I propose this PR https://github.com/apache/tomcat/pull/375.

Of course I created the extension point for my need, and it may be changed to a more generic way or split into multiple extension points.

Anyway I'm sorry to read that such integration has (almost) no change to get merged and I would like to know why do you think that.

------------

I also saw you now provide generated named bundle through BND.
I tested it and got 3 issues :
- missing uses, you can either configure BND (el-api.jar.tmp.bnd) to add it or add "ExpressionFactory.class.getModule().addUses(ExpressionFactory.class);" in the ExpressionFactory class.
- missing Module package on org.apache.tomcat.jasper.el, the includepackage option of jasper-el.jar.tmp.bnd seems ignored.
- JSP compilation failed, but I did not look onto it yet
Comment 7 Mark Thomas 2020-11-03 11:13:51 UTC
(In reply to Gael Lalire from comment #6)
> Yes, WebResource are a terrific extension point. However it's a pity that
> scanners don't use it and rely on constructed jar: URL.

Which code are you referring to here? The StandardJarScanner obtains URLs for the JARs via the WebResources implementation.

> Anyway I already use WebResource but I need an entry point.

That is expected to be the definition of a custom implementation for one or more components of the web resources implementation.

> As long as JPMS named module are not activated I can overload Tomcat classes
> by a classpath hack.
> 
> But with JPMS named module activated I cannot use an already defined package
> that why I propose this PR https://github.com/apache/tomcat/pull/375.

My preference continues to be doing this via WebResources and making changes there as necessary to permit extensions like this.

> Anyway I'm sorry to read that such integration has (almost) no change to get
> merged and I would like to know why do you think that.

Too invasive.


> I also saw you now provide generated named bundle through BND.
> I tested it and got 3 issues :

We are aware of those and they are being fixed via other bug reports.
Comment 8 Gael Lalire 2020-11-26 00:07:27 UTC
(In reply to Mark Thomas from comment #7)
> Which code are you referring to here? The StandardJarScanner obtains URLs
> for the JARs via the WebResources implementation.

StandardJarScanner.process is expecting either "jar:" or "file:" URL, so even if WebResource is allowing any URL we are forced to use one of these.
The good think is that JarScannerCallback is now using a org.apache.tomcat.Jar so if I can overload StandardJarScanner.process it will be good but :
- UrlJar is opening a connection and cast in JarURLConnection
- FragmentJarScannerCallback.extractJarFileName is expecting a jar URL.
- TagLibraryInfoImpl.<init> is opening a connection instead of using an inexistent Jar.getLastModified(<no-arg>) which could call WebResource.getLastModified
- StandardRoot.BaseLocation.<init> is expecting either jar or file. With jar it is expected a file inside. Should also use JarFactory.newInstance instead.
- maybe other ...

However it is not an issue for the moment I can use file and jar URL and still load from Maven repository.

(In reply to Mark Thomas from comment #7)
> My preference continues to be doing this via WebResources and making changes > there as necessary to permit extensions like this.

OK, I updated my PR putting only what I need. I changed only 5 tomcat files with minor changes (https://github.com/apache/tomcat/pull/375/files). I understand that the TomcatController I proposed is too invasive. Can you guide me on how to transform it so it becomes acceptable ?