Bug 57129 - Regression. Load WEB-INF/lib jarfiles in alphabetical order
Summary: Regression. Load WEB-INF/lib jarfiles in alphabetical order
Status: RESOLVED WONTFIX
Alias: None
Product: Tomcat 8
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 8.0.x-trunk
Hardware: PC Linux
: P2 normal with 30 votes (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-10-22 06:39 UTC by Jörgen Persson
Modified: 2019-07-16 09:05 UTC (History)
4 users (show)



Attachments
Adds Arrays.sort(...) in the two identified methods (1.88 KB, patch)
2014-10-22 06:42 UTC, Jörgen Persson
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jörgen Persson 2014-10-22 06:39:35 UTC
When the classpath are created for the webapp classloader, the ordering of the jar files are not the same in Tomcat7 vs Tomcat8. 

This is due to that in Tomcat7, the FileDirContext.list(File) method sorts the jar files in the WEB-ING/lib folder alpabetically:
...
        Arrays.sort(names);             // Sort alphabetically
        NamingEntry entry = null;

        for (int i = 0; i < names.length; i++) {
...

The new design in Tomcat8 does not do this. I've identified two places where WEB-INF/lib is read:
StandardRoot.list(String, boolean)
and
DirResourceSet.listWebAppPaths(String)

Even though it is not a requirement that the entries are ordered alphabetically, it would be nice if they were. And there is no harm in doing it for web applications that does not depend on classpath ordering.

I've attached a patch file, tomcat8.patch, based on tomcat8 trunk (@ rev. 1633538).
Comment 1 Jörgen Persson 2014-10-22 06:42:19 UTC
Created attachment 32134 [details]
Adds Arrays.sort(...) in the two identified methods
Comment 2 Jörgen Persson 2014-10-22 07:31:39 UTC
I found the problem running Linux Mint 17.
On Windows 7 64 bit, the files seems to be ordered alphabetically.
Comment 3 Mark Thomas 2014-10-22 07:32:10 UTC
Applications that depend on JARs being searched for classes in a particular order are broken and should be fixed.

I am -1 on adding this unncessary bloat to the new resources implementation in Tomcat 8.

Broken web applications that need a JAR to be searched for classes before all other JARs can force this via configuration in the context.xml file. Something along the lines of the following should work:

<Resources>
  <!-- Trick to force this JAR to be searched for classes before all others
       to work around a Jira bug -->
  <PreResources className="org.apache.catalina.webresources.FileResourceSet"
                base="${catalina.base}/webapps/jira/WEB-INF/lib/jira-api-6.2.jar"
                webAppMount="/WEB-INF/lib/jira-api-6.2.jar" />
</Resources>
Comment 4 Christopher Schultz 2014-10-22 13:59:31 UTC
(In reply to Jörgen Persson from comment #2)
> I found the problem running Linux Mint 17.
> On Windows 7 64 bit, the files seems to be ordered alphabetically.

This has to do with the order in which the directory entries are returned by the underlying file system. Neither NTFS nor extXfs, etc. guarantee in which order directory entries are returned from a readdir. What you are observing on Windows versus Linux is entirely coincidental.
Comment 5 Guillaume Smet 2015-05-15 12:55:44 UTC
Hi Mark,

(In reply to Mark Thomas from comment #3)
> Applications that depend on JARs being searched for classes in a particular
> order are broken and should be fixed.
> 
> I am -1 on adding this unncessary bloat to the new resources implementation
> in Tomcat 8.

Any chance this could be revisited? I see 2 reasons why having a predictible order is necessary:
- you might consider an application which depends on the order of the jars broken but the issue here is that, if the order is inconsistent depending on the OS/filesystem, you have a good chance to have your application failing when you deploy it on another OS/FS or even from a deploy to another.
- I'm pretty sure a lot of people used this feature to override classes of other jars in a easily maintanable way (eg having 000-hibernate-override-1.0.0.jar for instance).

> Broken web applications that need a JAR to be searched for classes before
> all other JARs can force this via configuration in the context.xml file.
> Something along the lines of the following should work:
> 
> <Resources>
>   <!-- Trick to force this JAR to be searched for classes before all others
>        to work around a Jira bug -->
>   <PreResources className="org.apache.catalina.webresources.FileResourceSet"
>                
> base="${catalina.base}/webapps/jira/WEB-INF/lib/jira-api-6.2.jar"
>                 webAppMount="/WEB-INF/lib/jira-api-6.2.jar" />
> </Resources>

It's not something maintanable in a continuous deployment/Maven/gradle world. We update the jar versions very often and it's really not something we can do.

I really think guaranteeing a predictible order is following the POLA and adding a sort is really worth it.

Thanks for your feedback!

-- 
Guillaume
Comment 6 Mark Thomas 2015-05-15 13:14:28 UTC
My position - and reasons for that position - remain unchanged.
Comment 7 David Corbin 2015-06-25 10:46:17 UTC
Just one more comment about this.  We've recently upgrade to Tomcat 8 and encountered this.  The net effect is that Tomcat is now non-deterministic.  We have the same war file running on the version of Tomcat on two different machines.  One works, and one doesn't because of this.  I'll agree our .WAR file is faulty, but now I can do all the testing in the world of my .WAR file, and not even know if it's broken -- until some machine starts flaking out because the operating system returned the .JARs in a different order.
Comment 8 Mark Thomas 2015-06-25 10:58:12 UTC
You can easily detect if the potential for problems exists. Look for classes duplicated in multiple JARs.
Comment 9 quaff 2015-07-07 06:48:08 UTC
I have the same problem, my application need to override resources of third-party libs, for example config files and templates even classes, It works fine with tomcat7 and other application servers. I hope tomcat8 could fix this problem for robustness and compatibility
Comment 10 Remy Maucherat 2015-07-07 07:21:32 UTC
Please look at the previous answers. You can add to the discussion if you like, but please do not reopen the report yourself.
Comment 11 Christopher Schultz 2015-07-08 01:24:08 UTC
(In reply to quaff from comment #9)
> I have the same problem, my application need to override resources of
> third-party libs, for example config files and templates even classes, It
> works fine with tomcat7 and other application servers. I hope tomcat8 could
> fix this problem for robustness and compatibility

You can always write your own ClassLoader that mostly delegates to the superclass (WebappClassLoader) but prioritizes whatever libraries you want.
Comment 12 quaff 2015-07-08 07:02:13 UTC
(In reply to Christopher Schultz from comment #11)
> (In reply to quaff from comment #9)
> > I have the same problem, my application need to override resources of
> > third-party libs, for example config files and templates even classes, It
> > works fine with tomcat7 and other application servers. I hope tomcat8 could
> > fix this problem for robustness and compatibility
> 
> You can always write your own ClassLoader that mostly delegates to the
> superclass (WebappClassLoader) but prioritizes whatever libraries you want.

 I think it is a regression bug, tomcat8 should fix it, maybe we should abandon tomcat8, stay with tomcat7 or switch to other servers.
Comment 13 Joachim Economou 2015-10-07 08:36:18 UTC
I understand the argument that an application that depends on Tomcat reading its jar files alphabetically is broken, however we are talking about a behavior that has persisted since at least Tomcat 5. It may have been unintentional or just plain wrong, but there are a lot of projects out there that have come to depend on it. Moreover it's awfully hard to debug, since most people probably don't understand that they use that behavior.

I believe that the core problem is that some apps will fail depended on the underlying file system. If the target is to discourage people having the same classes in different jars, a warning could be added during classloading notifying users about that. Still, as Guillaume Smet has already mentioned, keeping track of them in large projects is not practical.

Even though I am on the fence about matching the behavior of earlier versions, at the very least it should be mentioned in the migration guide, with the PreResources workaround.
Comment 14 Guillaume Smet 2015-10-07 11:43:11 UTC
Hi Joachim,

The issue with PreResources is that you need to use an absolute path which is quite impractical.

Moreover, it doesn't solve the issue of having something predictable for class loading order.

It's far from being perfect but I created https://github.com/openwide-java/tomcat-classloader-ordered to work around this issue.

That being said, I still think it should be fixed directly in Tomcat.

HTH

-- 
Guillaume
Comment 15 Christopher Schultz 2015-10-07 15:37:00 UTC
(In reply to Guillaume Smet from comment #14)
> The issue with PreResources is that you need to use an absolute path which
> is quite impractical.

While absolute paths are required, they still can be parameterized. See Mark's example in comment #3 for how to make a JAR relative to where Tomcat is running.
Comment 16 Philippe Busque 2015-10-22 20:43:17 UTC
We just begun converting your tomcat 6 and tomcat 7 Webfarm to Tomcat 8, and honestly, this is a show stopper for us.

We cannot, in a cluster setup, start Tomcat and have each instance have a different class loading behaviour.  This is especially important when serialization come into play with session sharing inside the cluster, where a class mismatch can lead to Serialization Exception.

Or with logging libraries, such as logback, which override existing libraries with their own to offer a logging facade. 

In an ideal world, yes, we would have a nice war with no overriding classes, with no dependencies pulling deep libraries. In an ideal world, we would use absolute path for a PreResources.

But this is not ideal. Maven-generated war with SNAPSHOT dependencies will have variables suffixes. Unless PreResources can support wildcards or prefixes in order to support versioning, this solution is not viable for most of us.

There was a similar use case back in Tomcat 7, when the support for War outside the webapp support was dropped, but added back later with a flag.


There exist applications that need a deterministic class loading. Heck, Java itself at it's core, is deterministic. That's the whole notion of classpath that is being ignored. If you do not want to make the jar sorting as the default behaviour, fine. But at the very least, offer it as an option for people to decide if they need it.

ie
<context jarloading="name|none|lastmodified" />


Thanks
Comment 17 CBen 2016-02-25 20:44:00 UTC
We have upgraded from Tomcat 6 / Java 6 to Tomcat 8 / Java 7.
Everything was deployed and working successfully in tests environments and 3 prod servers. However, the deployment was failing in the two prod servers. The error faced is “java.lang.NoSuchMethodError”.

After reading this thread mentioning that Tomcat 8 loads the libraries in same folder depends on File System and not in alphabetical order. As a workaround, we decided explicitly just to rename the library facing the issue by putting “a-“ at the front of the name to see if the issue with the alphabetical ordering will persist. This resolved the issue. We tried also by putting “z-“ and it worked too. As soon as we re-named it to its original, the issue was evident again.

The behavior is confirmed when we display the libraries loaded in debug mode. We clearly see that the order of libraries loaded is different from an environment to the other.
Comment 18 wargre 2016-03-17 09:59:50 UTC
Hello,

Give back a deterministic, reproducible classloader!!! 

 P.Busque ask at least an option to do that  
<context jarloading="name|none|lastmodified" />
I vote +1000000 on that!


I manage ~100 of web application on an old application server that has non deterministic classloader. I got regulary issue with 
- duplicate class (easy to fix)
- random XML parser / transformer /... (service based thing) 
- random bug when parsing XML to java class ( can be linked to xml parser, can be linked to different schema with different version because application need to manage this...)

Developers really like to say it runs on their computer and testing so production is not their problem but yours...   Violence is not the answer... they said...
Comment 19 fchristol@gmail.com 2017-05-25 12:55:12 UTC
Hi everybody,

We also have some problems due to random jar loading order after we moved from Tomcat 7 to Tomcat 8.  Application is not starting and displays error 'signer information does not match signer information of other classes in the same package'.
This is because we use birt report runtime library which contains some classes of 'commons.io' package.  Under Tomcat 7, 'commons.io' Jar is loaded first, but on Tomcat 8, birt Jar is loaded first, and Tomcat 8 raises an error when it loads 'commons.io' Jar Later.

We are talking about two lines of code to add a 'sort', compared to hundred of teams that will lose days and money to debug their applications. That's just unfair. So, please, add the 'sort' to load jars in predictable order.

Our application is huge and deployed on more than 20 Prod instances. We cannot use Tomcat 8 because every instance would have some different random errors due to random jar loading order.

What is more painful than a random behavior, especialy when the application is running in production, with thousand of users connected ?

We are forced to stay on Tomcat 7 :-(
Comment 20 fchristol@gmail.com 2017-05-25 13:01:54 UTC
(In reply to Mark Thomas from comment #8)
> You can easily detect if the potential for problems exists. Look for classes
> duplicated in multiple JARs.

Even if we can detect duplicated classes, we have no way to fix the problem because these classes are in dependencies JARs. So, For now, we cannot move to Tomcat 8.
Comment 21 fchristol@gmail.com 2017-05-25 17:33:26 UTC
Hello,
Please reconsider this bug, this is a show stopper for us.  Our application has unpredictible behavior under Tomcat 8. This is really a pain.  We cannot move to Tomcat 8.  Ordering Jars will not hurt anybody, but help a lot of persons.  Please, compare the time to fix the bug (add a call to sort) to the days maybe weeks lost because of bugs due to the random behavior.
Thanks.
Comment 22 Mark Thomas 2017-05-26 07:22:22 UTC
See comment #3
Comment 23 Sebastien Tardif 2018-01-24 16:09:24 UTC
It seems this issue is about fundamentalist versus pragmatism. Even if the order is deterministic, some application will still fail because not the order they are used to, but at least always fail. 

I hate when things are random and I try to compare logs file, it's a pain. Just for that reason I would have fixed this.

I can tell you that my organization spent so far at least $3000 USD in lost of time due to this.

I learned in 3rd grade high school that probability of success get lower fast if you have many tiny problem that doesn't seem useful to fix by themselves.

My app is like 10 years old, with many millions line of code, coded by 100+ developers.
Comment 24 Christopher Schultz 2018-01-24 21:54:27 UTC
(In reply to Sebastien Tardif from comment #23)
> It seems this issue is about fundamentalist versus pragmatism. Even if the
> order is deterministic, some application will still fail because not the
> order they are used to, but at least always fail. 
> 
> I hate when things are random and I try to compare logs file, it's a pain.
> Just for that reason I would have fixed this.
> 
> I can tell you that my organization spent so far at least $3000 USD in lost
> of time due to this.
> 
> I learned in 3rd grade high school that probability of success get lower
> fast if you have many tiny problem that doesn't seem useful to fix by
> themselves.
> 
> My app is like 10 years old, with many millions line of code, coded by 100+
> developers.

While I did not perform an exhaustive search, I know of no servlet container which explicitly guarantees JAR-file load-ordering within a particular directory. Yes, WEB-INF/classes will be loaded before WEB-INF/lib/*.jar but there is no explicit guarantee of the ordering among the JAR files. The servlet spec also does not mandate any JAR-load-ordering.

I actually support the idea of alphabetical JAR-load-ordering if for no other reason than it allows you to patch a server by dropping a new JAR file into WEB-INF/lib and starting the context. If that option is not available, you need to use container-specific features such a Tomcat's <PreResources>, etc.
Comment 25 Kuba 2019-07-16 07:39:50 UTC
Well,
after spending hundreds of man days to migrate our applications (including legacy) to new java and tomcat 9 I encoutered this... When installing application on production environment after all tests on test environments. It's non deterministic!

It's seems that some of you guys have not worked with real, huge and old application. That's why you are arguing to not implement of lib sorting...

Well, I want to redirect my all negative feelings into something constructive: if I'll fork tomcat on hithub and implement Philippe Busque proposition - is there any chance you will accept merge request?
Comment 26 Remy Maucherat 2019-07-16 08:10:24 UTC
It doesn't look like this will be integrated based on previous comments, but the move to git is supposed to make fork and customization like this easier.
Comment 27 Mark Thomas 2019-07-16 09:05:28 UTC
Something to keep in mind is the possible impact of the Java module system in future versions of the Servlet spec. My understanding is that Java modules do not allow packages to exist in more than one JAR.

On the plus side, this should reduce the chances of an app having multiple JARs with the same classes as libraries refactor to meet the requirements of the Java module system. On the negative side, if Jakarta EE adopts the Java module system (I'd argue against that but I suspect I'd be in the minority) then the issue described here is only going to be the start of the problems apps are going to see.

I continue to be of the view that this is an issue best fixed in applications.

An offer to provide a patch is appreciated. I don't have easy access to the source code but somewhere in org.apache.catalina.webresources is probably the place to start.

This might end up as an option in Tomcat, it might turn into better hooks in Tomcat for a custom resources implementation (there are other use cases I am aware of that would benefit from that) or it might be a patch that could be dropped into Tomcat's lib dir.  Where it ends up will depend on what changes are required. The patch would have to be very minimal and the behaviour optional to be considered for inclusion in Tomcat.