Bug 57129 - Regression. Load WEB-INF/lib jarfiles in alphabetical order
Summary: Regression. Load WEB-INF/lib jarfiles in alphabetical order
Status: RESOLVED WONTFIX
Alias: None
Product: Tomcat 8
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 8.0.x-trunk
Hardware: PC Linux
: P2 normal with 35 votes (vote)
Target Milestone: ----
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-10-22 06:39 UTC by Jörgen Persson
Modified: 2021-03-25 12:37 UTC (History)
6 users (show)



Attachments
Adds Arrays.sort(...) in the two identified methods (1.88 KB, patch)
2014-10-22 06:42 UTC, Jörgen Persson
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jörgen Persson 2014-10-22 06:39:35 UTC
When the classpath are created for the webapp classloader, the ordering of the jar files are not the same in Tomcat7 vs Tomcat8. 

This is due to that in Tomcat7, the FileDirContext.list(File) method sorts the jar files in the WEB-ING/lib folder alpabetically:
...
        Arrays.sort(names);             // Sort alphabetically
        NamingEntry entry = null;

        for (int i = 0; i < names.length; i++) {
...

The new design in Tomcat8 does not do this. I've identified two places where WEB-INF/lib is read:
StandardRoot.list(String, boolean)
and
DirResourceSet.listWebAppPaths(String)

Even though it is not a requirement that the entries are ordered alphabetically, it would be nice if they were. And there is no harm in doing it for web applications that does not depend on classpath ordering.

I've attached a patch file, tomcat8.patch, based on tomcat8 trunk (@ rev. 1633538).
Comment 1 Jörgen Persson 2014-10-22 06:42:19 UTC
Created attachment 32134 [details]
Adds Arrays.sort(...) in the two identified methods
Comment 2 Jörgen Persson 2014-10-22 07:31:39 UTC
I found the problem running Linux Mint 17.
On Windows 7 64 bit, the files seems to be ordered alphabetically.
Comment 3 Mark Thomas 2014-10-22 07:32:10 UTC
Applications that depend on JARs being searched for classes in a particular order are broken and should be fixed.

I am -1 on adding this unncessary bloat to the new resources implementation in Tomcat 8.

Broken web applications that need a JAR to be searched for classes before all other JARs can force this via configuration in the context.xml file. Something along the lines of the following should work:

<Resources>
  <!-- Trick to force this JAR to be searched for classes before all others
       to work around a Jira bug -->
  <PreResources className="org.apache.catalina.webresources.FileResourceSet"
                base="${catalina.base}/webapps/jira/WEB-INF/lib/jira-api-6.2.jar"
                webAppMount="/WEB-INF/lib/jira-api-6.2.jar" />
</Resources>
Comment 4 Christopher Schultz 2014-10-22 13:59:31 UTC
(In reply to Jörgen Persson from comment #2)
> I found the problem running Linux Mint 17.
> On Windows 7 64 bit, the files seems to be ordered alphabetically.

This has to do with the order in which the directory entries are returned by the underlying file system. Neither NTFS nor extXfs, etc. guarantee in which order directory entries are returned from a readdir. What you are observing on Windows versus Linux is entirely coincidental.
Comment 5 Guillaume Smet 2015-05-15 12:55:44 UTC
Hi Mark,

(In reply to Mark Thomas from comment #3)
> Applications that depend on JARs being searched for classes in a particular
> order are broken and should be fixed.
> 
> I am -1 on adding this unncessary bloat to the new resources implementation
> in Tomcat 8.

Any chance this could be revisited? I see 2 reasons why having a predictible order is necessary:
- you might consider an application which depends on the order of the jars broken but the issue here is that, if the order is inconsistent depending on the OS/filesystem, you have a good chance to have your application failing when you deploy it on another OS/FS or even from a deploy to another.
- I'm pretty sure a lot of people used this feature to override classes of other jars in a easily maintanable way (eg having 000-hibernate-override-1.0.0.jar for instance).

> Broken web applications that need a JAR to be searched for classes before
> all other JARs can force this via configuration in the context.xml file.
> Something along the lines of the following should work:
> 
> <Resources>
>   <!-- Trick to force this JAR to be searched for classes before all others
>        to work around a Jira bug -->
>   <PreResources className="org.apache.catalina.webresources.FileResourceSet"
>                
> base="${catalina.base}/webapps/jira/WEB-INF/lib/jira-api-6.2.jar"
>                 webAppMount="/WEB-INF/lib/jira-api-6.2.jar" />
> </Resources>

It's not something maintanable in a continuous deployment/Maven/gradle world. We update the jar versions very often and it's really not something we can do.

I really think guaranteeing a predictible order is following the POLA and adding a sort is really worth it.

Thanks for your feedback!

-- 
Guillaume
Comment 6 Mark Thomas 2015-05-15 13:14:28 UTC
My position - and reasons for that position - remain unchanged.
Comment 7 David Corbin 2015-06-25 10:46:17 UTC
Just one more comment about this.  We've recently upgrade to Tomcat 8 and encountered this.  The net effect is that Tomcat is now non-deterministic.  We have the same war file running on the version of Tomcat on two different machines.  One works, and one doesn't because of this.  I'll agree our .WAR file is faulty, but now I can do all the testing in the world of my .WAR file, and not even know if it's broken -- until some machine starts flaking out because the operating system returned the .JARs in a different order.
Comment 8 Mark Thomas 2015-06-25 10:58:12 UTC
You can easily detect if the potential for problems exists. Look for classes duplicated in multiple JARs.
Comment 9 quaff 2015-07-07 06:48:08 UTC
I have the same problem, my application need to override resources of third-party libs, for example config files and templates even classes, It works fine with tomcat7 and other application servers. I hope tomcat8 could fix this problem for robustness and compatibility
Comment 10 Remy Maucherat 2015-07-07 07:21:32 UTC
Please look at the previous answers. You can add to the discussion if you like, but please do not reopen the report yourself.
Comment 11 Christopher Schultz 2015-07-08 01:24:08 UTC
(In reply to quaff from comment #9)
> I have the same problem, my application need to override resources of
> third-party libs, for example config files and templates even classes, It
> works fine with tomcat7 and other application servers. I hope tomcat8 could
> fix this problem for robustness and compatibility

You can always write your own ClassLoader that mostly delegates to the superclass (WebappClassLoader) but prioritizes whatever libraries you want.
Comment 12 quaff 2015-07-08 07:02:13 UTC
(In reply to Christopher Schultz from comment #11)
> (In reply to quaff from comment #9)
> > I have the same problem, my application need to override resources of
> > third-party libs, for example config files and templates even classes, It
> > works fine with tomcat7 and other application servers. I hope tomcat8 could
> > fix this problem for robustness and compatibility
> 
> You can always write your own ClassLoader that mostly delegates to the
> superclass (WebappClassLoader) but prioritizes whatever libraries you want.

 I think it is a regression bug, tomcat8 should fix it, maybe we should abandon tomcat8, stay with tomcat7 or switch to other servers.
Comment 13 Joachim Economou 2015-10-07 08:36:18 UTC
I understand the argument that an application that depends on Tomcat reading its jar files alphabetically is broken, however we are talking about a behavior that has persisted since at least Tomcat 5. It may have been unintentional or just plain wrong, but there are a lot of projects out there that have come to depend on it. Moreover it's awfully hard to debug, since most people probably don't understand that they use that behavior.

I believe that the core problem is that some apps will fail depended on the underlying file system. If the target is to discourage people having the same classes in different jars, a warning could be added during classloading notifying users about that. Still, as Guillaume Smet has already mentioned, keeping track of them in large projects is not practical.

Even though I am on the fence about matching the behavior of earlier versions, at the very least it should be mentioned in the migration guide, with the PreResources workaround.
Comment 14 Guillaume Smet 2015-10-07 11:43:11 UTC
Hi Joachim,

The issue with PreResources is that you need to use an absolute path which is quite impractical.

Moreover, it doesn't solve the issue of having something predictable for class loading order.

It's far from being perfect but I created https://github.com/openwide-java/tomcat-classloader-ordered to work around this issue.

That being said, I still think it should be fixed directly in Tomcat.

HTH

-- 
Guillaume
Comment 15 Christopher Schultz 2015-10-07 15:37:00 UTC
(In reply to Guillaume Smet from comment #14)
> The issue with PreResources is that you need to use an absolute path which
> is quite impractical.

While absolute paths are required, they still can be parameterized. See Mark's example in comment #3 for how to make a JAR relative to where Tomcat is running.
Comment 16 Philippe Busque 2015-10-22 20:43:17 UTC
We just begun converting your tomcat 6 and tomcat 7 Webfarm to Tomcat 8, and honestly, this is a show stopper for us.

We cannot, in a cluster setup, start Tomcat and have each instance have a different class loading behaviour.  This is especially important when serialization come into play with session sharing inside the cluster, where a class mismatch can lead to Serialization Exception.

Or with logging libraries, such as logback, which override existing libraries with their own to offer a logging facade. 

In an ideal world, yes, we would have a nice war with no overriding classes, with no dependencies pulling deep libraries. In an ideal world, we would use absolute path for a PreResources.

But this is not ideal. Maven-generated war with SNAPSHOT dependencies will have variables suffixes. Unless PreResources can support wildcards or prefixes in order to support versioning, this solution is not viable for most of us.

There was a similar use case back in Tomcat 7, when the support for War outside the webapp support was dropped, but added back later with a flag.


There exist applications that need a deterministic class loading. Heck, Java itself at it's core, is deterministic. That's the whole notion of classpath that is being ignored. If you do not want to make the jar sorting as the default behaviour, fine. But at the very least, offer it as an option for people to decide if they need it.

ie
<context jarloading="name|none|lastmodified" />


Thanks
Comment 17 CBen 2016-02-25 20:44:00 UTC
We have upgraded from Tomcat 6 / Java 6 to Tomcat 8 / Java 7.
Everything was deployed and working successfully in tests environments and 3 prod servers. However, the deployment was failing in the two prod servers. The error faced is “java.lang.NoSuchMethodError”.

After reading this thread mentioning that Tomcat 8 loads the libraries in same folder depends on File System and not in alphabetical order. As a workaround, we decided explicitly just to rename the library facing the issue by putting “a-“ at the front of the name to see if the issue with the alphabetical ordering will persist. This resolved the issue. We tried also by putting “z-“ and it worked too. As soon as we re-named it to its original, the issue was evident again.

The behavior is confirmed when we display the libraries loaded in debug mode. We clearly see that the order of libraries loaded is different from an environment to the other.
Comment 18 wargre 2016-03-17 09:59:50 UTC
Hello,

Give back a deterministic, reproducible classloader!!! 

 P.Busque ask at least an option to do that  
<context jarloading="name|none|lastmodified" />
I vote +1000000 on that!


I manage ~100 of web application on an old application server that has non deterministic classloader. I got regulary issue with 
- duplicate class (easy to fix)
- random XML parser / transformer /... (service based thing) 
- random bug when parsing XML to java class ( can be linked to xml parser, can be linked to different schema with different version because application need to manage this...)

Developers really like to say it runs on their computer and testing so production is not their problem but yours...   Violence is not the answer... they said...
Comment 19 fchristol@gmail.com 2017-05-25 12:55:12 UTC
Hi everybody,

We also have some problems due to random jar loading order after we moved from Tomcat 7 to Tomcat 8.  Application is not starting and displays error 'signer information does not match signer information of other classes in the same package'.
This is because we use birt report runtime library which contains some classes of 'commons.io' package.  Under Tomcat 7, 'commons.io' Jar is loaded first, but on Tomcat 8, birt Jar is loaded first, and Tomcat 8 raises an error when it loads 'commons.io' Jar Later.

We are talking about two lines of code to add a 'sort', compared to hundred of teams that will lose days and money to debug their applications. That's just unfair. So, please, add the 'sort' to load jars in predictable order.

Our application is huge and deployed on more than 20 Prod instances. We cannot use Tomcat 8 because every instance would have some different random errors due to random jar loading order.

What is more painful than a random behavior, especialy when the application is running in production, with thousand of users connected ?

We are forced to stay on Tomcat 7 :-(
Comment 20 fchristol@gmail.com 2017-05-25 13:01:54 UTC
(In reply to Mark Thomas from comment #8)
> You can easily detect if the potential for problems exists. Look for classes
> duplicated in multiple JARs.

Even if we can detect duplicated classes, we have no way to fix the problem because these classes are in dependencies JARs. So, For now, we cannot move to Tomcat 8.
Comment 21 fchristol@gmail.com 2017-05-25 17:33:26 UTC
Hello,
Please reconsider this bug, this is a show stopper for us.  Our application has unpredictible behavior under Tomcat 8. This is really a pain.  We cannot move to Tomcat 8.  Ordering Jars will not hurt anybody, but help a lot of persons.  Please, compare the time to fix the bug (add a call to sort) to the days maybe weeks lost because of bugs due to the random behavior.
Thanks.
Comment 22 Mark Thomas 2017-05-26 07:22:22 UTC
See comment #3
Comment 23 Sebastien Tardif 2018-01-24 16:09:24 UTC
It seems this issue is about fundamentalist versus pragmatism. Even if the order is deterministic, some application will still fail because not the order they are used to, but at least always fail. 

I hate when things are random and I try to compare logs file, it's a pain. Just for that reason I would have fixed this.

I can tell you that my organization spent so far at least $3000 USD in lost of time due to this.

I learned in 3rd grade high school that probability of success get lower fast if you have many tiny problem that doesn't seem useful to fix by themselves.

My app is like 10 years old, with many millions line of code, coded by 100+ developers.
Comment 24 Christopher Schultz 2018-01-24 21:54:27 UTC
(In reply to Sebastien Tardif from comment #23)
> It seems this issue is about fundamentalist versus pragmatism. Even if the
> order is deterministic, some application will still fail because not the
> order they are used to, but at least always fail. 
> 
> I hate when things are random and I try to compare logs file, it's a pain.
> Just for that reason I would have fixed this.
> 
> I can tell you that my organization spent so far at least $3000 USD in lost
> of time due to this.
> 
> I learned in 3rd grade high school that probability of success get lower
> fast if you have many tiny problem that doesn't seem useful to fix by
> themselves.
> 
> My app is like 10 years old, with many millions line of code, coded by 100+
> developers.

While I did not perform an exhaustive search, I know of no servlet container which explicitly guarantees JAR-file load-ordering within a particular directory. Yes, WEB-INF/classes will be loaded before WEB-INF/lib/*.jar but there is no explicit guarantee of the ordering among the JAR files. The servlet spec also does not mandate any JAR-load-ordering.

I actually support the idea of alphabetical JAR-load-ordering if for no other reason than it allows you to patch a server by dropping a new JAR file into WEB-INF/lib and starting the context. If that option is not available, you need to use container-specific features such a Tomcat's <PreResources>, etc.
Comment 25 Kuba 2019-07-16 07:39:50 UTC
Well,
after spending hundreds of man days to migrate our applications (including legacy) to new java and tomcat 9 I encoutered this... When installing application on production environment after all tests on test environments. It's non deterministic!

It's seems that some of you guys have not worked with real, huge and old application. That's why you are arguing to not implement of lib sorting...

Well, I want to redirect my all negative feelings into something constructive: if I'll fork tomcat on hithub and implement Philippe Busque proposition - is there any chance you will accept merge request?
Comment 26 Remy Maucherat 2019-07-16 08:10:24 UTC
It doesn't look like this will be integrated based on previous comments, but the move to git is supposed to make fork and customization like this easier.
Comment 27 Mark Thomas 2019-07-16 09:05:28 UTC
Something to keep in mind is the possible impact of the Java module system in future versions of the Servlet spec. My understanding is that Java modules do not allow packages to exist in more than one JAR.

On the plus side, this should reduce the chances of an app having multiple JARs with the same classes as libraries refactor to meet the requirements of the Java module system. On the negative side, if Jakarta EE adopts the Java module system (I'd argue against that but I suspect I'd be in the minority) then the issue described here is only going to be the start of the problems apps are going to see.

I continue to be of the view that this is an issue best fixed in applications.

An offer to provide a patch is appreciated. I don't have easy access to the source code but somewhere in org.apache.catalina.webresources is probably the place to start.

This might end up as an option in Tomcat, it might turn into better hooks in Tomcat for a custom resources implementation (there are other use cases I am aware of that would benefit from that) or it might be a patch that could be dropped into Tomcat's lib dir.  Where it ends up will depend on what changes are required. The patch would have to be very minimal and the behaviour optional to be considered for inclusion in Tomcat.
Comment 28 Mateusz Matela 2020-06-01 11:45:31 UTC
(In reply to Mark Thomas from comment #27)
> The patch would have to be very minimal and the behaviour
> optional to be considered for inclusion in Tomcat.

Can you explain why this has to be optional? Is there any conceivable scenario where someone would prefer to have non-deterministic behavior?
Comment 29 Christopher Schultz 2020-06-01 20:55:29 UTC
(In reply to Mateusz Matela from comment #28)
> (In reply to Mark Thomas from comment #27)
> > The patch would have to be very minimal and the behaviour
> > optional to be considered for inclusion in Tomcat.
> 
> Can you explain why this has to be optional?

Because it's very nearly a spec violation. Assuming that users never switch application servers, it's probably harmless. But if you use a sorting-Tomcat and move to JBoss and your stuff stops working, JBoss will tell you the same thing: you were relying on some wacky behavior those crazy kids at Apache Tomcat were willing to do, and now you have to grow up and adhere to published specifications.

> Is there any conceivable scenario where someone would prefer to
> have non-deterministic behavior?

Probably not. Why do you have JAR files that rely on specific ordering to maintain determinism? I can't understand why someone would build an application like that.

Your move.
Comment 30 quaff 2020-06-04 03:00:47 UTC
It's absolute regression. I used to create second party library to overlay resources(template or config) in third party library, now I need to modify third party library or extract files from second party library to classes directory before creating WAR.
Comment 31 Mateusz Matela 2020-09-29 18:30:57 UTC
(In reply to Christopher Schultz from comment #29)
> > Can you explain why this has to be optional?
>
> Because it's very nearly a spec violation.

That's a big stretch. If the spec doesn't specify some aspect of behavior, is it really the best approach to derive this behavior from some unrelated circumstances like the underlying file system?

> Assuming that users never switch
> application servers, it's probably harmless. But if you use a sorting-Tomcat
> and move to JBoss and your stuff stops working,

So you consider it a harm when some apps stop working on migration from Tomcat to JBoss, but it's good if they stop working on migration between versions of Tomcat? I see no logic here.

> JBoss will tell you the same
> thing: you were relying on some wacky behavior those crazy kids at Apache
> Tomcat were willing to do, and now you have to grow up and adhere to
> published specifications.

You can call it wacky, I call it practical.
What is your point here? Do you think that after this change people will get angry at Tomcat (more than in this bug) and demand that their apps stop working too? Going further this way, maybe there should be a config for more randomization in jar loading so that "incorrect" apps fail more often?

> Why do you have JAR files that rely on specific ordering to
> maintain determinism? I can't understand why someone would build an
> application like that.

Oh well, so I don't live in a perfect world and my apps are not perfect. But I lived with these imperfections for many years and everything was fine. Until someone in their ivory tower decided that these imperfections are actually more important than anything else I might want to work on... Or that I'm not worthy to use newer versions of Tomcat. Oh, thank you, Master!
Comment 32 Martin Knoblauch 2020-09-30 08:02:49 UTC
Found this gem by accident. Now I understand some of our problems going from 7.0 to 9.0.

This is clearly a regression. With an easy fix. Not even "bloat".

I really fail the resistance of the - otherwise - very reasonable developers.
Comment 33 mgrigorov 2020-09-30 08:51:16 UTC
At https://bz.apache.org/bugzilla/show_bug.cgi?id=57129#c27 (July 2019) Mark Thomas said that a patch/PR would be welcome for review!
No one cared to send such for more than an year! If you really need this functionality then please do the work!
Comment 34 Mateusz Matela 2020-09-30 09:04:01 UTC
(In reply to mgrigorov from comment #33)

There's already a patch provided that adds two lines of code.
The unresolved issue is why this is not enough (i.e. there should be a new option for this).
Comment 35 Martin Knoblauch 2020-09-30 09:49:41 UTC
Exactly. A patch was submitted long time ago, but ignored due to the refusal to see this as a regression (not on technical merit as far as I can see. Except the "bloat" comment). No need to ask for a new patch.
Comment 36 Remy Maucherat 2020-09-30 09:58:17 UTC
This is of course not a regression. Please read comments 3, 26 and 27 again.
Comment 37 Martin Knoblauch 2020-09-30 10:20:48 UTC
Hi Remy,

so I do not see any of them explain why it is not a regression compared to the past (reliable) behavior.

#3 blames "broken applications" for a relying on a behavior that worked for years. Broken applications will always exist, but they are not broken until they suddenly start to fail. But I come from a different open source environment where breaking "user space" is not allowed except under severe [security related] circumstances.

#26 adds no technical content (sorry).

#27 talks about a [near] future Java feature that might completely change the situation by disallowing the current brokenness of applications. Which is fine, as it may force application writers to fix up their part. Or it just makes them stay on old Java. The same way as some people are now forced (or motivated) not to upgrade Tomcat.

Cheers
Martin
PS: sorry that I joined. But I am currently agitated about similar things in different environments
PPS: maybe this better goes into the mailing list
Comment 38 mgrigorov 2020-09-30 11:02:47 UTC
We still believe that depending on the order of the jars (or on the moon phase) is wrong!
Since there are some users (to be precise: 13 in 6 years!) who depend on this behavior then we are fine to add the sorting back but disabled by default, so new applications do not fall into this trap. Thus a new patch with an option is needed!
Comment 39 Mark Thomas 2020-09-30 11:09:07 UTC
The more I think about this, the more convinced I am that the proposed patch, even with the ordering behaviour made optional via configuration, is not the right way to address this issue.

The proposed approach (ordering the JARs alphabetically and then having Tomcat search them in order for classes) will not address the following issues:

- There is no guarantee that test environments will use the same ordering. Tests may execute with one version of a class while production runs with a different version leading to issues being missed during testing.

- When the Servlet pluggability requirements are considered things could get very messy, very quickly. The pluggability scan scans every JAR. It would be possible for an annotation to be found on one class during the scan only for that annotation to appear to be missing when Tomcat loaded the class.

- Any use of the service loader mechanism would need to be wrapped/customized/etc to ensure the expected behaviour.

The first two issues above are the issues I came up with while thinking about this for a few minutes and the third occurred to me while trying to write the conclusion to this post. I suspect there are other issues I haven't thought of.

I accept that some applications won't be affected by the above issues and/or there are ways to work around the above issues. However, those workarounds are likely to be  fragile.

A complete solution in Tomcat would be relatively invasive. Looking at the annotation scanning code with this in mind, changes would be required - particularly to the recently added option to scan JARs in parallel.

My concern with trying to make this an officially supported feature of Tomcat is that it will result in a steady stream of edge cases emerging which in turn would result in increasingly complex code - particularly around pluggability. I accept that this is a subjective view and that things might turn out differently but that is my best assessment based on the requirements expressed in this issue and my experiences working with the Tomcat code base.

I continue to think that the best and right way to fix this issue is to fix the applications so they no longer include multiple versions of the same class in different JARs in WEB-INF/lib.

For those users who prefer to modify Tomcat's behaviour, I'm happy to work on making it easier to provide custom WebResource implementations and to replace selected components of Tomcat's default WebResource implementation (so it would be easy to drop in a custom DirResourceSet that provided the behaviour in the attached patch for example). It was always the intention the WebResource would be easily extensible and the current code falls short in that area.

I'd be happy to discuss ways to improve the extensibility of the WebResources implementation on the dev@ list if anyone is interested in pursuing that option.

Finally, another option if you need one JAR to override an other (which I don't think has been mentioned in this thread) is to extract the contents of that JAR to WEB-INF/classes which is guaranteed by the spec to be searched before JARs in WEB-INF/lib. As long as there are no conflicts, you could probably get away with doing this for multiple JARs if necessary.
Comment 40 Mateusz Matela 2020-09-30 11:34:49 UTC
(In reply to Mark Thomas from comment #39)

So maybe this is the root of the misunderstanding here: nobody's asking for this to be a fully fledged officially supported feature with guarantees that it will work in every conceivable scenario. We're just asking not to break backwards compatibility - apps that worked in Tomcat 7 should not suddenly stop working in newer Tomcats just because of this issue that is so easy to avoid.

When people start to demand more, that's when you're fully justified to send them off.
Comment 41 Mateusz Matela 2020-09-30 11:52:48 UTC
(In reply to mgrigorov from comment #38)
> Since there are some users (to be precise: 13 in 6 years!) who depend on
> this behavior

That's only the number of people motivated enough to create/have an account here and comment. We don't know how many just read this thread, shake their head in disbelief and move on to maybe solve their problems some other (costly) way.

> then we are fine to add the sorting back but disabled by
> default, so new applications do not fall into this trap.

This will still force people who already are in this trap to potentially spend lots of time to investigate what's wrong, learn about this feature and turn it on. 
I don't think this trap is really so horrible to justify this approach (unless you intentionally make it to be).
Comment 42 mgrigorov 2020-09-30 12:14:45 UTC
(In reply to Mateusz Matela from comment #41)

> This will still force people who already are in this trap to potentially
> spend lots of time to investigate what's wrong, learn about this feature and
> turn it on. 
> I don't think this trap is really so horrible to justify this approach
> (unless you intentionally make it to be).

They really need to know about this problem and start fixing their application!

As Mark mentioned since recently Tomcat added functionality to scan the jars in parallel. No matter whether the jars are sorted or not the results from the parallel scanning will be shuffled.
Comment 43 Bernhard Seebass 2020-09-30 14:14:54 UTC
(In reply to mgrigorov from comment #42)
> 
> As Mark mentioned since recently Tomcat added functionality to scan the jars
> in parallel. No matter whether the jars are sorted or not the results from
> the parallel scanning will be shuffled.

Now, this will be real fun. Instead of platform dependency, which can be handled by proper testing in staging environment, we will get race conditions and total unpredictability! 

Seems to be the time to have a closer look at
https://github.com/basepom/duplicate-finder-maven-plugin
to clean up conflicting maven dependencies.

I just hope that PreResources (see Comment 3) stay supported and won't be loaded in parallel to allow overriding buggy classes in included jar-files.

Regards
 Bernhard
Comment 44 Mateusz Matela 2020-09-30 14:27:10 UTC
(In reply to mgrigorov from comment #42)

> As Mark mentioned since recently Tomcat added functionality to scan the jars
> in parallel. No matter whether the jars are sorted or not the results from
> the parallel scanning will be shuffled.

My app doesn't even use annotation scanning. A lot of legacy apps don't.

Look, you can't save everyone who may have strange ideas about resource loading in the future. What we have now doesn't even save anyone, it only potentially makes them suffer earlier in hopes it would be less severe. But in order to achieve that, it makes lots of people suffer a lot right now. Mostly people, who just try to run their legacy app in a new Tomcat, because Tomcat 7 EOL approaches. That's mostly apps that won't use any fancy new features and may potentially never bump into this problem otherwise.

> They really need to know about this problem and start fixing their
> application!

OK, if this is such a serious and prevalent problem, then the solution should not be "oh, let's have the system behave non-deterministically. But only if the underlying file system is also non-deterministic." Do you see how that sounds?

Maybe Tomcat should scan the jars at context loading, detect all the duplicated resource paths and list them in logs with a big warning.
What we have now definitely causes more pain than it saves.