Bug 57129

Created attachment 32134 [details]
Adds Arrays.sort(...) in the two identified methods

I found the problem running Linux Mint 17.
On Windows 7 64 bit, the files seems to be ordered alphabetically.

Applications that depend on JARs being searched for classes in a particular order are broken and should be fixed.

I am -1 on adding this unncessary bloat to the new resources implementation in Tomcat 8.

Broken web applications that need a JAR to be searched for classes before all other JARs can force this via configuration in the context.xml file. Something along the lines of the following should work:

<Resources>
  <!-- Trick to force this JAR to be searched for classes before all others
       to work around a Jira bug -->
  <PreResources className="org.apache.catalina.webresources.FileResourceSet"
                base="${catalina.base}/webapps/jira/WEB-INF/lib/jira-api-6.2.jar"
                webAppMount="/WEB-INF/lib/jira-api-6.2.jar" />
</Resources>

(In reply to Jörgen Persson from comment #2)
> I found the problem running Linux Mint 17.
> On Windows 7 64 bit, the files seems to be ordered alphabetically.

This has to do with the order in which the directory entries are returned by the underlying file system. Neither NTFS nor extXfs, etc. guarantee in which order directory entries are returned from a readdir. What you are observing on Windows versus Linux is entirely coincidental.

Hi Mark,

(In reply to Mark Thomas from comment #3)
> Applications that depend on JARs being searched for classes in a particular
> order are broken and should be fixed.
> 
> I am -1 on adding this unncessary bloat to the new resources implementation
> in Tomcat 8.

Any chance this could be revisited? I see 2 reasons why having a predictible order is necessary:
- you might consider an application which depends on the order of the jars broken but the issue here is that, if the order is inconsistent depending on the OS/filesystem, you have a good chance to have your application failing when you deploy it on another OS/FS or even from a deploy to another.
- I'm pretty sure a lot of people used this feature to override classes of other jars in a easily maintanable way (eg having 000-hibernate-override-1.0.0.jar for instance).

> Broken web applications that need a JAR to be searched for classes before
> all other JARs can force this via configuration in the context.xml file.
> Something along the lines of the following should work:
> 
> <Resources>
>   <!-- Trick to force this JAR to be searched for classes before all others
>        to work around a Jira bug -->
>   <PreResources className="org.apache.catalina.webresources.FileResourceSet"
>                
> base="${catalina.base}/webapps/jira/WEB-INF/lib/jira-api-6.2.jar"
>                 webAppMount="/WEB-INF/lib/jira-api-6.2.jar" />
> </Resources>

It's not something maintanable in a continuous deployment/Maven/gradle world. We update the jar versions very often and it's really not something we can do.

I really think guaranteeing a predictible order is following the POLA and adding a sort is really worth it.

Thanks for your feedback!

-- 
Guillaume

My position - and reasons for that position - remain unchanged.

Just one more comment about this.  We've recently upgrade to Tomcat 8 and encountered this.  The net effect is that Tomcat is now non-deterministic.  We have the same war file running on the version of Tomcat on two different machines.  One works, and one doesn't because of this.  I'll agree our .WAR file is faulty, but now I can do all the testing in the world of my .WAR file, and not even know if it's broken -- until some machine starts flaking out because the operating system returned the .JARs in a different order.

You can easily detect if the potential for problems exists. Look for classes duplicated in multiple JARs.

I have the same problem, my application need to override resources of third-party libs, for example config files and templates even classes, It works fine with tomcat7 and other application servers. I hope tomcat8 could fix this problem for robustness and compatibility

Please look at the previous answers. You can add to the discussion if you like, but please do not reopen the report yourself.

(In reply to quaff from comment #9)
> I have the same problem, my application need to override resources of
> third-party libs, for example config files and templates even classes, It
> works fine with tomcat7 and other application servers. I hope tomcat8 could
> fix this problem for robustness and compatibility

You can always write your own ClassLoader that mostly delegates to the superclass (WebappClassLoader) but prioritizes whatever libraries you want.

(In reply to Christopher Schultz from comment #11)
> (In reply to quaff from comment #9)
> > I have the same problem, my application need to override resources of
> > third-party libs, for example config files and templates even classes, It
> > works fine with tomcat7 and other application servers. I hope tomcat8 could
> > fix this problem for robustness and compatibility
> 
> You can always write your own ClassLoader that mostly delegates to the
> superclass (WebappClassLoader) but prioritizes whatever libraries you want.

 I think it is a regression bug, tomcat8 should fix it, maybe we should abandon tomcat8, stay with tomcat7 or switch to other servers.

I understand the argument that an application that depends on Tomcat reading its jar files alphabetically is broken, however we are talking about a behavior that has persisted since at least Tomcat 5. It may have been unintentional or just plain wrong, but there are a lot of projects out there that have come to depend on it. Moreover it's awfully hard to debug, since most people probably don't understand that they use that behavior.

I believe that the core problem is that some apps will fail depended on the underlying file system. If the target is to discourage people having the same classes in different jars, a warning could be added during classloading notifying users about that. Still, as Guillaume Smet has already mentioned, keeping track of them in large projects is not practical.

Even though I am on the fence about matching the behavior of earlier versions, at the very least it should be mentioned in the migration guide, with the PreResources workaround.

Hi Joachim,

The issue with PreResources is that you need to use an absolute path which is quite impractical.

Moreover, it doesn't solve the issue of having something predictable for class loading order.

It's far from being perfect but I created https://github.com/openwide-java/tomcat-classloader-ordered to work around this issue.

That being said, I still think it should be fixed directly in Tomcat.

HTH

-- 
Guillaume

(In reply to Guillaume Smet from comment #14)
> The issue with PreResources is that you need to use an absolute path which
> is quite impractical.

While absolute paths are required, they still can be parameterized. See Mark's example in comment #3 for how to make a JAR relative to where Tomcat is running.

We just begun converting your tomcat 6 and tomcat 7 Webfarm to Tomcat 8, and honestly, this is a show stopper for us.

We cannot, in a cluster setup, start Tomcat and have each instance have a different class loading behaviour.  This is especially important when serialization come into play with session sharing inside the cluster, where a class mismatch can lead to Serialization Exception.

Or with logging libraries, such as logback, which override existing libraries with their own to offer a logging facade. 

In an ideal world, yes, we would have a nice war with no overriding classes, with no dependencies pulling deep libraries. In an ideal world, we would use absolute path for a PreResources.

But this is not ideal. Maven-generated war with SNAPSHOT dependencies will have variables suffixes. Unless PreResources can support wildcards or prefixes in order to support versioning, this solution is not viable for most of us.

There was a similar use case back in Tomcat 7, when the support for War outside the webapp support was dropped, but added back later with a flag.


There exist applications that need a deterministic class loading. Heck, Java itself at it's core, is deterministic. That's the whole notion of classpath that is being ignored. If you do not want to make the jar sorting as the default behaviour, fine. But at the very least, offer it as an option for people to decide if they need it.

ie
<context jarloading="name|none|lastmodified" />


Thanks

We have upgraded from Tomcat 6 / Java 6 to Tomcat 8 / Java 7.
Everything was deployed and working successfully in tests environments and 3 prod servers. However, the deployment was failing in the two prod servers. The error faced is “java.lang.NoSuchMethodError”.

After reading this thread mentioning that Tomcat 8 loads the libraries in same folder depends on File System and not in alphabetical order. As a workaround, we decided explicitly just to rename the library facing the issue by putting “a-“ at the front of the name to see if the issue with the alphabetical ordering will persist. This resolved the issue. We tried also by putting “z-“ and it worked too. As soon as we re-named it to its original, the issue was evident again.

The behavior is confirmed when we display the libraries loaded in debug mode. We clearly see that the order of libraries loaded is different from an environment to the other.

Hello,

Give back a deterministic, reproducible classloader!!! 

 P.Busque ask at least an option to do that  
<context jarloading="name|none|lastmodified" />
I vote +1000000 on that!


I manage ~100 of web application on an old application server that has non deterministic classloader. I got regulary issue with 
- duplicate class (easy to fix)
- random XML parser / transformer /... (service based thing) 
- random bug when parsing XML to java class ( can be linked to xml parser, can be linked to different schema with different version because application need to manage this...)

Developers really like to say it runs on their computer and testing so production is not their problem but yours...   Violence is not the answer... they said...

Hi everybody,

We also have some problems due to random jar loading order after we moved from Tomcat 7 to Tomcat 8.  Application is not starting and displays error 'signer information does not match signer information of other classes in the same package'.
This is because we use birt report runtime library which contains some classes of 'commons.io' package.  Under Tomcat 7, 'commons.io' Jar is loaded first, but on Tomcat 8, birt Jar is loaded first, and Tomcat 8 raises an error when it loads 'commons.io' Jar Later.

We are talking about two lines of code to add a 'sort', compared to hundred of teams that will lose days and money to debug their applications. That's just unfair. So, please, add the 'sort' to load jars in predictable order.

Our application is huge and deployed on more than 20 Prod instances. We cannot use Tomcat 8 because every instance would have some different random errors due to random jar loading order.

What is more painful than a random behavior, especialy when the application is running in production, with thousand of users connected ?

We are forced to stay on Tomcat 7 :-(

(In reply to Mark Thomas from comment #8)
> You can easily detect if the potential for problems exists. Look for classes
> duplicated in multiple JARs.

Even if we can detect duplicated classes, we have no way to fix the problem because these classes are in dependencies JARs. So, For now, we cannot move to Tomcat 8.

Hello,
Please reconsider this bug, this is a show stopper for us.  Our application has unpredictible behavior under Tomcat 8. This is really a pain.  We cannot move to Tomcat 8.  Ordering Jars will not hurt anybody, but help a lot of persons.  Please, compare the time to fix the bug (add a call to sort) to the days maybe weeks lost because of bugs due to the random behavior.
Thanks.

See comment #3

It seems this issue is about fundamentalist versus pragmatism. Even if the order is deterministic, some application will still fail because not the order they are used to, but at least always fail. 

I hate when things are random and I try to compare logs file, it's a pain. Just for that reason I would have fixed this.

I can tell you that my organization spent so far at least $3000 USD in lost of time due to this.

I learned in 3rd grade high school that probability of success get lower fast if you have many tiny problem that doesn't seem useful to fix by themselves.

My app is like 10 years old, with many millions line of code, coded by 100+ developers.

(In reply to Sebastien Tardif from comment #23)
> It seems this issue is about fundamentalist versus pragmatism. Even if the
> order is deterministic, some application will still fail because not the
> order they are used to, but at least always fail. 
> 
> I hate when things are random and I try to compare logs file, it's a pain.
> Just for that reason I would have fixed this.
> 
> I can tell you that my organization spent so far at least $3000 USD in lost
> of time due to this.
> 
> I learned in 3rd grade high school that probability of success get lower
> fast if you have many tiny problem that doesn't seem useful to fix by
> themselves.
> 
> My app is like 10 years old, with many millions line of code, coded by 100+
> developers.

While I did not perform an exhaustive search, I know of no servlet container which explicitly guarantees JAR-file load-ordering within a particular directory. Yes, WEB-INF/classes will be loaded before WEB-INF/lib/*.jar but there is no explicit guarantee of the ordering among the JAR files. The servlet spec also does not mandate any JAR-load-ordering.

I actually support the idea of alphabetical JAR-load-ordering if for no other reason than it allows you to patch a server by dropping a new JAR file into WEB-INF/lib and starting the context. If that option is not available, you need to use container-specific features such a Tomcat's <PreResources>, etc.

Well,
after spending hundreds of man days to migrate our applications (including legacy) to new java and tomcat 9 I encoutered this... When installing application on production environment after all tests on test environments. It's non deterministic!

It's seems that some of you guys have not worked with real, huge and old application. That's why you are arguing to not implement of lib sorting...

Well, I want to redirect my all negative feelings into something constructive: if I'll fork tomcat on hithub and implement Philippe Busque proposition - is there any chance you will accept merge request?

It doesn't look like this will be integrated based on previous comments, but the move to git is supposed to make fork and customization like this easier.

Something to keep in mind is the possible impact of the Java module system in future versions of the Servlet spec. My understanding is that Java modules do not allow packages to exist in more than one JAR.

On the plus side, this should reduce the chances of an app having multiple JARs with the same classes as libraries refactor to meet the requirements of the Java module system. On the negative side, if Jakarta EE adopts the Java module system (I'd argue against that but I suspect I'd be in the minority) then the issue described here is only going to be the start of the problems apps are going to see.

I continue to be of the view that this is an issue best fixed in applications.

An offer to provide a patch is appreciated. I don't have easy access to the source code but somewhere in org.apache.catalina.webresources is probably the place to start.

This might end up as an option in Tomcat, it might turn into better hooks in Tomcat for a custom resources implementation (there are other use cases I am aware of that would benefit from that) or it might be a patch that could be dropped into Tomcat's lib dir.  Where it ends up will depend on what changes are required. The patch would have to be very minimal and the behaviour optional to be considered for inclusion in Tomcat.

(In reply to Mark Thomas from comment #27)
> The patch would have to be very minimal and the behaviour
> optional to be considered for inclusion in Tomcat.

Can you explain why this has to be optional? Is there any conceivable scenario where someone would prefer to have non-deterministic behavior?

(In reply to Mateusz Matela from comment #28)
> (In reply to Mark Thomas from comment #27)
> > The patch would have to be very minimal and the behaviour
> > optional to be considered for inclusion in Tomcat.
> 
> Can you explain why this has to be optional?

Because it's very nearly a spec violation. Assuming that users never switch application servers, it's probably harmless. But if you use a sorting-Tomcat and move to JBoss and your stuff stops working, JBoss will tell you the same thing: you were relying on some wacky behavior those crazy kids at Apache Tomcat were willing to do, and now you have to grow up and adhere to published specifications.

> Is there any conceivable scenario where someone would prefer to
> have non-deterministic behavior?

Probably not. Why do you have JAR files that rely on specific ordering to maintain determinism? I can't understand why someone would build an application like that.

Your move.

It's absolute regression. I used to create second party library to overlay resources(template or config) in third party library, now I need to modify third party library or extract files from second party library to classes directory before creating WAR.

(In reply to Christopher Schultz from comment #29)
> > Can you explain why this has to be optional?
>
> Because it's very nearly a spec violation.

That's a big stretch. If the spec doesn't specify some aspect of behavior, is it really the best approach to derive this behavior from some unrelated circumstances like the underlying file system?

> Assuming that users never switch
> application servers, it's probably harmless. But if you use a sorting-Tomcat
> and move to JBoss and your stuff stops working,

So you consider it a harm when some apps stop working on migration from Tomcat to JBoss, but it's good if they stop working on migration between versions of Tomcat? I see no logic here.

> JBoss will tell you the same
> thing: you were relying on some wacky behavior those crazy kids at Apache
> Tomcat were willing to do, and now you have to grow up and adhere to
> published specifications.

You can call it wacky, I call it practical.
What is your point here? Do you think that after this change people will get angry at Tomcat (more than in this bug) and demand that their apps stop working too? Going further this way, maybe there should be a config for more randomization in jar loading so that "incorrect" apps fail more often?

> Why do you have JAR files that rely on specific ordering to
> maintain determinism? I can't understand why someone would build an
> application like that.

Oh well, so I don't live in a perfect world and my apps are not perfect. But I lived with these imperfections for many years and everything was fine. Until someone in their ivory tower decided that these imperfections are actually more important than anything else I might want to work on... Or that I'm not worthy to use newer versions of Tomcat. Oh, thank you, Master!

Found this gem by accident. Now I understand some of our problems going from 7.0 to 9.0.

This is clearly a regression. With an easy fix. Not even "bloat".

I really fail the resistance of the - otherwise - very reasonable developers.

At https://bz.apache.org/bugzilla/show_bug.cgi?id=57129#c27 (July 2019) Mark Thomas said that a patch/PR would be welcome for review!
No one cared to send such for more than an year! If you really need this functionality then please do the work!

(In reply to mgrigorov from comment #33)

There's already a patch provided that adds two lines of code.
The unresolved issue is why this is not enough (i.e. there should be a new option for this).

Exactly. A patch was submitted long time ago, but ignored due to the refusal to see this as a regression (not on technical merit as far as I can see. Except the "bloat" comment). No need to ask for a new patch.

This is of course not a regression. Please read comments 3, 26 and 27 again.

Hi Remy,

so I do not see any of them explain why it is not a regression compared to the past (reliable) behavior.

#3 blames "broken applications" for a relying on a behavior that worked for years. Broken applications will always exist, but they are not broken until they suddenly start to fail. But I come from a different open source environment where breaking "user space" is not allowed except under severe [security related] circumstances.

#26 adds no technical content (sorry).

#27 talks about a [near] future Java feature that might completely change the situation by disallowing the current brokenness of applications. Which is fine, as it may force application writers to fix up their part. Or it just makes them stay on old Java. The same way as some people are now forced (or motivated) not to upgrade Tomcat.

Cheers
Martin
PS: sorry that I joined. But I am currently agitated about similar things in different environments
PPS: maybe this better goes into the mailing list

We still believe that depending on the order of the jars (or on the moon phase) is wrong!
Since there are some users (to be precise: 13 in 6 years!) who depend on this behavior then we are fine to add the sorting back but disabled by default, so new applications do not fall into this trap. Thus a new patch with an option is needed!

The more I think about this, the more convinced I am that the proposed patch, even with the ordering behaviour made optional via configuration, is not the right way to address this issue.

The proposed approach (ordering the JARs alphabetically and then having Tomcat search them in order for classes) will not address the following issues:

- There is no guarantee that test environments will use the same ordering. Tests may execute with one version of a class while production runs with a different version leading to issues being missed during testing.

- When the Servlet pluggability requirements are considered things could get very messy, very quickly. The pluggability scan scans every JAR. It would be possible for an annotation to be found on one class during the scan only for that annotation to appear to be missing when Tomcat loaded the class.

- Any use of the service loader mechanism would need to be wrapped/customized/etc to ensure the expected behaviour.

The first two issues above are the issues I came up with while thinking about this for a few minutes and the third occurred to me while trying to write the conclusion to this post. I suspect there are other issues I haven't thought of.

I accept that some applications won't be affected by the above issues and/or there are ways to work around the above issues. However, those workarounds are likely to be  fragile.

A complete solution in Tomcat would be relatively invasive. Looking at the annotation scanning code with this in mind, changes would be required - particularly to the recently added option to scan JARs in parallel.

My concern with trying to make this an officially supported feature of Tomcat is that it will result in a steady stream of edge cases emerging which in turn would result in increasingly complex code - particularly around pluggability. I accept that this is a subjective view and that things might turn out differently but that is my best assessment based on the requirements expressed in this issue and my experiences working with the Tomcat code base.

I continue to think that the best and right way to fix this issue is to fix the applications so they no longer include multiple versions of the same class in different JARs in WEB-INF/lib.

For those users who prefer to modify Tomcat's behaviour, I'm happy to work on making it easier to provide custom WebResource implementations and to replace selected components of Tomcat's default WebResource implementation (so it would be easy to drop in a custom DirResourceSet that provided the behaviour in the attached patch for example). It was always the intention the WebResource would be easily extensible and the current code falls short in that area.

I'd be happy to discuss ways to improve the extensibility of the WebResources implementation on the dev@ list if anyone is interested in pursuing that option.

Finally, another option if you need one JAR to override an other (which I don't think has been mentioned in this thread) is to extract the contents of that JAR to WEB-INF/classes which is guaranteed by the spec to be searched before JARs in WEB-INF/lib. As long as there are no conflicts, you could probably get away with doing this for multiple JARs if necessary.

(In reply to Mark Thomas from comment #39)

So maybe this is the root of the misunderstanding here: nobody's asking for this to be a fully fledged officially supported feature with guarantees that it will work in every conceivable scenario. We're just asking not to break backwards compatibility - apps that worked in Tomcat 7 should not suddenly stop working in newer Tomcats just because of this issue that is so easy to avoid.

When people start to demand more, that's when you're fully justified to send them off.

(In reply to mgrigorov from comment #38)
> Since there are some users (to be precise: 13 in 6 years!) who depend on
> this behavior

That's only the number of people motivated enough to create/have an account here and comment. We don't know how many just read this thread, shake their head in disbelief and move on to maybe solve their problems some other (costly) way.

> then we are fine to add the sorting back but disabled by
> default, so new applications do not fall into this trap.

This will still force people who already are in this trap to potentially spend lots of time to investigate what's wrong, learn about this feature and turn it on. 
I don't think this trap is really so horrible to justify this approach (unless you intentionally make it to be).

(In reply to Mateusz Matela from comment #41)

> This will still force people who already are in this trap to potentially
> spend lots of time to investigate what's wrong, learn about this feature and
> turn it on. 
> I don't think this trap is really so horrible to justify this approach
> (unless you intentionally make it to be).

They really need to know about this problem and start fixing their application!

As Mark mentioned since recently Tomcat added functionality to scan the jars in parallel. No matter whether the jars are sorted or not the results from the parallel scanning will be shuffled.

(In reply to mgrigorov from comment #42)
> 
> As Mark mentioned since recently Tomcat added functionality to scan the jars
> in parallel. No matter whether the jars are sorted or not the results from
> the parallel scanning will be shuffled.

Now, this will be real fun. Instead of platform dependency, which can be handled by proper testing in staging environment, we will get race conditions and total unpredictability! 

Seems to be the time to have a closer look at
https://github.com/basepom/duplicate-finder-maven-plugin
to clean up conflicting maven dependencies.

I just hope that PreResources (see Comment 3) stay supported and won't be loaded in parallel to allow overriding buggy classes in included jar-files.

Regards
 Bernhard

(In reply to mgrigorov from comment #42)

> As Mark mentioned since recently Tomcat added functionality to scan the jars
> in parallel. No matter whether the jars are sorted or not the results from
> the parallel scanning will be shuffled.

My app doesn't even use annotation scanning. A lot of legacy apps don't.

Look, you can't save everyone who may have strange ideas about resource loading in the future. What we have now doesn't even save anyone, it only potentially makes them suffer earlier in hopes it would be less severe. But in order to achieve that, it makes lots of people suffer a lot right now. Mostly people, who just try to run their legacy app in a new Tomcat, because Tomcat 7 EOL approaches. That's mostly apps that won't use any fancy new features and may potentially never bump into this problem otherwise.

> They really need to know about this problem and start fixing their
> application!

OK, if this is such a serious and prevalent problem, then the solution should not be "oh, let's have the system behave non-deterministically. But only if the underlying file system is also non-deterministic." Do you see how that sounds?

Maybe Tomcat should scan the jars at context loading, detect all the duplicated resource paths and list them in logs with a big warning.
What we have now definitely causes more pain than it saves.

Oh my god this is just a gem that I got unnoticed until now...

It explain so much of the random behaviour we started to have on our legacy webapps since 2 years now when we ingenuously upgraded from Tomcat 7 to 9 !

Now we were so freaked that a new deployment would break everything that we didn't released even security fixes for those webapps unless it is an absolute case of force majeure !

We even had to put multiple test cases at server startup to ensure the behaviour of the webapps was consistent with what it was on our test platforms, that was so a sick thing to do ! And now we can finaly put a word onto this mess !

For god sake, why was it so important to break a behaviour that was consistent for at least 10 years and was known to many even if not mentioned anywhere in the specs, only by pure rigorism or even "fundamentalist" as the term was rightly pinned in a previous comment ?

How can a 17 characters sort instruction, that was there for as long I can recall using tomcat, be considered as a threat to be removed while it was providing consistency whatever operating and file system was used ?

I can still remember having to cope with those classloading madness on JBoss in 2006 and the relief it was when our organisation decided to migrate to Tomcat whith which I was absolutly sure (too much as it seems) there would be no mess caused by third party libraries holding old classes that would override the right ones on the server environment while working like a charm on our development and test platforms.

As those times are apparently gone and ramdomness seems a more enjoyable behaviour to a lot of person here, we will have to do the only reasonable thing for the overriden classes that are used in those legacy webapps (as we can't possibly adapt the context.xml on our virtual environments each time we make a new release nor copying those jars to the bootstrap folder and coding a custom ClassLoader is clearly not an option here) : 

We will have to extract every overriding class from the concerned jars into the classes path by adding those to the resources folder of each component in order to be sure that they will be picked first in our tests and on the server side. Every module, for every war artefact, for every project. Yuck ! 🤢

So thank you for this dreadfull decision that took us so much time to figure and that will take some more to adapt to. And don't try invoking java jigsaw (that every sane minded people disable by default) or paralell laoding to justify that, it is just non sense.

Regards.

(In reply to Y. Savanier from comment #45)
> 
> For god sake, why was it so important to break a behaviour that was
> consistent for at least 10 years

You are wrong. The behavour was not consistent, even 10+ years ago. If you have ever run your app on servers with different file systems (e.g. on Linux vs on Windows), you would have noticed it right away.

Nice try but no, I've always run my webapps on linux machines and the Tomcat handling of jars at loading was always alphabeticaly ordered as far as I can remember which is what I'm talking about here (if you read my previous message correctly, I'm well aware that the handling on others webservers like Websphere or JBoss is unordered and fs dependent).

For example this is the source code for the tomcat 4.0 FileDirContext list method in 2001 :

    protected Vector list(File file) {

        Vector entries = new Vector();
        if (!file.isDirectory())
            return entries;
        String[] names = file.list();
        Arrays.sort(names);             // Sort alphabetically
....

So yeah, from early 2000 to 2014 appears like more than 10 years to me.

Cheers.

This bug report has been RESOLVED WONTFIX.

Please don't use Bugzilla to conduct a flame war. If you want to discuss this (again), please raise the issue on the users' or developers' mailing list.

(In reply to Christopher Schultz from comment #48)
> This bug report has been RESOLVED WONTFIX.

This ticket does not report a bug, and the issue was not resolved (as of Tomcat 8.5.13 anyway).


> Please don't use Bugzilla to conduct a flame war. If you want to discuss
> this (again), please raise the issue on the users' or developers' mailing
> list.

Mailing lists are discussion forums. Unless someone proposes a solution which has drawbacks and wants to consult about acceptability, the issue tracker is the place to discuss issues.

Thank you very much for reporting Jörgen and all those who commented constructively


My employer is one of those organizations which lost hours due to variability in loading order. For those who are still struggling to understand the behavior they are experiencing, this not only causes specific WAR files to change behavior on different Tomcat versions, but it also causes the same Tomcat version (8+) to treat 2 WAR files with the same archived contents differently. Indeed, 2 WAR files which extract to identical file sets can still be binarily different by having their files at different offsets.

Typically, with Maven, if a project depends on 3 libraries and pom.xml lists B before A and finally C, the WAR file can contain b.jar before a.jar and finally c.jar. But it can also be alphabetical (a.jar, b.jar, c.jar). I've verified that the same Maven version (3.8.6) building identical code, when invoked in equivalent ways, can result in such binarily different WAR-s, which - in our case - therefore behave differently under Tomcat 8.5 on Red Hat Enterprise Linux 7. Red Hat Enterprise Linux 7's readdir visibly returns in creation order. My best guess is Maven packages .jar files in alphabetical order on Microsoft Windows, while it uses Maven's Resolved Dependencies order when packaging on Red Hat Enterprise Linux.

If so, this effectively means that moving builds from Windows to GNU/Linux could suffice to cause regressions. I do not know how to avoid that besides ensuring that the WAR-s do not contain classes with identical fully qualified names.

(In reply to Mateusz Matela from comment #28)
> (In reply to Mark Thomas from comment #27)
> > The patch would have to be very minimal and the behaviour
> > optional to be considered for inclusion in Tomcat.
> 
> Can you explain why this has to be optional? Is there any conceivable
> scenario where someone would prefer to have non-deterministic behavior?

I would not call the behavior of Tomcat 8+ non-deterministic. It may be *unspecified*, and it may not be as deterministic as that of previous versions, but if we look only at the order of loading on a static system (where a package is already installed and remains untouched), readdir's order looks at least generally deterministic to me.

If the current order is considered determined and differs from the previous (alphabetical) order, that means going back to the previous behavior would change that order again, potentially causing the same kind of regressions that the change brought. Considering that the change was released more than 8 years ago, I do think of obvious scenarios where someone would prefer the current behavior.

(In reply to Philippe Cloutier from comment #50)
> Typically, with Maven, if a project depends on 3 libraries and pom.xml lists
> B before A and finally C, the WAR file can contain b.jar before a.jar and
> finally c.jar. But it can also be alphabetical (a.jar, b.jar, c.jar).

You should have your project structured such that each dependency is isolated. You should not, for example, have libfoo-with-dependencies.jar libbar.jar and libbaz.jar if libfoo-with-dependencies.jar contains its own dependencies e.g. classes from libbar.jar and libbaz.jar.

> I do not know how to avoid that besides
> ensuring that the WAR-s do not contain classes with identical fully
> qualified names.

They would have to be included in multiple JAR files.

this may be helpful for you:
https://github.com/ChristopherSchultz/duplicate-resource-finder