Hi, we upgraded to tomcat-9.0.88 last night and our application is broken. We reverted to tomcat-9.0.87 and everything works fine. We use Ubuntu 22 LTS and Temurin OpenJDK jdk8u402. Looking at the stack trace, I think this changelog is relevant: https://tomcat.apache.org/tomcat-9.0-doc/changelog.html#Tomcat_9.0.88_(remm) Prevent the web application's ClassLoader from being pinned by the JSP compiler if an application uses a custom XMLInputFactory. Based upon a suggestion from Simon Niederberger. (schultz) We apply a custom XMLInputFactory and I think it is a classloader related problem. ----------------------------------- Here are the Stack Traces: 2024-04-17 06:24:27.904 ERROR 4q001b G5U001 ecadia.ecadia.GUIController.ActionServlet Unexpected Exception javax.servlet.ServletException: javax.xml.stream.FactoryConfigurationError: Provider com.ctc.wstx.stax.WstxInputFactory not found at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:332) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:200) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:641) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:415) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:347) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:284) at GUIController.ActionServlet.httpResponse(ActionServlet.java:1186) at GUIController.ActionServlet.doPost(ActionServlet.java:786) at GUIController.ActionServlet.doGet(ActionServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:200) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ecadia.servlet.filters.RedirectFilter.doFilter(RedirectFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:424) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1786) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) Caused by: javax.xml.stream.FactoryConfigurationError: Provider com.ctc.wstx.stax.WstxInputFactory not found at javax.xml.stream.FactoryFinder.newInstance(FactoryFinder.java:201) at javax.xml.stream.FactoryFinder.newInstance(FactoryFinder.java:152) at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:265) at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:227) at javax.xml.stream.XMLInputFactory.newFactory(XMLInputFactory.java:205) at org.apache.jasper.compiler.EncodingDetector.<clinit>(EncodingDetector.java:41) at org.apache.jasper.compiler.ParserController.determineSyntaxAndEncoding(ParserController.java:324) at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:201) at org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:128) at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:203) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:392) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:368) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:352) at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327) ... 40 more Caused by: java.lang.ClassNotFoundException: com/ctc/wstx/stax/WstxInputFactory at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at javax.xml.stream.FactoryFinder.getProviderClass(FactoryFinder.java:125) at javax.xml.stream.FactoryFinder.newInstance(FactoryFinder.java:189) ... 56 more --------------------------------------------- And then 2024-04-17 06:24:27.941 ERROR 4q001b G5U001 ecadia.ecadia.GUIController.ActionServlet forward error page failed! javax.servlet.ServletException: java.lang.NoClassDefFoundError: Could not initialize class org.apache.jasper.compiler.EncodingDetector at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:332) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:200) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:641) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:415) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:347) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:284) at GUIController.ActionServlet.httpResponse(ActionServlet.java:1205) at GUIController.ActionServlet.doPost(ActionServlet.java:786) at GUIController.ActionServlet.doGet(ActionServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:200) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at ecadia.servlet.filters.RedirectFilter.doFilter(RedirectFilter.java:104) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:169) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:424) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1786) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.jasper.compiler.EncodingDetector at org.apache.jasper.compiler.ParserController.determineSyntaxAndEncoding(ParserController.java:324) at org.apache.jasper.compiler.ParserController.doParse(ParserController.java:201) at org.apache.jasper.compiler.ParserController.parseDirectives(ParserController.java:128) at org.apache.jasper.compiler.Compiler.generateJava(Compiler.java:203) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:392) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:368) at org.apache.jasper.compiler.Compiler.compile(Compiler.java:352) at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:379) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:327) ... 40 more
I located the problem further: The WebApplication uses "woodstox-core-6.6.0.jar" which has a /META-INF/services/javax.xml.stream.XMLInputFactory pointing to com.ctc.wstx.stax.WstxInputFactory The Tomcat now seems to recognize this class but after the change in 9.0.88 is of cause not able to load this instance, since its in the parent (catalina) classloader.
Yeah, it comes down to these changes: https://github.com/apache/tomcat/commit/a2167e13c19115aecd220cd3be19d43d36126f3b https://github.com/apache/tomcat/commit/3b8f277a7ffc1193ed6c6d4fff85db6dc7327e39 We set a fixed "javax.xml.stream.XMLInputFactory" System.setProperty(javax.xml.stream.XMLInputFactory.class.getName(), WstxInputFactory.class.getName()); Before the change, the ContextClassLoader of our WebApplication was used to load the class. Now the ContextClassLoader is forced to the EncodingDetector.class.getClassLoader(). Was this anticipated?
:( Sorry about that. We'll re-evaluate our solution to the opposite problem, which is ClassLoader-pinning when the webapp's ClassLoader is used to obtain the XMLInputFactory.
(In reply to Chris from comment #2) > > System.setProperty(javax.xml.stream.XMLInputFactory.class.getName(), > WstxInputFactory.class.getName()); > The above line of code affects the whole JVM, not just your web application, breaking Tomcat and any other web applications that may be deployed in parallel. A web application by design is not allowed to change System properties. There are proper ways to reconfigure your environment, if you are really re-configuring your environment. (The conf/catalina.properties file and placing the shared classes into the lib/ directory, or any other directory specified via the "common.loader" property).
You are absolutely right. Setting a SystemProperty within a web application is not a desired approach. The circumstances of our web application are rather homogeneous: it is always installed as the one and only web application in Tomcat 9.0.* using Java 8 as a requirement. We are relying on some parsing features of woodstox and always instantiate their factory in our own code. But we call some libraries that themselves use "XMLInputFactory.newFactory()". We had issues in some class loader constellations that these libraries did not find the "/META-INF/services/javax.xml.stream.XMLInputFactory" of woodstox in the classpath and hence loaded the java sun default Implementation. We saw the SystemProperty as the only way to "enforce" the usage of woodstox throughout our entire web application.
Is it possible for you to use your custom XMLInputFactory directly instead of calling getInstance(), or do you need Tomcat's JSP compiler to use your custom XMLInputFactory?
Hi, we have a central utility method to create our own instances of "com.ctc.wstx.stax.WstxInputFactory" that is not the issue. But we use libraries, that in the depth just call "XMLInputFactory.newFactory()". They kept having problems in the context of sub ClassLoaders, not considering the service definition within the woodstox*.jar (/META-INF/services/javax.xml.stream.XMLInputFactory) and than failing with some missing features of the JDKs default implementation. And no, we do not require Tomcat to work with com.ctc.wstx.stax.WstxInputFactory. This was just a side effect that did not matter until now. Tomcat works with WstxInputFactory and with the sun default implementation. Chris
Arguably the this issue isn't a Tomcat issue. This bug is a result of the application using the work-around of setting the system property. That in turn is used due to application and/or library issues loading the correct XML parser. The fix to EncodingDetector is the right one to avoid web application class loader pinning. It appears from comment #2 that the system property is set programmatically in the web application. If that is the case then we may have a workaround for your workaround. If you add org.apache.jasper.compiler.EncodingDetector to the classesToInitialize property of the JreMemoryLeakPreventionListener that should trigger loading of the EncodingDetector before the web application starts and the system property is set. Long term, I'd suggest looking at the application and/or the libraries and trying to fix whatever issues mean you need to set the system property. Whether the above workaround solves the problem or not, I am leaning to resolving this as INVALID since it is caused by setting the system property which isn't appropriate in a web application environment.
Hello Mark, thanks for your lines. Indeed this is the workaround, I manually applied last week to apply the latest Tomcat versions to all installations. I herby withdraw my bug, since it is no issue of Tomcat. Thanks guys for your support. Chris