Bug 65273

Summary:	NoClassDefFoundError in Apache POI dependency after upgrading to Tomcat 8.5.57 in Jira
Product:	Tomcat 8	Reporter:	Angelica Salazar <angelica.salazar>
Component:	Packaging	Assignee:	Tomcat Developers Mailing List <dev>
Status:	RESOLVED INVALID
Severity:	major
Priority:	P2
Version:	8.5.57
Target Milestone:	----
Hardware:	PC
OS:	All

Description Angelica Salazar 2021-04-28 14:26:47 UTC

Hello,

I am unsure if this is the correct channel to make this report. Let me know either way. I am from Easesolutions, vendor of R4J - Requirements Management for Jira. 

In our application, we are using the Apache POI library to support our Word Export capabilities. Recently, we have been receiving reports from customers with the same error:
java.lang.NoClassDefFoundError: Could not initialize class org.apache.poi.openxml4j.opc.internal.marshallers.ZipPackagePropertiesMarshaller
java.lang.NoClassDefFoundError: Could not initialize class org.apache.poi.openxml4j.opc.internal.marshallers.ZipPackagePropertiesMarshaller
	at org.apache.poi.openxml4j.opc.OPCPackage.init(OPCPackage.java:161) [?:?]
	at org.apache.poi.openxml4j.opc.OPCPackage.<init>(OPCPackage.java:141) [?:?]
	at org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:97) [?:?]
	at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:324) [?:?]

This looks like a dependency/classpath issue. The thing is, we have not changed our POI dependencies since 2017 and have released several versions with no change and now it's suddenly acting up. And after trying many things, we are now very lost. Ref to our developer community post: https://community.developer.atlassian.com/t/class-cannot-be-loaded-when-restarting-jira-software/47421/3

This is being experienced intermittently by customers. For example, some  have our application with the same version installed in multiple nodes but only experience the issue in 1 node. We have not been able to reproduce the issue in our own environments.

It seems that the common action for all these customers before experiencing the issue was that they upgraded Jira to a version that had Apache version 8.57 and up. Ref: https://confluence.atlassian.com/jiracore/bundled-tomcat-and-java-versions-1013854250.html

Hope you can help point us in the right direction.

Thank you and best regards,
Angelica

Comment 1 Angelica Salazar 2021-04-28 14:30:43 UTC

Correction: Apache version 8.5.57 and up

Comment 2 Mark Thomas 2021-04-28 18:13:11 UTC

First the good news. I can recreate this. I downloaded trail versions of Jira and R4J, created a single issue, requested an export and saw the exception and at the bottom of the stack trace:

"Caused by: java.lang.ClassCastException: org.apache.xerces.stax.XMLEventFactoryImpl cannot be cast to javax.xml.stream.XMLEventFactory"

The bad news is the Jira has its own class loader structure for plug-ins. I suspect a change in this area may be a factor. That isn't something the Tomcat team is going to be able to help with.

I'm technically a committer for POI (I helped out with the ALv1 to ALv2 license migration many, many years ago) but I do not know POI at all. I'll have a look to see if I can create a simpler test case and/or find any clues to what might be going on but the most likely result is that this issue will be closed as INVALID as there isn't any evidence that points to a Tomcat issue at the root if it.

Comment 3 Angelica Salazar 2021-04-29 03:28:43 UTC

(In reply to Mark Thomas from comment #2)
> First the good news. I can recreate this. I downloaded trail versions of
> Jira and R4J, created a single issue, requested an export and saw the
> exception and at the bottom of the stack trace:
> 
> "Caused by: java.lang.ClassCastException:
> org.apache.xerces.stax.XMLEventFactoryImpl cannot be cast to
> javax.xml.stream.XMLEventFactory"
> 
> The bad news is the Jira has its own class loader structure for plug-ins. I
> suspect a change in this area may be a factor. That isn't something the
> Tomcat team is going to be able to help with.
> 
> I'm technically a committer for POI (I helped out with the ALv1 to ALv2
> license migration many, many years ago) but I do not know POI at all. I'll
> have a look to see if I can create a simpler test case and/or find any clues
> to what might be going on but the most likely result is that this issue will
> be closed as INVALID as there isn't any evidence that points to a Tomcat
> issue at the root if it.

Hello Mark,

May I ask for the steps you did to produce the error: "Caused by: java.lang.ClassCastException: org.apache.xerces.stax.XMLEventFactoryImpl cannot be cast to javax.xml.stream.XMLEventFactory"?

An idea on the type of environment you installed it in will also be helpful as we're struggling to reproduce the issue on our own environments.

After this, if it's not related to Apache Tomcat, please close the issue. I truly appreciate the response. 

Thank you and best regards,
Angelica

Comment 4 Mark Thomas 2021-04-29 11:14:51 UTC

I can confirm that this isn't a Tomcat issue so I will be closing this as invalid.

To reproduce the issue I used the following:
- Ubuntu 20.04 fully patched
- Installed Jira latest (8.16.1)
- Installed R4J
- Created a simple project (TEST)
- Created a single issue
- Activated the TEST project in R4J
- Selected TEST from the Requirements menu
- Clicked Export -> Word

You only see the ClassCastException the first time you click export. To see it again you need to restart Jira.

I have done some digging via remote debugging and I can confirm that this is a class loading issue.

When R4J calls POI the XMLEventFactory class is loaded by an instance of BundleWiringImpl$BundleClassLoader. The parent of that class loader is Tomcat's web application class loader.

When POI calls XMLEventFactory.newInstance() that triggers the standard search using the Thread's context class loader (TCCL). Interestingly, the TCCL at this point is not the class loader that loaded XMLEventFactory above. The result is an instance of org.apache.xerces.stax.XMLEventFactoryImpl which is loaded by Tomcat's web application class loader. Its super class is XMLEventFactory BUT that class is loaded by the system class loader.

Hence when the assignment is attempted, you get the ClassCastException because you have two versions of XMLEventFactory loaded by different class loaders.

I don't know Jira's class loader structure well enough to provide a specific solution but I can say what a possible general solution might look like.

Option 1. Don't ship stax-api with R4J. Let it use the version provided by the JRE.

Option 2. Provide a stax implementation with R4J *and* ensure that the search mechanism finds it. (Maybe set TCCL to XMLEventFactory.class.getClassLoader() before the call. Remember to use a finally block to restore the old value afterwards.)

Option 3. Newer versions of POI don't use STAX - at least not at the point where the error is occurring. Updating to a newer version of POI may avoid the issue.