Bug 61478

Any chance you could work up a small junit unit test (probably with dummy classloader) to show the problem, and/or a fix?

I verified that this occurs also with POI 3.15.

I should be able to come up with a classloader code snippet that demonstrates the problem.  It will occur when trying to parse any Windows Office file, e.g. xlsx or docx.

I looked at providing an example but unfortunately, this occurs under the execution of Tika, which has many dozen dependent jars. If you want a code snippet, you'll either need to set up directory with all Tika dependent jars in it, or you will need to provide me a snippet of code which parses a Microsoft Office file.  Alternatively I can upload a many-megabyte zip file containing the Tika parser with all dependencies that you can just unpack.  Please let me know what you prefer.

Another way forward is to discuss how you use reflection in POI.  If you use the following method to locate your classes, all should be well:

https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#forName-java.lang.String-

But if you use this method, then you will have to be very certain you know what you are doing to get the right class loader:

https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#forName-java.lang.String-boolean-java.lang.ClassLoader-

I suspect it is the latter, and perhaps you are using the thread class loader rather than the current class's class loader?

Does using XWPFWordExtractor <http://poi.apache.org/apidocs/org/apache/poi/xwpf/extractor/XWPFWordExtractor.html> trigger the problem in your environment? That's the easiest way to do roughly what Tika does in just a few lines

XWPFWordExtractor doc = new XWPFWordExtractor(OPCPackage.open("input.docx"));
doc.getText();
doc.close();

If not, just send over a unit test that triggers the problem with Tika, we'll pop the test in the Tika codebase + fix here + check with Tika after our next release

(There's a good overlap in POI and Tika committers!)

Created attachment 35277 [details]
Testbed for exercising POI classloader

This testbed needs xmlbeans-2.6.0.jar, poi-3.15.jar, poi-ooxml-3.15.jar, and poi-ooxml-schemas-3.15.jar added to the lib directory after unzipping.

Please check if #60226, which was applied after 3.15, makes any difference

I've uploaded the testbed.  The testbed does not seem to cause the failure we were seeing, however.

If the codepath for a docx document goes through poi-ooxml-schemas, that means that we're seeing something that's being triggered somehow by Tika.

In the MCF setup, we have tika-core at the root level, and tika-parsers (and most of its dependencies, including poi) at the connector level.  This has worked in the past, at least until POI started using reflection to look up classes in poi-ooxml-schemas.

The classloader setup I'm using in the testbed is cribbed directly from ManifoldCF classes that set the class loaders up, so that's clearly not the issue.  Any ideas?

Could you provide some stack traces?

Created attachment 35278 [details]
User stack trace with all jars loaded by connector classloader

Image of user stacktrace when all Tika parser dependencies (including POI) loaded with connector classpath attached.

Created attachment 35279 [details]
Image of user stacktrace when poi-ooxml-schema.jar moved to root classloader

This is the user stacktrace when poi-ooxml-schema and poi-ooxml both moved to root classloader.

Mr. Beeker, I cannot readily modify Tika to call the new method.

Could you call the setClassLoader method before calling Tika code?
https://svn.apache.org/viewvc/poi/trunk/src/ooxml/java/org/apache/poi/POIXMLTypeLoader.java?view=markup&pathrev=1763922

Is the patched POI binary available on Maven yet?

Try poi 3.16 or 3.17-beta1. Both are in maven central.

OK -- I am certain that this workaround would solve the problem.  But it is pretty ugly, and we already have a workaround implemented and a patch release is up for a vote.

May I ask if the POI team attends to address this in a more official manner?  If not, perhaps the Tika team should?  If neither team wishes to address the issue, I will put this fix into ManifoldCF.

> But it is pretty ugly, and we already have a workaround implemented and a patch release is up for a vote.

Could you recommend how to use/identify the correct classloader?
(as we need access the .xsb files, it's neither the thread context classload, nor the classloader of any generated OOXML xml beans class)

As I had not much of an idea what the OSGi classloader does, I thought that this is the KISS way to solve the problem, but I'm happy to learn ... and if you ask nicely, we could postpone our final version which I wanted to prepare the release candidate today. 

> May I ask if the POI team attends to address this in a more official manner?

Official manner, as in a technical solution, if you know how, we will adapt it.
Otherwise it's officially in our FAQ:

http://poi.apache.org/faq.html#faq-N1029C

> We discovered this trying to integrate the latest Tika (which uses POI version 3.9) ...

Seriously? I always rant about users not able to use a recent version, but I never would
imagine that a PMC chair would write something like that

>>>
Seriously? I always rant about users not able to use a recent version, but I never would
imagine that a PMC chair would write something like that
<<<

My apologies -- the actual POI version we were using was 3.15, not 3.9, as I explained elsewhere.  I hope that is clear now.

>>>
Could you recommend how to use/identify the correct classloader?
<<<

Yes, as I explained above, what you really want to emulate is what happens when you do Class.forName(String classname).  There is a Class.forName() variant which accepts a passed-in class loader, which is what you use.  So you need to do this:

>>>
Invoking this method is equivalent to:
Class.forName(className, true, currentLoader)
where currentLoader denotes the defining class loader of the current class.
<<<

The defining class loader of the current class is:

xxx.Class.getClassLoader();

That should be the default behavior, I believe.  What do you think?

Just a short feedback ... as I'm not sure, if I can finish it this weekend.
When I tested #60226 / #57857, I used a sample project [1]
Additional to Glucazeaus project, one needs the additional service mix patch in #57857

If the Class.forName approach works in this and manifoldCF, it should be hopefully sufficient to render the workaround obsolete.


[1] https://github.com/glucazeau/test-poi-sling

From the attached stacktrace, and the sources, it can be seen that the error originates from 

org.openxmlformats.schemas.wordprocessingml.x2006.main.DocumentDocument.Factory.parse(*)

which is found in

ooxml-schemas-1.3.jar

This is a generated XMLBeans class.  Further, it isn't doing any explicit class loading, it is just importing a class.  So it looks to me like the issue is down in XMLBeans somewhere, or in the user's ClassLoader hierarchy.  

I searched the entire POI codebase, and there are no calls to *.forName(String, ClassLoader) or anything similar.  All calls are using the static Class.forName(String classname) method, which should be fine.

I'm fairly certain the issue for the end user is the placement of ooxml-schemas-1.3.jar in their class loader library structure.  It could be defined at too high a level, and thus inaccessible to a lower classloader, in which case that library should be pushed down rather than POI pulled up.  Note that this is a standard dependency for POI.

Argh, I should go to bed.  I see now the issue was calling from DocumentDocument.Factory in ooxml-schemas-1.3.jar to POIXMLTypeLoader in poi-ooxml.jar.

So it still could be the placement of ooxml-schemas-1.3.jar, but in the reverse case - it could be that the client code can see that jar, but that jar can't see POI.  It could be that ooxml-schemas.1.3.jar is loaded too early, and thus can't see POI.  Perhaps some library at a higher-order classloader also loads that jar somehow.

(In reply to Greg Woolsey from comment #19)
> From the attached stacktrace, and the sources, it can be seen that the error
> originates from 
> 
> org.openxmlformats.schemas.wordprocessingml.x2006.main.DocumentDocument.
> Factory.parse(*)
> 
> which is found in
> 
> ooxml-schemas-1.3.jar
> 
> This is a generated XMLBeans class.  Further, it isn't doing any explicit
> class loading, it is just importing a class.  So it looks to me like the
> issue is down in XMLBeans somewhere, or in the user's ClassLoader hierarchy.
> 
> 
> I searched the entire POI codebase, and there are no calls to
> *.forName(String, ClassLoader) or anything similar.  All calls are using the
> static Class.forName(String classname) method, which should be fine.
> 
> I'm fairly certain the issue for the end user is the placement of
> ooxml-schemas-1.3.jar in their class loader library structure.  It could be
> defined at too high a level, and thus inaccessible to a lower classloader,
> in which case that library should be pushed down rather than POI pulled up. 
> Note that this is a standard dependency for POI.

Hi Greg,

As I've explained before, there are exactly two classloader levels in ManifoldCF - the root level, and the "connectors" level.  In previous (working) Tika integrations, tika-core was at the root level, and tika-parsers (and all its dependencies, including xmlbeans) was at the "connectors" level.

When we went to the most recent version of Tika we discovered that having poi-ooxml-schemas at the higher level meant that the Tika connector could not find anything in poi-ooxml-schemas, even though that also was at the higher level, as was xmlbeans.  We needed to move ALL poi jars and their dependencies to the lower level in order to get it to work again.  (I'm in the process of voting on a point release for this as we speak so that our users will not be impacted.)

We'd much prefer having these jars once again live at the higher level, since putting them at the lower level means they must be included multiple times (in multiple webapps etc.), and that bloats our binary considerably.

Hope this helps.

If there are no invocations of Class.forName(String, boolean, ClassLoader), then there's no way the POI libraries can be a problem.  But a glance at the code shows that that's not entirely true; the POIXMLTypeLoader class does the latter.

It may be the case that the only reason for that code was written was to work around this problem when it was discovered by others (e.g. #60226).  But, in that case, the problem might well be that some other dependency, e.g. xmlbeans, is doing the wrong thing and you guys need to request a fix from them.

Here are some data points:

- Running all poi jars and their dependencies at the "connectors" level with poi-3.15 *definitely* uses the wrong classloader at some point -- probably the thread classloader
- The testbed I constructed and uploaded, on the other hand, *succeeds* - and that setup mimics MCF's classloader setup, but without Tika in between the MCF "connector" and the poi jars

Maybe the right approach is to analyze the difference in code paths between these two ways of calling into poi and see what differences there are, if any?  The stack traces are helpful here, yes, but maybe also looking at the testbed I uploaded could provide some insight into a way of getting into poi that does not seem to have the problem?

The only thing I'm pretty sure of is that it probably isn't Tika that does this, because it *does* manage to find the poi classes, just not those in poi-ooxml-schemas.

Created attachment 35283 [details]
Get Classloader from SchemaType instead of ContextClassLoader/preset CL

I've tested that patch against the OSGi example mentioned above (the latest servicemix bundle for POI 3.16 is also incomplete as far as I can say).
I wasn't sure (and I'm still not), if the cached SchemaTypeLoader would result  in a failing resource loading. At least I've added some kind of OOM-check to see if the ThreadLocals would result in memory leaks.

I should add ... if the schema type loader is not cached, this will result in OOM in the tests.

Karl (or should I say Mr. Wright?), I'll wait for your response, before I commit the patch and release(-candidate) POI 3.17 final - the last version to support Java 6.

Please apply the patch to the POI trunk, build it and give us feedback if it works for you.

Will do, as time permits.  Hopefully you should have an answer by tomorrow evening at the latest.

I was able to confirm that this fix corrected the problem in the MCF environment with Tika extraction.

Thanks to all for your help on this!

When do you expect this code to "go live" and be available for download from Maven?

Re-opening the bug, as I think Andreas still needs to commit the proposed fix to svn!

Release wise - we're holding off doing the 3.17 release candidate for this fix, as we want to play nicely with other ASF projects :)

Applied via r1807418

Colleagues, this fix caused a problem on WebLogic server.

1. WebLogic has own classloaders hierarchy: https://docs.oracle.com/cd/E24329_01/web.1211/e24368/classloading.htm#WLPRG323 
2. WebLogic Java process starts with own implementation of XMLBeans: WebLogic 10.3.6 uses XmlBeans 2.2.0 and 2.5.0 for internal purposes. This dependency is loaded by system classloader and used in child classloaders if it is not overridden explicitly. To make POI 3.17 work I had to add XMLBeans 2.6.0 to system classpath before WebLogic's dependencies.
2. Shared libraries for all web applications are placed in /lib directory. These libs are loaded by application classloader. I placed POI here (poi.jar, poi-ooxml.jar, ooxml-schemas.jar). 
3. Web applications are loaded by WebLogic classloaders. 

As a result, POIXMLTypeLoader creates SchemaTypeLoader using system classloader (which loaded XmlBeans), but lib with schemas (ooxml-schemas.jar) loaded by application classloader. System classloader doesn't have access to JARs loaded by application classloader which causes invalid XML generation for Word documents. 

Solution is simple: add all POI-related libs to system classpath, but it took me a lot of time to understand root cause of the problem.

Thanks for the advice, but puttings jars (with dependencies) into the /lib directory is something I really like to workaround.

But ...

This issue has already quite some history and OSGi problems are a bit different to cascaded classloading of web application servers.
- you haven't mentioned something about parent last setting [1]
- shared libs classloading is another topic which makes things complicated
- we usually don't provide fixes for old releases
- Weblogic 10 doesn't support Java 8 [2], so POI 4.0.0 is out-of-reach for you

With a Java 8 based Weblogic (12.1.2+), I would like to see, if we can keep the jars in the application context.

So if anyone wants to improve the situation for Weblogic 12.1.2, please open a new bugzilla entry ...

[1] https://stackoverflow.com/questions/31583968/how-to-specify-order-of-jar-class-loading-in-weblogic

[2] https://stackoverflow.com/questions/22513660/jre-8-compatibility-with-weblogic-10-3-6-11g