Bug 57699

It looks like some namespace munging is going to be required to properly support this. After making changes to ExtractorFactory and POIXMLDocumentPart to handle the differing core relationship type, it now fails at the xmlbeans level:

org.apache.xmlbeans.XmlException: error: The document is not a workbook@http://schemas.openxmlformats.org/spreadsheetml/2006/main: document element namespace mismatch expected "http://schemas.openxmlformats.org/spreadsheetml/2006/main" got "http://purl.oclc.org/ooxml/spreadsheetml/main"
	at org.apache.poi.xssf.usermodel.XSSFWorkbook.onDocumentRead(XSSFWorkbook.java:399)

Caused by: org.apache.xmlbeans.XmlException: error: The document is not a workbook@http://schemas.openxmlformats.org/spreadsheetml/2006/main: document element namespace mismatch expected "http://schemas.openxmlformats.org/spreadsheetml/2006/main" got "http://purl.oclc.org/ooxml/spreadsheetml/main"
	at org.apache.xmlbeans.impl.store.Locale.verifyDocumentType(Locale.java:459)
	at org.apache.xmlbeans.impl.store.Locale.autoTypeDocument(Locale.java:364)
	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1280)
	at org.apache.xmlbeans.impl.store.Locale.parseToXmlObject(Locale.java:1264)
	at org.apache.xmlbeans.impl.schema.SchemaTypeLoaderBase.parse(SchemaTypeLoaderBase.java:345)
	at org.openxmlformats.schemas.spreadsheetml.x2006.main.WorkbookDocument$Factory.parse(Unknown Source)

The purl namespace crops up in most of the xml files at least somewhere, so a general mapping solution is probably required if we want to take this further

*** Bug 57914 has been marked as a duplicate of this bug. ***

http://pyxb.sourceforge.net/PyXB-1.2.2/bundles.html has a list of namespace URLs that could be used in a mapping class.

Without spending much time on this, I have been unable to track down the XSDs with the purl namespaces (OOXML Strict). From accounts, they should be very similar to the OOXML Transitional schemas other than the namespaces.
2 approaches pop to mind.
1. In poi-ooxml-schemas, we could create XmlBeans for the OOXML Strict namespaces by using modified versions of the OOXML Transitional schemas.
2. support a transformation of the XML in input docs so that the OOXML Strict namespaces are replaced by OOXML transitional equivalents.

I have added some basic prototype code to convert Strict OOXML files to https://github.com/pjfanning/ooxml-strict-converter - there is still a lot of work to do but I'm just posting it here if anyone wants to review what I'm doing.

Looks good so far.

In the interest of wanting to start committing this early so that we can update our unit tests to handle XSSF Strict:
* Are we planning on having XSSFWorkbook transparently handle strict workbooks or will be have a different class for that?
Will this be in the o.a.p.xssf.usermodel package or are we going to package it in o.a.p.xssf.extractor or create o.a.p.xssf.strict?

In the long term, I would like for POI to be able to read and write strict files without having to downconvert to non-strict. This probably affects how we go about packaging this--making it more than a distant examples or static utility converter class.

FYI, there is also a converter provided by Microsoft: https://www.microsoft.com/en-us/download/details.aspx?id=38828, could come in handy when doing development work on this topic.

Hi Javen,
I can understand that we will want to be able to save POI documents using Strict OOXML but my focus for now is just on the down-porting to Transitional OOXML to allow parsing.
For now, I'm looking at a standalone utility to down-port but this could be plugged into XSSFWorkbook and XSSF extractor under the hood. They could either do some pre-processing of the input doc to determine if it is Strict OOXML and the down-port to a temp file and then read from the temp file.
My prototype code is working now for the SimpleStrict.xlsx in the POI test data folder.
I'll see about testing with more input files.

Created attachment 35988 [details]
spam

Over two years have passed - has there been any work done on this / any milestone?

No, it seems none of the contributors needs it urgently enough to warrant spending time on it.

As this is a purely community supported project without commercial backing, your best bet to get progress on this will be to provide patches/time yourself if you can contribute in any way.

Yeah, which is why I was asking :> there were some partial results done by some people, I'll see if something can be done.

I am interested in working on this issue. Will be willing to work with some if somebody is already is working on it otherwise I can take it independently.

*** Bug 63847 has been marked as a duplicate of this bug. ***

There is some basic support in excel-streaming-reader, documented at https://github.com/pjfanning/excel-streaming-reader#ooxml-strict-format

Good Morning, I like to contribute to the support of strict OOXML files (as I want to get rid of my workaround scripts). What is the current state? Are there already some changes/progress in any branch? If not, I would try to use the information from the comments here and start working on that.

Hi Sebastian - supporting Strict OOXML format is a big undertaking and my impression is that this format has not really taken off (there aren't too many people asking about it). If you read my comment from 2021-02-13, you'll see that there is some support already but it is limited.

I would encourage you to spell out your plan because we may not necessarily want to support the strict format in poi-ooxml lib and might want to add a new jar dedicated to strict format - because we could end up bloating poi-ooxml by trying to support both formats in one jar.

Hello

I am in contact with MS about adding OOXML Strict conformance validation to their OOXML Validator (the respository is however hosted personally by an MS developer and it might in fact not be official software but the Open XML SDK links to it).

See the following correspondences for context:

https://github.com/mikeebowen/ooxml-validator-vscode/discussions/13
https://github.com/mikeebowen/ooxml-validator-vscode/issues/14

We are also very interested in having Apache POI support Strict data manipulation and validation.

For instance the prototype archival software Spreadsheet Complexity Analyser uses Apache POI to analyse spreadsheet documents and it could be improved by adding support for OOXML Strict conformance validation.

See repository:
https://github.com/RvanVeenendaal/Spreadsheet-Complexity-Analyser

Thanks Asbjorn. Could you read https://poi.apache.org/devel/guidelines.html and in particular, the piece about Microsoft NDAs? If you have any non-public info - possibly even private emails with a Microsoft employee - we will be unable to accept any contributions from you.

Asbjorn - supporting strict ooxml strict format is a massive task. poi-ooxml uses generated classes based on the OOXML Transitional XSDs. Generating the same classes for OOXML Strict XSDs and then having the all the XSSF classes or variants of them work with the OOXML Strict generated classes would be months of work and noone is volunteering to do it.

The best I can offer is https://github.com/pjfanning/excel-streaming-reader can read OOXML Strict files because it does not use the XSDs - but its API support is more limited than POI.

I do not have Excel installed but use https://onedrive.live.com/ to work with excel files. This hosted version of Excel doesn't even offer the ability to save in OOXML Strict format - only supports OOXML Transitional. As far as I can see OOXML Strict files are rare compared to OOXML Transitional files.

(In reply to PJ Fanning from comment #19)
> Thanks Asbjorn. Could you read https://poi.apache.org/devel/guidelines.html
> and in particular, the piece about Microsoft NDAs? If you have any
> non-public info - possibly even private emails with a Microsoft employee -
> we will be unable to accept any contributions from you.

Hello
Thanks for informing me.

I have not signed any NDA and all mentioned contact in my previous post has happened through their public repositories with MS developers.

Secondly, I am in verbal contact with one of their non-dev employees regarding our question of getting public information about what are their plans for future adoption and support of OOXML Strict. This is relevant for us to know, because we are a public archive and most documents in our government agencies are created in OOXML, and we want to know what are their future plans, before we potentially accept submission of OOXML documents into our archives. I hope you understand the relevancy for us to know this, and we also accept that any such knowledge should only be given to us, if it is suitable for public domain. I have no private e-mails answering this question and I have not received any verbal reply yet, but I have a future meeting in the calendar and I will make sure to mention, that I only want to receive information available to public. I will therefore share any information given, if anyone are interested.

I am very happy you inform me of this clause, so I will be aware of never signing any NDA with MS in the future. I hope our honesty, intentions and your guidance are sufficient for continuing conversation.

(In reply to PJ Fanning from comment #20)
> Asbjorn - supporting strict ooxml strict format is a massive task. poi-ooxml
> uses generated classes based on the OOXML Transitional XSDs. Generating the
> same classes for OOXML Strict XSDs and then having the all the XSSF classes
> or variants of them work with the OOXML Strict generated classes would be
> months of work and noone is volunteering to do it.
> 
> The best I can offer is https://github.com/pjfanning/excel-streaming-reader
> can read OOXML Strict files because it does not use the XSDs - but its API
> support is more limited than POI.
> 
> I do not have Excel installed but use https://onedrive.live.com/ to work
> with excel files. This hosted version of Excel doesn't even offer the
> ability to save in OOXML Strict format - only supports OOXML Transitional.
> As far as I can see OOXML Strict files are rare compared to OOXML
> Transitional files.


The lack of software support for conversion to and validation of OOXML Strict, might inform us to adopt OOXML Transitional for long-term preservation of archived spreadsheets, but it is not ideal, because we expect OOXML Transitional to be replaced by OOXML Strict in the future, hence we will have to migrate any already submitted OOXML Transitional spreadsheets. The question is really, will OOXML Strict ever replace OOXML Transitional!? As you say, OOXML Strict files are very rare today and software support both from MS and open-source community is insufficient.

Having Apache POI support conversion and validation would have tremendous impact from an archival point of view. I understand it is a huge task and no one is voluntering. I wanted to share our usercase, however, so the community will know demand exists, however marginal it might be.

Just found this issue and wanted to add another use case. We are using Tika, which uses POI, to extract plain text from MS files, including xlsx, docx etc. Some of these fall foul to this issue.

It would be great if this could be addressed in some way

I don't think there is any appetite among POI developers to take this on. One option would be to raise a Tika issue about them supporting https://github.com/pjfanning/excel-streaming-reader - it has support for OOXML Strict xlsx files.

As I understand it, your program converts Strict spreadsheets to Transitional spreadsheets. Getting rid of Strict doesn't really mean it supports Strict. It just gets rid of it so the rest of the processing can happen. It's a nice program and a nice hack, but it's not full support unless it can also save to Strict.

ndavies asked about Tika - and Tika only reads files to extract text - so excel-streaming-reader could be used to do this.

Anybody who wants to support writing ooxml-strict files are welcome to write code to do this.

Hi @PJ Fanning i am getting Strict OOXML isn't currently supported error for line in my code 'Workbook wb = WorkbookFactory.create(inputStream)' i want to get rid of this. You said ur codde is doing this but i am not understanding which code i should use as there are so many links you have provided. can you please tell which one to use.

Hi Asbjørn S. if you know which one to use can you pls help.

https://github.com/pjfanning/excel-streaming-reader supports OOXML strict format files.

Hi PJ Fanning in perticular do i need to use this link https://github.com/pjfanning/excel-streaming-reader-sample/blob/main/src/main/java/com/github/pjfanning/poi/sample/StreamingRead.java
please tell me this will be a great help.