Issue 125194 - Cannot work with shared documents on Sharepoint webdav
Summary: Cannot work with shared documents on Sharepoint webdav
Status: CLOSED FIXED
Alias: None
Product: General
Classification: Code
Component: code (show other issues)
Version: 4.1.0
Hardware: All Windows 7
: P3 Major with 2 votes (vote)
Target Milestone: 4.1.2
Assignee: AOO issues mailing list
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-07-03 12:32 UTC by davide
Modified: 2016-08-30 21:29 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT
Latest Confirmation in: ---
Developer Difficulty: ---
pescetti: 4.1.2_release_blocker+


Attachments
Webdav tests (13.66 KB, application/vnd.oasis.opendocument.spreadsheet)
2014-07-21 13:38 UTC, davide
no flags Details
Patch to fix i125194 (4.09 KB, patch)
2015-05-05 08:33 UTC, Giuseppe Castagno (aka beppec56)
no flags Details | Diff
Proposed patch to fix the issue, version #2 (3.16 KB, patch)
2015-06-10 08:33 UTC, Giuseppe Castagno (aka beppec56)
no flags Details | Diff
Last version of the patch to fix the issue (3.38 KB, patch)
2015-07-11 10:12 UTC, Giuseppe Castagno (aka beppec56)
giuseppe.castagno: review?
Details | Diff
Last minute fix, try to correct a malfunction (1.24 KB, patch)
2015-10-20 13:15 UTC, Giuseppe Castagno (aka beppec56)
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this issue.
Description davide 2014-07-03 12:32:10 UTC
Situation:
Server: Sharepoint 2010, IIS 7.5, Windows 2010 R2
Autentication: NTLM

When two users try to open the same document with AOO, the second receive the error: "The operation on \\mysite\sites\Rossi_M\Shared Documents\test.ods was started with an invalid parameter."

It seems that AOO cannot manage the PROPFIND answer received from Sharepoint.

This is the Open command from IP 10.10.40.65

2014-06-20 09:01:22 10.10.65.225 GET /sites/Rossi_M/Shared+Documents/test.ods - 80 RXXSDM\Rossi_M 10.10.40.65 Microsoft-WebDAV-MiniRedir/5.1.2600 rxxpoint 200 0 0 46

This is the creation of lock file

2014-06-20 09:02:47 10.10.65.225 PROPFIND /sites/Rossi_M/Shared+Documents/.~lock.test.ods# - 80 RXXSDM\Rossi_M 10.10.40.65 Microsoft-WebDAV-MiniRedir/5.1.2600 rxxpoint 400 0 0 31

and this is the temtative to open the file from another address 10.10.40.146

2014-06-20 09:07:19 10.10.65.225 GET /sites/Rossi_M/Shared+Documents/test.ods
- 80 RXXSDM\Rossi_M 10.10.40.146 Microsoft-WebDAV-MiniRedir/6.1.7601 rxxpoint 200 0 0 421

this is the reply that the file is locked with the PROPFIND

2014-06-20 09:07:45 10.10.65.225 LOCK /sites/Rossi_M/Shared+Documents/test.ods
- 80 REXXDM\Rossi_M 10.10.40.146 Microsoft-WebDAV-MiniRedir/6.1.7601
2014-06-20 09:07:45 10.10.65.225 PROPFIND /sites/Rossi_M/Shared+Documents/test.ods - 80 RXXSDM\Rossi_M 10.10.40.146 Microsoft-WebDAV-MiniRedir/6.1.7601 rxxpoint 207 0 0 0

A similar test has been done using .doc file with Word and Writer together. 

test.doc file has been open by Word on a webdav share.
Then the same file has been open by AOO. I obtain the same error:
"The operation on \\mysite\sites\Rossi_M\Shared Documents\test.doc was started with an invalid parameter."

Fiddler has been used to analize the traffic.
This is the HTTP request when AOO try to open the file:

LOCK http://mysite//sites/Rossi_M/Shared+Documents/test.doc HTTP/1.1
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/xml; charset="utf-8"
User-Agent: Microsoft-WebDAV-MiniRedir/6.1.7601
translate: f
Timeout: Second-3600
Connection: Keep-Alive
Content-Length: 206
Host: mysite

<?xml version="1.0" encoding="utf-8" ?><D:lockinfo xmlns:D="DAV:"><D:lockscope><D:exclusive/></D:lockscope><D:locktype><D:write/></D:locktype><D:owner><D:href>RXXSDM\rossi_m</D:href></D:owner></D:lockinfo>

and this is the Sharepoint answer that AOO cannot interpret and generates the error:

TTP/1.1 423 LOCKED
Cache-Control: private,max-age=0
Content-Length: 553
Content-Type: text/xml
Expires: Wed, 11 Jun 2014 09:43:56 GMT
Server: Microsoft-IIS/7.5
SPRequestGuid: 29938145-8eba-4530-b477-c185b9299677
X-SharePointHealthScore: 0
Public-Extension: http://schemas.microsoft.com/repl-2
X-MSDAVEXT_Error: 589838; Il%20file%20%22http%3a%2f%2fmysite%2fsites%2frossi%5fg%2fDocumenti%20Condivisi%2ftest%2edoc%22%20%c3%a8%20stato%20estratto%20per%20la%20modifica%20da%20RXXSDM%5crossi%5fg%2e
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.6115
X-MS-InvokeApp: 1; RequireReadOnly
Date: Thu, 26 Jun 2014 09:43:56 GMT

<?xml version="1.0" encoding="utf-8" ?><D:prop xmlns:D="DAV:" xmlns:Office="urn:schemas-microsoft-com:office:office" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:Z="urn:schemas-microsoft-com:"><D:lockdiscovery><D:activelock><D:locktype><D:write/></D:locktype><D:lockscope><D:exclusive/></D:lockscope><D:depth>0</D:depth><D:owner>RXXSDM\rossi_m</D:owner><D:timeout>Second-3567</D:timeout><D:locktoken><D:href>opaquelocktoken:{19543B86-762F-4610-BAB6-A2C71C8C5267}20140626T094324Z</D:href></D:locktoken></D:activelock></D:lockdiscovery></D:prop>
Comment 1 davide 2014-07-03 12:39:49 UTC
While AOO on Win7 produce an error, working on WinXP the file doesn't get locked so the last user who saves the file overwrites all the changes.
Comment 2 Ariel Constenla-Haile 2014-07-03 14:46:05 UTC
Did you try using the webdav protocol to open the file?
Comment 3 davide 2014-07-03 15:24:30 UTC
Files are open by using webdav protocol provided by Sharepoint/IIS.

We tried accessing both though URL via IE and share mounted as network drive. The behavior is the same.
Comment 4 Ariel Constenla-Haile 2014-07-03 15:44:36 UTC
The problems seems to be (that is, I don't have a Sharepoint server to test) that OpenOffice sees the file as a regular file, and thus tries to lock the file using the lock system it knows (the /sites/Rossi_M/Shared+Documents/.~lock.test.ods#).

If you use the webdav protocol (by using http:// or webdav://) then OpenOffice will not try to use the file based lock mechanism (though it will reveal another bug: LOCK is implemented in the webdav code, but it is not used in the office to lock the file; so that you can open locked files, but without being warned that the file is locked, and saving them will fail).
Comment 5 davide 2014-07-03 16:18:43 UTC
This happens also when files are open directly via webdav:

http://mysite//sites/Rossi_M/Shared+Documents/test.doc

It's the second case reported into the issue.
Comment 6 Ariel Constenla-Haile 2014-07-03 17:21:49 UTC
(In reply to davide from comment #5)
> This happens also when files are open directly via webdav:
> 
> http://mysite//sites/Rossi_M/Shared+Documents/test.doc
> 
> It's the second case reported into the issue.

If you open that document with OpenOffice using http://..., do you still get the error "The operation on \\mysite\sites\Rossi_M\Shared Documents\test.doc was started with an invalid parameter."?
Comment 7 Andrea Pescetti 2014-07-03 21:42:08 UTC
Davide: for better investigation, please set the "version" field appropriately. It is the FIRST problematic version. So, if you set it to 4.1.0, it means it used to work in 4.0.1; if you set it to 3.4.0 it means it used to work in 3.3.0. I'm not asking you to try with all versions, but especially if this is a regression it's helpful to know when it was working (we also replaced libraries with libserf at one point in history).
Comment 8 davide 2014-07-07 14:48:11 UTC
Ariel: we have tried to open the file directly via https://...

Credentials are asked and then AOO opens the file regardless it has been locked or not.
When the file is saved the following error is reported:

"Error saving the document test_webdav: the object cannot be created in directory https://... "
Comment 9 davide 2014-07-07 14:49:29 UTC
Andrea: what are the last version is useful for test if it's a regression? I mean what is the last version with neon?
Comment 10 Andrea Pescetti 2014-07-07 20:58:52 UTC
Replacement was discussed in November 2011, see http://markmail.org/message/7l2urjrbveeuhwrl (it is worth reading, there is some good discussion of other options).

So, version 3.4.0-beta and version 3.4.0 are good candidates to find out whether the issue is due to the neon->serf change happened between 3.4.0-beta and 3.4.0. 3.4.0 is available at http://www.openoffice.org/download/archive.html while 3.4.0-beta does not seem to be in the archives, e-mail me if you need a copy (or just use 3.3.0).
Comment 11 davide 2014-07-21 13:38:20 UTC
Created attachment 83717 [details]
Webdav tests

We have done some tests covering all the combinations between OS (win7, XP), applications (Word, Excel 2003 & 2010, Open Office 4.1 & 3.3) and documents opening order.

The only combination running is when AOO (both 3.3 & 4.1) opens .doc and .xls first in Win7 and MSO opens second regardless di SO.

All the other combinations (including AOO first and AOO second regardless the format, AOO first on WinXP) bring to an error or NO LOCK (LOCK ignored) situation.

Attached the complete report. 
Let me know how we can proceed in order to evaluate the effort to solve this problem.
Comment 12 Giuseppe Castagno (aka beppec56) 2014-12-02 14:37:25 UTC
Davide got in touch with me, and after discussing about this issue, I had a first look into the AOO code that manages WEBDAV.

For the reading part, that is mapping PROFIND properties to internal ucb properties, I found
something here:

http://opengrok.adfinis-sygroup.org/source/xref/aoo-trunk/main/ucb/source/ucp/webdav/ContentProperties.cxx#458

it seems that the profind elements (LOCKDISCOVERY & SUPPORTEDLOCK, for instance) are not added to the internal props
when the ucb properties are added.
Though it seems that the code was just prepared the place, but the code is missing.
Specifically for SUPPORTEDLOCK, here:

http://opengrok.adfinis-sygroup.org/source/xref/aoo-trunk/main/ucb/source/ucp/webdav/ContentProperties.cxx#567

I still have to find where the state (lock/no lock) is to be set or alternatively (e.g. when a file is opened for editing),
if there is already a foreseen place where the managing code should be added.

I guess that in the end this is the spec that should be looked at:

http://www.webdav.org/specs/rfc4918.html#locking

What I have in mind:
1) discuss about the current WEBDAV implementation in AOO, and then
2) implement the WEBDAV lock mechanism, that seems missing.

@Ariel
Any suggestion on all the above?
Comment 13 Giuseppe Castagno (aka beppec56) 2015-05-05 08:31:21 UTC
After some months of work, I came up with a possible solution for this issue.
The repository I use is on Github, a fork of the AOO mirror repo, link:

https://github.com/beppec56/openoffice

We needed to test using a stable release as a code base so we based all the work on AOO vers. 4.1.1.

A description of what happens under the hood
=============================================

On Windows 7 there is a driver installed, Windows MiniRedir (called Web Client service), that is used to carry out the low interface work with WebDAV.
With some limitation, since it seems to work only with Sharepoint WebDAV, and not, for example, with Apache WebDAV.

In the case described by this issue AOO tries to access the UNC provided (ex. \\server\shared-resource\file-to-open.doc)
using the local filesystem.

Observing the Wireshark log we registered, it appears that Windows tries to access the remote share first using Lanmanager protocols (SMB)
then by using the Web Client service.
The Web Client service checks the remote share to see if it lives on a Sharepoint server, in this case it takes care of the handshaking and perform
all that is needed to manage the resource on WebDAV, while presenting it to the requesting application, in this case AOO, as it where a
local file system resource.

What originate the AOO behavior is a missing Windows API error management in sal/osl.

To fix this specifically, I added the unmanaged Windows API error ERROR_FILE_CHECKED_OUT, treated as a file locked situation.
I added the necessary code to manage it in ucb project as well.

A side effect is AOO trying to use its lock file, but this is not possible, since the tilde '~' at the beginning of the private lock file
prevents the file creation on WebDAV.

To disable this behavior, I changed a default setting of the flag UseDocumentOOoLockFile in configuration and added a Windows API error not managed: ERROR_INVALID_NAME.
The '~' side effect disabled above is managed adding a new Windows API error: ERROR_INVALID_NAME.

I don't know if the configuration setting change impacts on AOO behavior, on current use it seems it doesn't.
It seems that this change enables AOO to honor the lock MS Office Word put on a file it opens when working locally, allowing interoperability of Word with Office
using the standard Windows file system.

I checked the patch only in Windows 7 using Word from Office 2010, the patched AOO trunk version and Sharepoint 2013.
I'm not sure how it works with different versions of Office.

In Windows XP doesn't work, because in XP the Web Client Service seems quite different from the one installed on Windows 7.

A patch containing the two changes, rebased onto trunk (r1677190) follows.
Comment 14 Giuseppe Castagno (aka beppec56) 2015-05-05 08:33:48 UTC
Created attachment 84729 [details]
Patch to fix i125194

See comment #13 above.
Comment 15 Giuseppe Castagno (aka beppec56) 2015-05-05 08:41:25 UTC
Possibly further upgrade on the issue solution
===============================================

While investigating the issue, we discovered that the Microsoft Office suite of programs works in another way WRT Sharepoint, according to some documentation
found on the net (see http://blogs.msdn.com/b/brian_jones/archive/2006/02/17/534142.aspx).

So may be the best thing to do is have AOO works the same way.

To implement this way we need:
- WebDAV locking in place, see below;
- have serf able to manage the NTLM authentication, something serf 1.2.1 currently in AOO does not support.
  According to serf more recent changelog, it was first introduced in sef 1.3.0.

WebDAV file locking implementation
==================================

I implemented the WebDAV file locking, per RFC4918 (http://tools.ietf.org/html/rfc4918).
It's on the repo I mentioned in comment #13:
https://github.com/beppec56/openoffice
At the moment the whole change is still spread among too many commit,
it contains too many debug logging I need to remove and needs to be rebased to trunk, being it on top of AOO 4.1.1.
I need to remove a new debug logging component I added (https://github.com/beppec56/openoffice/commit/9031a493ab5000227c4faf459ccb776eab703946),
that's not meant for general use.

To mantain clarity, I'll open another issue specific for WebDAV lock, updating the older issues on the same to point to this new one
(some are on OOo 1.1.1, other for OOo 3.x, all refers to pre serf era).
I'll do this as soon I have a simple test using Windows 7 Professional resident WebDAV, where I can enable Windows authentication.

I'll open a issue for serf upgrade as well, in order to have the Windows (NTLM) authentication support.
Comment 16 Giuseppe Castagno (aka beppec56) 2015-05-16 09:08:49 UTC
See issue 126305 for WebDAV lock implementation
Comment 17 Giuseppe Castagno (aka beppec56) 2015-06-10 08:33:17 UTC
Created attachment 84783 [details]
Proposed patch to fix the issue, version #2

This new version of the patch fixing the issue (posted on comment #14) contains a small change introduced to correct a regression showing up while managing AOO specific file locking in other OS, discovered on Linux.
Comment 18 Giuseppe Castagno (aka beppec56) 2015-06-10 08:37:39 UTC
Comment on attachment 84783 [details]
Proposed patch to fix the issue, version #2

In my own opinion, this patch should be evaluated to be added to AOO 4.1.2 version.
Comment 19 Andrea Pescetti 2015-06-21 19:00:58 UTC
Setting target 4.1.2 as per comments in issue and mailing lists.
Comment 20 Giuseppe Castagno (aka beppec56) 2015-07-11 10:12:46 UTC
Created attachment 84821 [details]
Last version of the patch to fix the issue

This version has a different comment, explaining what it changes, along with some typo corrected.

It's the last version, ready for inclusion.
It's based on trunk r1687177.
Comment 21 Andrea Pescetti 2015-08-26 06:54:04 UTC
Nominating as blocker for OpenOffice 4.1.2. This is the first patch in a set of three patches that, combined, solve a major interoperability (Sharepoint) bug.
Comment 22 Andrea Pescetti 2015-09-24 23:29:18 UTC
Now committed to both trunk and AOO410 for OpenOffice 4.1.2. Thank you, Giuseppe!
Comment 23 SVN Robot 2015-09-24 23:49:59 UTC
"pescetti" committed SVN revision 1705198 into branches/AOO410:
#i125194# Fix WebDAV file locking.
Comment 24 SVN Robot 2015-09-25 00:01:11 UTC
"pescetti" committed SVN revision 1705196 into trunk:
#i125194# Fix WebDAV file locking.
Comment 25 Giuseppe Castagno (aka beppec56) 2015-10-20 13:15:44 UTC
Created attachment 85055 [details]
Last minute fix, try to correct a malfunction

This patch tries to solve a problem that showed up while testing RC2.

It is created against 412-rc3 to-be (svn r1709406).
Comment 26 Andrea Pescetti 2015-10-20 21:56:40 UTC
Thank you Giuseppe, applied to trunk with revision 1709687 and to AOO410 for OpenOffice 4.1.2-RC3 with revision 1709685.
Comment 27 Giuseppe Castagno (aka beppec56) 2015-10-26 09:15:14 UTC
Checked on target Sharepoint 2013, found fixed on AOO 4.1.2-rc3
Comment 28 Giuseppe Castagno (aka beppec56) 2015-10-26 09:18:26 UTC
Forgot the platform: Windows 7 32bit.
Comment 29 Andrea Pescetti 2015-10-26 14:47:45 UTC
Marking VERIFIED as per comments by Giuseppe.
Comment 30 Kay 2016-08-30 21:29:15 UTC
CLosing.