Bug 45449 - [PATCH] Add support to WebDav to MOVE/COPY to remote servers
Summary: [PATCH] Add support to WebDav to MOVE/COPY to remote servers
Status: RESOLVED LATER
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_dav (show other bugs)
Version: 2.2.9
Hardware: Other Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: MassUpdate, PatchAvailable
Depends on:
Blocks:
 
Reported: 2008-07-21 21:02 UTC by Rafał Malinowski
Modified: 2018-11-07 21:07 UTC (History)
0 users



Attachments
Support for remote MOVE on resources (not collections) (21.31 KB, patch)
2008-07-21 21:02 UTC, Rafał Malinowski
Details | Diff
Version with MOVE and collections support. (29.01 KB, patch)
2008-07-23 10:30 UTC, Rafał Malinowski
Details | Diff
Version with LOCK support (31.96 KB, patch)
2008-07-25 11:39 UTC, Rafał Malinowski
Details | Diff
Version with copy-properties support (35.53 KB, patch)
2008-07-26 14:06 UTC, Rafał Malinowski
Details | Diff
Version with MULTI-STATUS responses. (36.51 KB, patch)
2008-08-02 06:24 UTC, Rafał Malinowski
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Rafał Malinowski 2008-07-21 21:02:29 UTC
Created attachment 22295 [details]
Support for remote MOVE on resources (not collections)

Hello. The WebDav RFC does not restrict MOVE/COPY to one server/dav provider, but the httpd implementation does.

My boss thinks that it should support remote MOVE/COPY operations.
I'am now working on it, and I'am sending a working path for remote COPY resources (not collections yet).

I would like to get some feedback and patch review, so I can continue this work and add functionality and remove some non-apache-way constructions from code (I just started to work on httpd code, so maybe I'am not aware on how thinks should be handled in code...).

Best regards
  Rafal Malinowski
Comment 1 Rafał Malinowski 2008-07-23 10:30:58 UTC
Created attachment 22304 [details]
Version with MOVE and collections support.

This version adds support for MOVE operations and also supports collections.
In next version I'll add use of LOCK on remote server when COPYing/MOVEing collections.

After that I'll try to add support for copying properties (as RFC says).
Comment 2 Rafał Malinowski 2008-07-25 11:39:22 UTC
Created attachment 22320 [details]
Version with LOCK support

This version LOCK remote NULL resource to prevent any conflicts. It releases the LOCK after the operation is finished (successfully or not).
Comment 3 Rafał Malinowski 2008-07-26 14:06:39 UTC
Created attachment 22321 [details]
Version with copy-properties support

This version copies also properties.
It needs some polish and some small fixes with other apache code (like the port problem with rev-proxy). And it needs a lot of testing...
Comment 4 Rafał Malinowski 2008-08-02 06:24:18 UTC
Created attachment 22345 [details]
Version with MULTI-STATUS responses.

This version is able to return MULTI STATUS response to client (if there is need to do it).
Comment 5 Mark Thomas 2009-06-10 13:29:25 UTC
Reset assignee so mails go to list.
Comment 6 Pierre Delaage 2009-10-07 10:39:50 UTC
(In reply to comment #5)
> Reset assignee so mails go to list.

Provided that proper usage of Apache Reverse Proxying is made, then NO PATCH is needed to make MOVE/COPY work as expected.

I Highly recommend that this patch SHOULD NOT be included in the apache source tree, although this is a good programming effort.

Here is a detailed memo I wrote on the subject, with 3 different solutions to this problem:

====================================================

I/ The problem: 
Briefly: one cannot rename, move or copy (from remote to remote) files on a https webdav location accessed via stunnel, from dav client such as Windows Webdav Mini-Redir (ie "net use" mapped drive).

However rename or copy work with "Add Network Place" wizard, but is very long for big files.
Also, via stunnel, accessing a classic www location works fine (provided that all url in html files are relative!)



II/ CAUSE:

Quick comments :
there is no bug in Apache,
no bug in stunnel,
and no bug (related to this pb)  in Windows webdav miniredir client

so what?...

The method MOVE and COPY of the dav protocol are most often implemented in dav clients by using an absolute destination url such as "http://myserver.mydomain/folder1" or "https://sslserver.otherdomain/folder2", WHILE the "source" document url is just expressed as a relative path to the "Host" designated in the http header of the request.
All other http and dav methods, which are essentially some kind of "get something methods" are most often implemented by using relative path such "/folderX/adoc" to designate a document of interest, "relative" means relative to the "host" field defined in the header of the http request. Note:  The PUT method is built the same way as the GET methods: ie with a relative path to the implicit "document root" of the server designated in the "Host:" http header field of the request (see RFC 2616 for http, RFC4918 for dav).
Well, from a vocabulary point of view, rfc2626 call my "relative path" as "abs-path" because they start with a "/", and not a dot (see chapter 3.2.1 of rfc2616).

That said, to secure a dav communication between a XP client and a DAV server, some people need to set up an ssl (s)tunnel between a WXP/Dav MiniRedir client and a HTTPS/dav server: this because WXP client are not able to manage "https connection" directly through "net use". So they usually use a setup like this on their client machine  stunnel.conf file:
accept: localhost:80
connect: remotehost:443
+ some other stuff for cert management.
The XP webdav miniredir client is only able to connect to 80 location. On XP "net use" does not support https locations nor forcing port number to any desired value.

So, with such setup, clients are about to build url like this "http://localhost/remotefolder/remotedoc" in their dav client, then put this request in an ssl tunnel (stunnel) which will transport this request to the 443 port of a remote server. 
Let's say that the remote server is an Apache one (for example).

A classic configuration of Apache is to setup two ip-based vhosts to access the same locations (it is very simple and even recommended because apache does not support ssl access on name-based vhost):
- one ip-based on 80 port (0.0.0.0:80 is the good way to write this, _default_  and * are not appropriate in that case)
- the other ip-based on 443 port
Some may restrict the 80 vhost to "apache server local network", and forbid access from the whole internet, and keep 443 access with client cert validation for external access.

So what happens when MOVE/RENAME is rejected :
the http client will build a MOVE request specifying :
  - a source document as a relative "file path", relative to "localhost", which is an "happily" AMBIGUOUS name (or address if used as 127.0.0.1)  that will allow BOTH the client to send its resquest in its local tunnel-end AND the server to interpret url as relative to ITSELF (ie the server, not the client) at the other end of the (s)tunnel, so that requested document can be reaaly reached.
I will explain this further but you have to understand that it is really a chance that the name "localhost" can serve two different purposes on two different machines, when it is used an an url exchanged between the two machines, because an url should express a unic "place" -not two! 
I call that the ubiquity of localhost based url, such as "/localhost/src_doc", relative to the "localhost machine" but in fact relevant on the SERVER machine
  - a destination location as an ABSOLUTE url, such as "http://localhost/newpath/newdoc_name", also relevant in fact only on the server machine

Then there are at least two potential sources of pb:
1/ nothing assures the client that the server will be able/allowed to reach/write to such a location "http://someserver" (even if someserver is the "itself" remote server contacted by the client, because http urls are not "the same" as https urls)
2/ even if that location is reachable, the server receives this request on its 443 port and may expect that url be specified as https, and may consider receiving an http url as a bug
(because Apache does not want to proxy by default between two of its vhosts: you have to ask it ...and I think this in NORMAL).
This is where the bug mainly is !

III/ Various comments and questions

1/ the "localhost" ubiquity good bug 
-----------------------------------------
There should be a 3rd problem, but this will fortunately not be the case because of the "ubiquity/ambivalence" of the "locahost" hostname both in the client network space, and on the server network space:
normally, when the client builds an url beginning with "http://localhost", it really wants to address and truly adresses a http request to ITS own host machine: 
by the stunnel magic that request comes to the REMOTE server which, except on particular requests, usually accepts this and serves it : 
that means that the server, when seeing such request,  interprets "localhost" as "ITSELF", and as "itself" is in general allowed to access all its web locations, then the server accepts to serve the request and send back a document in the tunnel back to the client.
You have to understand that this is NOT (so) normal because the "locahost" requested by the client is TRULY not the same host machine as that "understood" by the server.
This is what I call the "localhost" ubiquity good-bug : without this bug no ssl access through stunnel would work (no www, no dav, no http, no https, nothing!).
This property is not at all related to a kind of "proxying" role of stunnel: this is something totally different.
Remember also that stunnel is NOT an http proxy so it transports http requests "as is".

2/ HTTP + SSL is NOT HTTPS
------------------------------------
Because absolute http url requested by the client are directed to an https server: as the client does not know that it is talking to an httpS server, it does not build correct urls.
So it is even a chance that most request works (due to localhost ambivalence and usage of relative path)...

3/ role of stunnel in HTTP over SSL 
-----------------------------------------

In that matter stunnel is not acting as a transparent proxy (this is not related to the "transparent proxy option" of stunnel) : if it were perfectly transparent it would not redirect a request to port 80 to a 443 port, and will not "route" requests for THE localhost to a machine on the internet.
It is also not used as a proxy because the client does not deal with the remote adress but needs to know the adress of the stunnel client machine (localhost in the present case).

In fact stunnel, when used only on one side of a end-to-end communication, is acting as a GATEWAY between two networks : one talking ssl, the other not.
As such it can translate protocols by removing the payload from an ordinary tcp/ip packet on one side and putting it in a new ssl tcp/ip packet on the other side.
We can also say that stunnel acts as a "reverse proxy" hiding a remote machine behind itself...

Well do not think that tricking your stunnel conf file with "accept 443" and "connect 443" will solve your problem: because absolute urls in your requests still begin with http ! and Apache will still not appreciate it (no implicit proxying).
Moreover MS XP clients will not allow you to mix http / 443 in a net use command  (it is possible on Vista but useless for other reasons: on vista you can directly connect to https locations with net use)

Stunnel is not either acting as a ...tunnel ! No it does not, because a pure tunnel will encapsulate requests at one end and remove encapsulation at the other end.

4/ Some workarounds (but the true solution is later)
-----------------------------------------------------------
WORKAROUND 1 : 
if you are impatient  to solve your problem, a good solution is to put stunnel at both ends AND remove SSL support from Apache: 
so put stunnel both on the client and on the server to REALLY build a TRANSPARENT tunnel: I mean transparent to the fact that both ends believes to talk directly "http on 80 port", as they are in fact tunneled through an ssl session.
Also note that this true ssl tunnel will work thanks to the "localhost ambivalence": without that ambivalence, not even a simple get request would work. 
NOTE: please check your apache conf with various directives such as "usecanonicalname" : I put if off in my conf, 

WORKAROUND 2: 
of course something else which works is "true https-able client connecting directly to true https server": this is possible by using the Windows Vista webdav mini-redir directly talking to apache server on 443. Well in that case you will still have ANOTHER MS BUG : "net use" DOES not support BOTH "client cert auth" AND "basic auth by username/password" to access directly a folder on the server (whereas "add network place" does!). If you want the Vista client to work you will be obliged to remove "basic auth restriction" at least at the (dav-)root / location on the server: this is not a good new.
Anyway this will work perfectly if you only use SSL to authenticate the server, not the client, and keep your basic auth settings unchanged.
Well this solution does not involve stunnel so let us continue...


5/ BUT why other requests such as put/get are working?
--------------------------------------------------------------------
Thanks to the localhost ambivalence AND to relative path in http requests other than MOVE and COPY (which are dav requests, and which are the SOLE requests having a destination field, and where this field is implemented as an absolute URL).

6/ NOTE about absolute url in MOVE/COPY destination field
----------------------------------------------------------------------
Some people think that the RFC4918 does not allow "relative url" in the destination field : this is WRONG.
But it is a fact that most clients implement "destination" just as "absolute urls": this is like it is.

Some people may think  "absolute url, well ok, so it implies that moving a doc between two servers or two "services" on the same server should be possible!" :
yes BUT:
1/ the RFC does not say that it is mandatory ! and 
2/ Moreover "miracles" are just not possible if various restrictions disallow this.
3/ Moreover requesting a "write access" on a location that is not in the "name space" of the server currently servicing the request is just not logical, here is why : 
if it were the case, it would mean that any server should act as a proxy to any other : although proxy server exist and are described in the rfc, a http server is not automatically a proxy server; proxy servers are a special family of servers that require EXPLICIT configuration, NOT implicit configuration. So "implicit proxying", although not explicitely forbidden by the RFC, is  a NON SENSE vs the RFC. 
Further it must be noted that the proxying mechanisms described in the RFC2616 are NOT involved when performing HTTP-DAV over SSL ! because this is a particular case of "proxying at tcp/ip level", not http level, no the RFC is not relevant to solve this case: ie a server is not supposed to serve a request in a situation not described in the RFC.


7/ BUT Why "rename is working with "add network place wizard", or some other dav client:
--------------------------------------------------------------------------------------------------------
Because your dav client implements some tricks :
Some clients (such as Windows "Add Network Place" wizard) first try a true MOVE or COPY method,
then they receive a 502 error ("Bad gateway") that I really think is a logical reason for rejecting the request:
the reached host (127.0.0.1:443)  is NOT implicitely a PROXY to 127.0.0.1:80 (as implied by the destination url in the MOVE/COPY request, forged as such by a client believing to act in a http/80 world), so it just CANNOT serve your request.
So when copy/move fail, those clients (as "Add Network Place") try a GET (to local hard drive) and PUT (to remote machine) :  well ok, it seems a good idea but it is NOT because of network speed (often  low in upload) and delays in case of big files.


IV/ SO what is the solution ?

Remember I already stated two workarounds :
1/ true SSL tunneling by using stunnel BOTH on the client AND the server, to join a http client/80 and a http/80 server, and removing of SSL support in Apache: 
to secure your server just use apache or firewall restrictions such as "port 80 only reachable from lan or loopback or localhost!"). 
If 80 is restricted to localhost, the stunnel server must be installed on the http server, of course...

2/ On Vista, by directly using "net use https://", forgetting stunnel, but provided that you disable "basic auth" at least at the root of the mounted remote location.

But these are just workarounds.

**********

A solution based on stunnel client and apache classic conf directives exists:

Well you have understood that the problem comes from the fact that Apache DOES NOT want to act AS A PROXY from "127.0.0.1:443 service" to "127.0.0.1:80" service.
And I really think it is normal.

So you may have 2 solutions :
1/ by doing some "rewrite" magic in your 443 vhost:
1.1/ either rewrite the destination field as A RELATIVE url
1.2/ either rewrite the destination field as beginning with https://localhost
BUT ALL THIS WILL NOT be possible using standard apache directives because apache rewrite engine does not give the ability to rewrite this field. Should it be able to do that : not sure...and I do not request the apache team to do that.

2/ the TRUE solution is here :
So you cannot rewrite the request, hmm, BUT you COULD REDIRECT IT to a service that COULD understand it as it is (because in fact the request can be considered as well-formed when received by an http server, although not by an httpS server), by using one of the apache redirection and/or proxying features.
Well, in that particular case, redirect is NOT the good solution.
The solution is in "PROXYING" !


The PROXYING solution :
When you have finished this paper you will say "all this blah blah" for a few lines of config...yes but it is necessary to really master apache conf, in general and because this conf is also able to open big security holes.

"Forward proxy" is not the solution (you are lucky, I will NOT explain why).
The solution is in "reverse proxying".
See the apache doc about ProxyPass directive, as this is the basis for setting up reverse proxy...

Anyway, If you roughly configure "ProxyPass" directives, it will work for MOVE/COPY with stunnel and "net use" but will make FAIL MANY OTHER things, and in particular for "true https clients talking directly to the 443 server": because  if you have a WELL formed destination url beginning with HTTPS, a raw proxying will send this to the "80 service" which  will REJECT IT !

SO the solution is to only (reverse) proxy requests that really need to be rev-proxied !
For that the only solution is to use the " rewritecond...rewriterule ...[P] flag" magics,
to ONLY rev-proxy <<MOVE and COPY methods arriving on 443 vhost AND having a destination field beginning with "http://">>, which defines perfectly http requests sent via a client SSL stunnel gateway.

Then here is the code to add this slective reverse proxying in the apache configuration : 

# ================================================================
#                       IP BASED VIRTUAL HOSTS 
# ================================================================

<VirtualHost 0.0.0.0:80>
# ServerName (not used, this is a reminder related to usecanonicalname-on; we prefer to use usecanonicalname Off!)
ServerName http://LAN-DNS-NAME:80
DocumentRoot "/var/www/html"
</VirtualHost>


<VirtualHost 0.0.0.0:443>

# ServerName useless but it is a reminder (see recommendation in ServerName apache doc however fort https vhost)
ServerName https://WAN-DNS-NAME:443
DocumentRoot "/var/www/html"

# SSL configuration: PUT WHAT YOU WANT HERE provided that you know what you are doing !
# -----------------

SSLEngine on
SSLOptions +StrictRequire

#   SSL Cipher Suite:
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

#   SSL Protocol support:
SSLProtocol all -SSLv2

# I only accept users from my CA 
SSLCACertificateFile /etc/pki/tls/cacert.pem

# CRL check
SSLCARevocationFile /etc/pki/tls/cacrl.pem

# Server Certificate Chain File
SSLCertificateChainFile /etc/pki/tls/certs/localhost-chain.crt

#   Server Certificate:
SSLCertificateFile /etc/pki/tls/certs/localhost.crt

#   Server Private Key:
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

# VERY IMPORTANT 

SSLVerifyClient require

SSLVerifyDepth 1

<Location />
SSLRequireSSL
</Location>

# REVERSE PROXYING : solution to "WEBDAV over STUNNEL and RENAME issue", the solution starts here
# ----------------------------------------------------------------------------------------------------------------------------

RewriteEngine On

# Some env vars useful later

RewriteRule "(.*)" - [env=V443_URL:$1]

RewriteCond %{SSL:SSL_CLIENT_S_DN} "^(.*)$" [NC]
RewriteRule "(.*)" - [env=MY_SSL_CLIENT:%1]

# REVERSE PROXYING : solution to "WEBDAV over STUNNEL and RENAME issue"
# -----------------------------------------------------------------------------------------
# This is also the solution to problem in renaming files on a subversion server accessed through dav.

RewriteCond %{REQUEST_METHOD} (MOVE|COPY) [NC]
RewriteCond %{HTTP:Destination} "^(http://.*)$" [NC]
RewriteRule ^/(.*)$ http://127.0.0.1:80/$1 [proxy]
# Note : [proxy] can be abreviated as [P]
# RAF : b flag not supported ? (cf unescape)

# SOME security on the proxy: DO NOT HESITATE TO HARDEN THIS more  than I did
<Proxy http://127.0.0.1:80/>
Order allow,deny
Allow from env=MY_SSL_CLIENT
</Proxy>

</VirtualHost>

# Some useful logs
CustomLog logs/mylogs.log "SSL_DN is  %{MY_SSL_CLIENT}e"
CustomLog logs/mylogs.log "V443URL is  %{V443_URL}e"

V/ Miscellaneous comments
--------------------------------------

1/ Some people have the same kind of problem:
- when using dav access to subversion location,
- with various dav clients (cadaver, goliath)
- when dealing with error 502 and MOVE/COPY methods
All this problems have the same cause and solution described above : reverse proxying config on your https server.

2/ a little note about www access (not dav):
Be careful to only have relative url in your various "href" properties in your links, and "usecanonicalname off" on apache, in order to have both http and https access working on the same directories.

3/ General note
Think also to have SSLRequireSSL directive in apache conf: do not think to solve the "move/rename/copy" problem with dav over stunnel by removing this important directive:
 you will only open security holes.

4/ Some people have proposed some similar solutions either by rewriting or by proxying, but these solutions usually propose to write some kind of Apache module or filters, and in fact circumvent apache security by activating a kind of "implicit proxying" which is really a bad idea.

5/ use apache logs (and particularly logs of envvars) and wireshark (ex-ethereal) to diagnose your dav-client behavior: but remember that you can only sniff clear traffic SO it in only possible on the client BUT unfortunately it is NOT possible to sniff the loopback interface on a XP/Vista machine : the trick is to sniff dav traffic on a clear local lan without SSL support, to understand how your client manage various http methods, then to extrapolate when this traffic is transferred through ssl.
In pure ssl traffic, apache logs can be enriched to track the "Destination field", with  an envvar extracted by the headers through some rewriterule.


VI/ Some useful links
------------------------.
Good discussion about dav+https (follow the 2 links at the bottom of the page: one to simple-groupware CMS, the other to "smallvoid" nett use guide)
http://devwfb.blogspot.com/2008/07/how-to-map-https-only-web-folder-to.html
http://www.simple-groupware.de/cms/WebDAV
http://smallvoid.com/article/winnt-webdav-network-drive.html


DAV and SUBVERSION:
http://www.phwinfo.com/forum/alt-apache-configuration/351885-webdav-over-ssl-move-copy.html
http://sethd.org/Site/SubversionProxyFix.html

Good info about webdav implementation by MS:
http://greenbytes.de/tech/webdav/webdav-redirector-list.html

an howto to setup dav+stunnel
http://www.autohotkey.com/forum/topic44471.html

some other howto:
http://librenix.com/?page=SSL
http://www.howtoforge.com/webdav_with_ssl_and_two_factor_authentication

Apache "bug" that I recommend NOT to fix :
https://issues.apache.org/bugzilla/show_bug.cgi?id=45449

==============

Yours sincerely,
Pierre Delaage
Comment 7 Wim Lewis 2013-03-19 22:41:44 UTC
(FWIW, this bug looks related to Bug 38182 )
Comment 8 William A. Rowe Jr. 2018-11-07 21:07:58 UTC
Please help us to refine our list of open and current defects; this is a mass update of old and inactive Bugzilla reports which reflect user error, already resolved defects, and still-existing defects in httpd.

As repeatedly announced, the Apache HTTP Server Project has discontinued all development and patch review of the 2.2.x series of releases. The final release 2.2.34 was published in July 2017, and no further evaluation of bug reports or security risks will be considered or published for 2.2.x releases. All reports older than 2.4.x have been updated to status RESOLVED/LATER; no further action is expected unless the report still applies to a current version of httpd.

If your report represented a question or confusion about how to use an httpd feature, an unexpected server behavior, problems building or installing httpd, or working with an external component (a third party module, browser etc.) we ask you to start by bringing your question to the User Support and Discussion mailing list, see [https://httpd.apache.org/lists.html#http-users] for details. Include a link to this Bugzilla report for completeness with your question.

If your report was clearly a defect in httpd or a feature request, we ask that you retest using a modern httpd release (2.4.33 or later) released in the past year. If it can be reproduced, please reopen this bug and change the Version field above to the httpd version you have reconfirmed with.

Your help in identifying defects or enhancements still applicable to the current httpd server software release is greatly appreciated.