I can't post files to an ssl site that are larger than 128k (120k works, 140k doesn't work. The error message I get in ssl_error_log is request body exceeds maximum size for SSL buffer could not buffer message body to allow SSL renegotiation to proceed and the client gets 413. This doesn't occur every time; apparently only when ssl renegotiation is needed.
*** This bug has been marked as a duplicate of 12355 ***
The fix to bug 12355 specificially lead to this bug. This is not a duplicate.
Ok, technically you are right and your report is not exactly a duplicate of 12355, but it is currently not planned to change this behaviour in the case that you have POST requests + SSL + Directory or Location based client certificates which require a SSL renegotiation. It would require to introduce disk buffering of the POST request body. If you are using client certificates for the whole virtual host everything works fine. So I mark it a WONTFIX.
There may be good functional reasons for POSTs larger than 128k and to require client certificates only for access to certain URLs. And asking for an optional client certificate at SSL connect bothers users of other URLs with unnecessary prompts for client certificates that they may not have and don't need (depending on the browser that they use). For us the hard-coded limit is still a problem. > It would require to introduce disk buffering of the POST request > body. This is not clear to me. Where is the hard-coded limit of 128k coming from? Could the code not look at the value of the directive LimitRequestBody and if it is set allow SSL request bodies of that size?
I really need at least 200K limit. If I understand correctly, I can 'patch' the code and increase this buffer from 128K to say 256K, recompile apache and it will work, right?
(In reply to comment #5) > I really need at least 200K limit. If I understand correctly, I can 'patch' > the code and increase this buffer from 128K to say 256K, recompile apache and > it will work, right? Correct. You can do this.
Well, the default could be bumped to 256K, that wouldn't be unreasonable. But you should really design your site to ensure that the first request to a client-cert-protected area is not a POST request with a large body; make it a GET or something. Any request body has to be buffered into RAM to handle this case, so represents an opportunity to DoS the server. To bump the limit you can build like: ./configure CPPFLAGS=-DSSL_MAX_IO_BUFFER=256000 Anybody for whom 128K is too small but 256K would be sufficient, please add a comment here, to gauge interest in making that change.
While I do believe the previous unlimited approach could be a DoS. Nice catch. I have to believe the "one size limit fits all" approach will not work for all the existing applications in the world. However, shouldn't we add a SSLMaxIOBuffer directive instead of hardcoding the value at build time? This way any pre-built server or existing applications have a way to raise/lower this value as needed for any given virtual host or directory?
> But you should really design your site to ensure that the first > request to a client-cert-protected area is not a POST request with a > large body; make it a GET or something. Not really an option with SOAP. > I have to believe the "one size limit fits all" approach will not > work for all the existing applications in the world. Agreed. > However, shouldn't we add a SSLMaxIOBuffer directive instead of > hardcoding the value at build time? That is a good way to remove the hard-coded limit. But is there a reason why one could not use the existing directive LimitRequestBody, as suggested above? It can be set for the client cert protected area and then defines the size of requests that should be handled, thus the amount that should be allowed to be buffered.
I'm fairly reluctant to add a config directive for this. I would be happy with a "one size fits most" hard-coded limit if we could arrive at such a value; what is your input on changing the limit to 256K? Would that be sufficient or not? Overloading LimitRequestBody for such a purpose is not acceptable, no - the default is unlimited.
> I'm fairly reluctant to add a config directive for this. I can understand your point of view. > I would be happy with a "one size fits most" hard-coded limit if we could arrive at such a value; what is your input on changing the limit to 256K? IMHO Since I work in support/engineering I can honestly say we have customers that are using soap messages anywhere from 1k to 40mb in size (or higher). So, if you are asking for my input regarding any hard coded value I would have to vote for the 40mb-50mb range. While I agree those values seem absurd for most small web site, but large SOAP web sites will need this type of limit out of the box. Without a large hard-coded value customers are going to request vendors like RedHat to give them a supported version of the Apache web Server with a higher hard coded value because their existing applications that have been deployed all over the world just stopped working when they installed http://www.linuxsecurity.com/content/view/120313. (I’ve already had three customers)
> what is your input on changing the limit to 256K? Would that be > sufficient or not? No. We're looking at megabyte SOAP POSTs. > Overloading LimitRequestBody for such a purpose is not acceptable, > no - the default is unlimited. With that overload idea, the default value of zero (unlimited) would be translated to the hard-coded value to protect against DOS attempts. Defining a positive size for LimitRequestBody would allow that size to be buffered for POSTs in mod_ssl (because it seems sensible to keep functioning up to the specified limit). I had something along these lines in mind: --- httpd-2.0.46/modules/ssl/ssl_engine_io.c.old ... +++ httpd-2.0.46/modules/ssl/ssl_engine_io.c.new ... @@ -1395,8 +1395,17 @@ struct modssl_buffer_ctx *ctx; apr_bucket_brigade *tempb; apr_off_t total = 0; /* total length buffered */ + apr_off_t max_ssl_buffered = 0; /* Maximum allowed memory buffering of ssl data. */ int eos = 0; /* non-zero once EOS is seen */ + max_ssl_buffered = ap_get_limit_req_body( r ); + + if (max_ssl_buffered == 0) { + /* If undefined/unlimited, use default limit to defend against + * DOS attempts. */ + max_ssl_buffered = SSL_MAX_IO_BUFFER; + } + /* Create the context which will be passed to the input filter. */ ctx = apr_palloc(r->pool, sizeof *ctx); ctx->bb = apr_brigade_create(r->pool, c->bucket_alloc); @@ -1460,7 +1469,7 @@ total, eos); /* Fail if this exceeds the maximum buffer size. */ - if (total > SSL_MAX_IO_BUFFER) { + if (total > max_ssl_buffered) { ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r, "request body exceeds maximum size for SSL buffer"); return HTTP_REQUEST_ENTITY_TOO_LARGE;
Here's an afterthought to the above patch to allow LimitRequestBody to control the size of the SSL buffer. When doing this, it may be a good idea to refer to the controlling directive in the error message and change "request body exceeds maximum size for SSL buffer" into, for instance, "request body exceeds maximum size for SSL buffer; try LimitRequestBody > 0"
We can allow to to configure this to be larger at a serious cost to how many requests you can process. The obvious answer for an 'upload' style operation is to ensure they never hit your upload page without going through a simpler front page which first enforces the renegotation. This can be your upload form page. Once the session is SSLClientVerify'ed it won't renegotate -again- so this problem won't occur. No matter what -we- do, if you design your huge-post page such that it won't cause renegotiation on large posts, your server will always have less stress. And that's a good thing IMHO. 2GB set asides are absurd, but pushing up a 2GB iso image isn't inconcievable.
While 256K suits our needs for now (I did recompile and it worked), tomorrow we'll have to post larger scanned documents (say 512K), some time later even larger. My apache is just a reverse proxy with SSL client authentication, so an option would be better or I'll have to recompile every change/update...
Will Rowe wrote: > The obvious answer for an 'upload' style operation is to ensure they > never hit your upload page without going through a simpler front > page which first enforces the renegotation. This can be your upload > form page. > Once the session is SSLClientVerify'ed it won't renegotate -again- > so this problem won't occur. This can work for interactive applications, but there are common situations without upload page: an application that wants to submit data to the web server in a SOAP POST request. Note: the above proposal for using an upload page request to renegotiate for the client certificate appears to work only with "SSLVerifyClient none" but not with "SSLVerifyClient optional" at top level. In the last case a renegotiation is performed on the subsequent form POST even when a client certificate is already present. Thus you again run into the 128K limit. This is probably explained by the following code in ssl_engine_kernel.c, which only treats "none" as a special case: /* optimization */ if ((dc->nOptions & SSL_OPT_OPTRENEGOTIATE) && (verify_old == SSL_VERIFY_NONE) && ((peercert = SSL_get_peer_certificate(ssl)) != NULL)) { renegotiate_quick = TRUE; X509_free(peercert); }
Hi, I am too am facing problem due to the fixed buffer size. I saw the suggestion about adding a directive to modify the buffer size as needed. Has there been any work done on this regard? Any other suggestion about how this problem could be fixed?
We are currently in the process of getting this 'fixed' via Red Hat commercial support (which we have). The fix of Peter Wagemans will probably be extended a little and hopefully checked in.
I'm hitting this limit too, using apache 2.2.3, built from a standard Gentoo-based ebuild. I'm using Apache here in this context as a front-end to Zope/Plone, and plone offers the user the option of uploading content. This content has no inherent plone-based size limits. So in my case, if I use SSL to secure my sites (which I do), and I use Apache as a front-end, as described in several places in plone documentation, two of which are here: http://plone.org/documentation/tutorial/plone-apache http://plone.org/documentation/how-to/apache-ssl ...and I upload large files, then I get nailed by this limit. Has any further work been done with this in 2.2? What info is still needed to resolve this bug?
Sorry. I should have added above that there are no client certificates involved in these uploads. I'm not savvy enough about the internals of either apache or plone to know, but I suppose that means it's possible that what I'm seeing is not actually this bug, but the behavior of my systems match the symptoms in every way except for the involvement of client certificates, so to me that means that if they are not the same, then they are at least, very probably strongly associated with each other. When I upload files 128kb and smaller, it works as expected. When I attempt to upload files 129kb and larger, I get this: Error message in browser: Title: 413 Request Entity Too Large Page: Request Entity Too Large The requested resource /Members/admin/portal_factory/Image/image.2007-03-03.9545920618/atct_edit does not allow request data with POST requests, or the amount of data provided in the request exceeds the capacity limit. Error message in logs: [Sat Mar 03 19:26:35 2007] [error] [client xxx.yyy.zzz.ttt] request body exceeds maximum size for SSL buffer, referer: https://www.example.com/Members/admin/portal_factory/Image/image.2007-03-03.9545920618/edit [Sat Mar 03 19:26:35 2007] [error] [client xxx.yyy.zzz.ttt] could not buffer message body to allow SSL renegotiation to proceed, referer: https://www.example.com/Members/admin/portal_factory/Image/image.2007-03-03.9545920618/edit I've spoken with someone on the plone list who's using RHEL and apache/ssl/plone in the same manner that I am, and he reports not suffering from this problem. I'm not sure if he has any upper limit at all, or if the upper limit is simply larger than 128kb. I'm still talking with him. I guess redhat has applied some sort of patch. Does anyone know about that? Is it the same one mentioned in this bug report? I'd like to have the limit (if it must exist) be up in the 40-50 MB range myself. If there's another patch, perhaps someone could refer me to it?
[somewhat sheepishly]: After all the discussion, and rereading documentation and config files and the bug report several times over, I noticed that my apache server config file used the SSLVerifyClient Directive at level "optional" and that the documentation states, "In practice only levels 'none' and 'require' are really interesting, because level 'optional' doesn't work with all browsers". I was also using the SSLVerifyDepth Directive at a depth number of 1. By commenting out these two directives, I solved the problem. When I remarked earlier that client certificates were not involved at all, I mistakenly considered only what was going on with the client, failing to consider client certificate directives on the server. Apologies if I should have thought of that sooner, and if I generated a lot of commotion over nothing.
I am running httpd 2.2.3 on CentOS 5. This problem also affects SugarCRM attachment uploads. The login page for SugarCRM uses a GET request, so the renegotiation should be fine, but users report that sometimes the attachment upload still fails with this error. Perhaps the client-certificate SSL session times out or something, which forces httpd to renegotiate again? If so, this is yet another use case that supports adding a configurable per-location buffer directive. I can find no work-around for this other than setting "SSLVerifyClient require" at the virtual host level. However, we have good reasons to *not* set "SSLVerifyClient require" at the virtual host level, since many of our SSL services do not require client certs. As stated in the docs, "SSLVerifyClient optional" doesn't work for all clients (e.g. WebDAV on win2k for one).
(In reply to comment #22) > Perhaps the client-certificate SSL session times out or something, which > forces httpd to renegotiate again? If so, this is yet another use case that > supports adding a configurable per-location buffer directive. I confirmed that the SSLSessionCacheTimeout affects renegotiation. Therefore, at least for interactive applications where the upload form uses a GET request, I believe this issue can be worked around by setting SSLSessionCacheTimeout to a value at least as large as the application session timeout. The default of 300 on CentOS 5 was easily exceeded by a user who is uploading an attachment, while also filling in associated description and other form fields before clicking Submit. > As stated in the docs, "SSLVerifyClient optional" doesn't work for all > clients (e.g. WebDAV on win2k for one). Correction: I'm not sure about WebDAV on win2k working with optional or not -- the test I did earlier was incorrect. However, the point stands since I do not want clients to be prompted for certificates anyway.
Created attachment 21473 [details] httpd-2.2.8-ssl-io-buffer.patch This patch adds SSLMaximumBufferSize tunable - it's global for whole module. Defaults to 0, which means to use the default 128k limit.
(In reply to comment #24) > Created an attachment (id=21473) [details] > httpd-2.2.8-ssl-io-buffer.patch > > This patch adds SSLMaximumBufferSize tunable - it's global for whole module. > Defaults to 0, which means to use the default 128k limit. thanks, works well for me. I used the patch on debian apache2-2.2.3 source. After one week still no problems with the patch. Thanks for your work
I'm wondering about the reasons for a patch with a configurable global limit (comment #24) instead of the patch described in comments #12 and #13 which uses the existing LimitRequestBody directive to give control over the SSL maximum buffer size at the level of server config, virtual host, directory or .htaccess. Is there perhaps a reason for keeping the SSL maximum buffer size smaller than a configured maximal request size?
I hope I am not overstepping my bounds, but I'm trying to increase the severity of this issue and bring some attention back to it. A virtualhost that mixes browser-based content and REST/SOAP services is not uncommon. This hardcoded buffer makes the REST/SOAP activities likely to fail. I am not sure why Joe marked this as needs info - what info is needed?
I've added a SSLRenegBufferSize directive in r726109 to make this buffer size configurable. Thanks for all the feedback.
Thanks.... finally an official fix. Still I'm curious about why an additional directive... see also comment #26
(In reply to comment #29) > Thanks.... finally an official fix. Still I'm curious about why an additional > directive... see also comment #26 Overloading the LimitRequestBody semantics would not be appropriate because it could make existing configurations do something completely surprising.
Unfortunately I found this report after posting https://issues.apache.org/bugzilla/show_bug.cgi?id=46508 Joe's patch in https://issues.apache.org/bugzilla/show_bug.cgi?id=39243#c28 >almost< works. See https://issues.apache.org/bugzilla/show_bug.cgi?id=46508 for the issue, a fix, and a backport to 2.2. Hopefully he'll make it official. Hope this helps anyone else who tries to use or backport as-is. (I didn't re-open this report since 46508 has all the detail and is open.)
Proposed for backport as r733472.
Thanks very much for the official fix. One remark on comment #30 from Joe Orton. > Overloading the LimitRequestBody semantics would not be appropriate > because it could make existing configurations do something > completely surprising. A separate directive is the best solution, but all the old patch in comments #12,#13 does is allow requests up to the configured LimitRequestBody value (if set > 0), also for requests > 128k. For me that's not a "completely surprising" behaviour change.
My point was that e.g. if someone had LimitRequestBody set to 100mb for some PHP script they were using in the same context, and mod_ssl started buffering up to 100mb of request body into RAM per process -- from untrusted users -- that would allow a DoS attack against the server. This would be surprising ;)
Yes, that's the side effect of the memory buffering. Explicitly allowing large POSTS and doing client certificate authentication is probably a rare combination for untrusted users. Anyway, I'm glad it is fixed after using the old workaround patch for 2.5 years. Thanks.
To overcome this limit in version 2.0.x, it looks like the options are either backporting the fix to 2.0, or recompiling the standard 2.0 source with " ./configure CPPFLAGS=-DSSL_MAX_IO_BUFFER=256000". The latter option seems the simplest. If I do this, which files will this modify? Is it just the mod_ssl.so file, or are other files impacted?
Note that I found out that the workaround mentioned in comment #22, to place the SSLVerifyClient directive outside of the Directory section, inside the main Vhost container, *inactivated* the complete client cert verification part, i.e. any client could connect. Moving the statements back into the <Directory> restored security. I'll see if I can isolate it and file another bug.
Note to my last comment, I was stung by this issue: https://issues.apache.org/bugzilla/show_bug.cgi?id=12355 So take care...
See SSLRenegBufferSize for adjusting the buffersize (http://httpd.apache.org/docs/2.2/mod/mod_ssl.html#sslrenegbuffersize)
It seems curl solves this issue by sending the HTTP 1.1 header "Expect" with value "100-continue" when you use a client certificate to perform an HTTP PUT of a huge file. Apparently this allows the SSL renegotiate to occur before the large payload (body) is even transferred. I noticed this due to my Java client failing with the error discussed in this thread, but curl (and libcurl) had no issues. Unfortunately the Sun Java built-in HttpsUrlConnection implementation doesn't support the "Expect" header. Use the Java Apache Foundation HTTP Client library or OpenJDK implementation instead. Hope this helps someone who digs up this thread.