Summary: | deflate_in_filter fails to inflate if CRC/length bytes are not in available stream | ||
---|---|---|---|
Product: | Apache httpd-2 | Reporter: | Laurence 'GreenReaper' Parry <greenreaper> |
Component: | mod_deflate | Assignee: | Apache HTTPD Bugs Mailing List <bugs> |
Status: | RESOLVED FIXED | ||
Severity: | normal | CC: | ksacry, lmeyer, somashekhar.sonnagi, t_guerin |
Priority: | P2 | Keywords: | FixedInTrunk |
Version: | 2.5-HEAD | ||
Target Milestone: | --- | ||
Hardware: | All | ||
OS: | All | ||
Attachments: |
Patch for bug 46146 where mod_deflate does not wait for all the validation data before trying to validate the input
Patch is for 2.2.24
new patch with spelling fixed Patch proposal files to reproduce the problem described in comment #12 file to reproduce the problem described in comment #12 (gzip) Delay filter removal after we provide (not read) the EOS |
Description
Laurence 'GreenReaper' Parry
2008-11-04 13:48:46 UTC
Created attachment 30285 [details] Patch for bug 46146 where mod_deflate does not wait for all the validation data before trying to validate the input Patch is for 2.2.24 This is a patch for 2.2.4 Just uploaded a patch that fixes this for 2.2.4. Mostly borowed from inflate_out_filter Sorry, meant to say 2.2.24, not 2.2.4 The following error message in the attached patch has a slight spelling error: "Zlib: didn't recieve valid gzip magic bytes" 'recieve' should be 'receive'. Created attachment 30314 [details]
new patch with spelling fixed
Thanks to Mike Rumph for pointing out the spelling error
Created attachment 31259 [details]
Patch proposal
Patch proposal tested on Apache 2.4.4
Move the CRC/Length check only after the 8 validation bytes are received.
Use the internal validation buffer for storage.
Test scenario.
I had two specific file upload that were causing the issue.
Transfer in chunked encoding and 6 bytes where in the last chunk.
The problem was systematic and is now solved.
I cannot attach the problematic files (customer data) and a custom HTTPClient is required to do the deflate/chunking.
*** Bug 52492 has been marked as a duplicate of this bug. *** There are multiples patches in Bug 55666 which address this too... *** Bug 55666 has been marked as a duplicate of this bug. *** Thanks for the patches, commited in r1572668 with a similar approach to Christian's one, and with related commits onto the other input/output filters. Commits are from the pacthes staged in Bug 55666. Hi, thanks a lot for the fix, I'm PUTting gzipped files and the upload would not work on some files (on 2.2.x and 2.4.x). So in its current state, sending gzipped content is indeed unusable; I really apreciate that work is being done in this area. However, the patches introduced here unfortunately result in a situation that is in fact worse with some files (I have attached one): - before the patches * with version 2.2.x, I get an error 500 and a message "Could not get next bucket brigade [500, #0]" in the error log. There is no file on the server. BUT if I retry several times (between 1 and 9 times, this varies), the file gets correctly written. * with version 2.4.x, I get an error 500 and a warning "AH01396: Verification data not available (bug?)", then an error "An error occurred while reading the request body (URI: /uploads/curl.xml) [500, #0]". There is no file on the server. BUT if I retry several times (more than 20 times, again this varies), the file gets written, but is truncated (its size varies) * with trunk, (before this patch is applied), I get an error 400 and a warning "AH01396: Verification data not available (bug?)", then an error "An error occurred while reading the request body (URI: /uploads/curl.xml) [400, #0]". There is no file on the server. I haven't tried multiple times. - after the patches, there is no message in the error log and a code 201 is returned BUT the written file is truncated. Its size is always the same. I have tested version 2.2.27 and 2.4.9 both on Windows & Debian, but the trunk has only been tested on Debian. Thz gzip stream is created via a Java program, but the error can be reproduced with the command 'gzip -n myFile.xml' To reproduce the problem: curl -H "Content-Encoding: gzip" --upload-file raw.xml http://localhost/uploads/ (with raw.xml being the result of the gzip command) I will attach both the xml file (curl.xml) and its gzipped version (gzip.xml). Almost any small alteration to the xml file (like adding/removing a space) solves the problem. I have added the comment to this bug, because its fix results (in my opinion) in things being unfotunately worse, but if you think that this belongs in a separate bug, I'll create one. Hope this helps. Created attachment 31526 [details] files to reproduce the problem described in comment #12 Created attachment 31527 [details] file to reproduce the problem described in comment #12 (gzip) Thanks for reporting, could you include your mod_deflate configuration? Sorry, forgot to mention that I used WebDav: DavLockDB "/usr/local/apache2/var/DavLock" Alias /uploads "/usr/local/apache2/uploads" <Directory "/usr/local/apache2/uploads"> Dav On Order Allow,Deny Allow from all Options Indexes FollowSymLinks SetInputFilter DEFLATE </Directory> (with of course mod_deflate, mod_dav and mod_dav_fs enabled) Created attachment 31528 [details]
Delay filter removal after we provide (not read) the EOS
Could you please try the attached patch?
I can't reproduce anymore with it applied.
Thanks again Thierry for the test case.
bug fixed, fantastic! Thanks a lot for your (lightning-fast) help. Do you think that this will be backported to the 2.2.x branch? (I suppose it will be backported to the 2.4.x (correct me if I'm wrong)) (In reply to Thierry Guérin from comment #18) > Do you think that this will be backported to the 2.2.x branch? (I suppose it > will be backported to the 2.4.x (correct me if I'm wrong)) The whole bundle for this and Bug 55666 is already proposed for backport in 2.4, waiting for more votes. Once/if accepted, I will propose a 2.2 backport too. Backported to upcoming 2.4.10. |