Bug 21523

Summary: mod_deflate compresses content wrongly
Product: Apache httpd-2 Reporter: John Huong <jahuong>
Component: mod_deflateAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: CLOSED FIXED    
Severity: normal    
Priority: P3    
Version: 2.0.46   
Target Milestone: ---   
Hardware: Other   
OS: All   
Attachments: Loop test log file
accept-encoding parsing fix

Description John Huong 2003-07-11 18:32:53 UTC
Based on

mod_deflate shouldn't do anything when let's say my
PHP script outputs with the header;

Content-Encoding: deflate

Unfortunately, I've noticed that it still does it

Here's part of the transmission that I've managed to
Begin client request:
POST http://cimserv.1stsi.com/fans/dsasoapserver.php
Accept: text/xml
Accept: multipart/*
Accept-Encoding: deflate
User-Agent: SOAP::Lite/Perl/0.55
Content-Encoding: deflate
Content-Length: 327
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://cimserv#check"

<snip out contents of requests>

Begin server response:
HTTP/1.1 200 OK
Connection: Close
Date: Fri, 11 Jul 2003 16:29:13 GMT
Server: NuSOAP Server v0.6.4
Vary: Accept-Encoding
Content-Encoding: deflate, gzip
Content-Length: 267
Content-Type: text/xml; charset=UTF-8
Client-Date: Fri, 11 Jul 2003 16:31:04 GMT
Status: 200 OK
X-Powered-By: PHP/4.3.2

<snip out the contents of response>

One thing is for sure, I definitely didn't add ",
gzip" to the server's response. Strange thing
though... it happens on and off.
Comment 1 André Malo 2003-07-11 18:44:18 UTC
First: please try 2.0.47, which contains some further fixes.
Second: I can't believe, that mod_deflate compresses gzip if Accept-Encoding:
contains no gzip. I'd guess the compression takes place somewhere else.
Comment 2 John Huong 2003-07-12 03:01:31 UTC
Hi, sorry for the cross post. Anyway I'm sure it is a mod_deflate problem as 
everything works fine for the last 5 hours after I try the work around as 
describe in bug 9222. My problem would normally appear every now and then 
within an hour or two. I will try Apache 2.0.47 later on and see if I get the 
same problem.
Comment 3 John Huong 2003-07-12 10:50:41 UTC
Ok I've upgraded to 2.0.47 this morning. I've done the tests with 5 SOAP::Lite 
clients sending 500 various types of requests(both valid and errorneous) each 
and without any special setenvif settings, and none of the server responses are 
gzipped. Looks great. Just curious, how come the 2.0.47 release didn't mention 
anything about fixing mod_deflate.. or was this problem attributed to other 
Apache components? Thanks.
Comment 4 André Malo 2003-07-12 12:00:18 UTC
Oops... In fact,the changes were made in 2.0.46. Strange. This points to other
sources of error as well, IMHO.
I'll close the report for now. Don't hesitate to reopen it, if there are further

Thanks for using Apache!
Comment 5 John Huong 2003-07-24 15:33:27 UTC
Created attachment 7491 [details]
Loop test log file
Comment 6 John Huong 2003-07-24 15:34:25 UTC
Looks like I spoke too soon.
I'm attaching the complete log file of my most recent loop test.
Comment 7 André Malo 2003-07-24 16:01:46 UTC
Hmm. The point is, mod_deflate only gets active, if the filter (DEFLATE) is
added *and* Accept-Encoding contains the gzip token.
Did you add the filter somehow? What's the particular configuration?
Comment 8 John Huong 2003-07-24 22:56:45 UTC
Here are my settings for the filters.

<Directory />
    Options FollowSymLinks
    AllowOverride None
Order Deny,Allow 
Deny from all 
AddOutputFilterByType DEFLATE text/html text/plain text/xml
AddOutputFilterByType DEFLATE application/ms* application/vnd* 

DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" In:%
{instream}n Out:%{outstream}n:%{ratio}npct" comdef
Comment 9 John Huong 2003-07-24 22:58:38 UTC
Oh by the way.. the workaround in bug 9222 doesn't work either... the problem 
still happens intermittently.
Comment 10 André Malo 2003-07-24 23:09:47 UTC
Well, and it does actually log instream/outstream/ratio for these requests?
Comment 11 John Huong 2003-07-25 00:53:57 UTC
I'll attach the access logs once I return from work.
Comment 12 John Huong 2003-07-25 10:31:07 UTC
Here is the line in the access log that matches the last entry in the attached 
log file. - - [23/Jul/2003:08:07:50 +0800] "POST /testsoap.php HTTP/1.1" 200 
268 "-" "SOAP::Lite/Perl/0.55" In:245 Out:250:102pct
Comment 13 Joe Orton 2003-07-25 12:36:51 UTC
Just to separate the two separate issues here:

1) if you think that mod_deflate is gzip-encoding a response where the request
did not have "gzip" in an Accept-Encoding header, can you attach a network trace
against 2.0.47, or a log which includes the input and output headers (e.g a
CustomLog with "%{accept-encoding}i {content-encoding}o"

2) mod_deflate will indeed gzip-encode a response which already has
Content-Encoding: deflate (given Accept-Encoding: gzip in the request, etc);
this is a different issue.
Comment 14 John Huong 2003-07-25 15:39:47 UTC
Ok I just tried what was recommended.. and I hit it again. Here is the output 
of the log. - - [25/Jul/2003:23:35:52 +0800] "POST /testsoap.php HTTP/1.1" 200 
268 "-" "SOAP::Lite/Perl/0.55" In:245 Out:250:102pct "deflate" "deflate, gzip"

Here's the logformat from my httpd.conf

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" In:%
{instream}n Out:%{outstream}n:%{ratio}npct \"%{Accept-Encoding}i\" \"%{Content-
Encoding}o\"" comdef
Comment 15 Joe Orton 2003-07-25 16:43:59 UTC
Created attachment 7518 [details]
accept-encoding parsing fix
Comment 16 Joe Orton 2003-07-25 16:44:27 UTC
I may be on a wild goose chase, but can you try that patch?
Comment 17 John Huong 2003-07-25 16:53:59 UTC
Sorry, unfortunately I don't have any compilers with me right now on my Windows 
system and at home. Is there something like a nightly build I could run my 
tests against?
Comment 18 John Huong 2003-08-19 02:30:28 UTC
Any chances of this being fixed in 2.0.48?
Comment 19 André Malo 2003-08-19 12:34:06 UTC
Ha, Joe, you've found the problem! We skip the \0 delimiter and search somewhere
in the memory... Therefore it occurs *sometimes*.
I've finally committed a fix to 2.1 and proposed it for backport.

Comment 20 Jeff Trawick 2003-11-21 22:25:48 UTC
already fixed in 2.0.48 :)