Bug 21523

Summary:	mod_deflate compresses content wrongly
Product:	Apache httpd-2	Reporter:	John Huong <jahuong>
Component:	mod_deflate	Assignee:	Apache HTTPD Bugs Mailing List <bugs>
Status:	CLOSED FIXED
Severity:	normal
Priority:	P3
Version:	2.0.46
Target Milestone:	---
Hardware:	Other
OS:	All
Attachments:	Loop test log file accept-encoding parsing fix

Description John Huong 2003-07-11 18:32:53 UTC

Based on
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=9222

mod_deflate shouldn't do anything when let's say my
PHP script outputs with the header;

Content-Encoding: deflate

Unfortunately, I've noticed that it still does it
occasionally.

Here's part of the transmission that I've managed to
log.
Begin client request:
POST http://cimserv.1stsi.com/fans/dsasoapserver.php
Accept: text/xml
Accept: multipart/*
Accept-Encoding: deflate
User-Agent: SOAP::Lite/Perl/0.55
Content-Encoding: deflate
Content-Length: 327
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://cimserv#check"

<snip out contents of requests>

Begin server response:
HTTP/1.1 200 OK
Connection: Close
Date: Fri, 11 Jul 2003 16:29:13 GMT
Server: NuSOAP Server v0.6.4
Vary: Accept-Encoding
Content-Encoding: deflate, gzip
Content-Length: 267
Content-Type: text/xml; charset=UTF-8
Client-Date: Fri, 11 Jul 2003 16:31:04 GMT
Client-Peer: 172.16.104.14:80
Status: 200 OK
X-Powered-By: PHP/4.3.2

<snip out the contents of response>


One thing is for sure, I definitely didn't add ",
gzip" to the server's response. Strange thing
though... it happens on and off.

Comment 1 André Malo 2003-07-11 18:44:18 UTC

First: please try 2.0.47, which contains some further fixes.
Second: I can't believe, that mod_deflate compresses gzip if Accept-Encoding:
contains no gzip. I'd guess the compression takes place somewhere else.

Comment 2 John Huong 2003-07-12 03:01:31 UTC

Hi, sorry for the cross post. Anyway I'm sure it is a mod_deflate problem as 
everything works fine for the last 5 hours after I try the work around as 
describe in bug 9222. My problem would normally appear every now and then 
within an hour or two. I will try Apache 2.0.47 later on and see if I get the 
same problem.

Comment 3 John Huong 2003-07-12 10:50:41 UTC

Ok I've upgraded to 2.0.47 this morning. I've done the tests with 5 SOAP::Lite 
clients sending 500 various types of requests(both valid and errorneous) each 
and without any special setenvif settings, and none of the server responses are 
gzipped. Looks great. Just curious, how come the 2.0.47 release didn't mention 
anything about fixing mod_deflate.. or was this problem attributed to other 
Apache components? Thanks.

Comment 4 André Malo 2003-07-12 12:00:18 UTC

Oops... In fact,the changes were made in 2.0.46. Strange. This points to other
sources of error as well, IMHO.
I'll close the report for now. Don't hesitate to reopen it, if there are further
issues.

Thanks for using Apache!

Comment 5 John Huong 2003-07-24 15:33:27 UTC

Created attachment 7491 [details]
Loop test log file

Comment 6 John Huong 2003-07-24 15:34:25 UTC

Looks like I spoke too soon.
I'm attaching the complete log file of my most recent loop test.

Comment 7 André Malo 2003-07-24 16:01:46 UTC

Hmm. The point is, mod_deflate only gets active, if the filter (DEFLATE) is
added *and* Accept-Encoding contains the gzip token.
Did you add the filter somehow? What's the particular configuration?

Comment 8 John Huong 2003-07-24 22:56:45 UTC

Here are my settings for the filters.

<Directory />
    Options FollowSymLinks
    AllowOverride None
Order Deny,Allow 
Deny from all 
AddOutputFilterByType DEFLATE text/html text/plain text/xml
AddOutputFilterByType DEFLATE application/ms* application/vnd* 
application/postscript
</Directory>

#
DeflateFilterNote Input instream
DeflateFilterNote Output outstream
DeflateFilterNote Ratio ratio


LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" In:%
{instream}n Out:%{outstream}n:%{ratio}npct" comdef

Comment 9 John Huong 2003-07-24 22:58:38 UTC

Oh by the way.. the workaround in bug 9222 doesn't work either... the problem 
still happens intermittently.

Comment 10 André Malo 2003-07-24 23:09:47 UTC

Well, and it does actually log instream/outstream/ratio for these requests?

Comment 11 John Huong 2003-07-25 00:53:57 UTC

I'll attach the access logs once I return from work.

Comment 12 John Huong 2003-07-25 10:31:07 UTC

Here is the line in the access log that matches the last entry in the attached 
log file.

127.0.0.1 - - [23/Jul/2003:08:07:50 +0800] "POST /testsoap.php HTTP/1.1" 200 
268 "-" "SOAP::Lite/Perl/0.55" In:245 Out:250:102pct

Comment 13 Joe Orton 2003-07-25 12:36:51 UTC

Just to separate the two separate issues here:

1) if you think that mod_deflate is gzip-encoding a response where the request
did not have "gzip" in an Accept-Encoding header, can you attach a network trace
against 2.0.47, or a log which includes the input and output headers (e.g a
CustomLog with "%{accept-encoding}i {content-encoding}o"

2) mod_deflate will indeed gzip-encode a response which already has
Content-Encoding: deflate (given Accept-Encoding: gzip in the request, etc);
this is a different issue.

Comment 14 John Huong 2003-07-25 15:39:47 UTC

Ok I just tried what was recommended.. and I hit it again. Here is the output 
of the log.

127.0.0.1 - - [25/Jul/2003:23:35:52 +0800] "POST /testsoap.php HTTP/1.1" 200 
268 "-" "SOAP::Lite/Perl/0.55" In:245 Out:250:102pct "deflate" "deflate, gzip"

Here's the logformat from my httpd.conf

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" In:%
{instream}n Out:%{outstream}n:%{ratio}npct \"%{Accept-Encoding}i\" \"%{Content-
Encoding}o\"" comdef

Comment 15 Joe Orton 2003-07-25 16:43:59 UTC

Created attachment 7518 [details]
accept-encoding parsing fix

Comment 16 Joe Orton 2003-07-25 16:44:27 UTC

I may be on a wild goose chase, but can you try that patch?

Comment 17 John Huong 2003-07-25 16:53:59 UTC

Sorry, unfortunately I don't have any compilers with me right now on my Windows 
system and at home. Is there something like a nightly build I could run my 
tests against?

Comment 18 John Huong 2003-08-19 02:30:28 UTC

Any chances of this being fixed in 2.0.48?

Comment 19 André Malo 2003-08-19 12:34:06 UTC

Ha, Joe, you've found the problem! We skip the \0 delimiter and search somewhere
in the memory... Therefore it occurs *sometimes*.
I've finally committed a fix to 2.1 and proposed it for backport.

Thanks!

Comment 20 Jeff Trawick 2003-11-21 22:25:48 UTC

already fixed in 2.0.48 :)