Bug 45023 - DEFLATE preventing 304 NOT MODIFIED response
Summary: DEFLATE preventing 304 NOT MODIFIED response
Status: REOPENED
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: mod_deflate (show other bugs)
Version: 2.2.8
Hardware: All All
: P2 regression with 19 votes (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords: RFC
: 56354 (view as bug list)
Depends on: 39727
Blocks:
  Show dependency tree
 
Reported: 2008-05-16 23:53 UTC by Noah
Modified: 2017-03-02 11:20 UTC (History)
10 users (show)



Attachments
A POC slightly better than the above conf lines, (2.68 KB, patch)
2008-08-19 06:41 UTC, rahul
Details | Diff
use ap_hook_post_read_request instead of input filter (2.12 KB, patch)
2008-08-21 08:13 UTC, rahul
Details | Diff
mod_deflate as found in apache2-src package on ubuntu version 2.2.8-1ubuntu0.3 (47.88 KB, text/plain)
2008-09-18 11:10 UTC, Sidharth Kshatriya
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Noah 2008-05-16 23:53:11 UTC
On my server, I recently added the DEFLATE filter for all application/x-javascript files, and noticed that apache has stopped sending 304 NOT MODIFIED on ALL my js files *always*.

Observe this header trace:

-----------------------------------------
GET /main.js HTTP/1.1
Host: xxxxxxxxxxxxx
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14
Accept: */*
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
If-Modified-Since: Mon, 12 May 2008 16:09:00 GMT
If-None-Match: "44d0ac3fd1f00"-gzip
Cache-Control: max-age=0


HTTP/1.x 200 OK
Date: Sat, 17 May 2008 06:09:25 GMT
Server: Apache
Last-Modified: Mon, 12 May 2008 16:09:00 GMT
Etag: "44d0ac3fd1f00"-gzip
Accept-Ranges: bytes
Cache-Control: max-age=86400, must-revalidate, private
Expires: Sun, 18 May 2008 06:09:25 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 9797
Keep-Alive: timeout=2, max=299
Connection: Keep-Alive
Content-Type: application/x-javascript
-----------------------------------------

It seems to me, that a 304 NOT MODIFIED would have been the appropriate response here.

Before I added DEFLATE to javascript files, the above response was always 304 NOT MODIFIED (except the first one).
Now with deflate, it is *always* 200 OK.

Is this by design, or is this a bug?

One thing to mention is - I have both mod_deflate and mod_expires statically compiled into httpd.
Comment 1 Takashi Sato 2008-05-17 00:22:47 UTC
If we remove "-gzip" from Etag for If-None-Match, a server returns 304.
mod_deflate adds "-gzip" and sends it to us, but condition check of If-None-Match runs before mod_deflate ...???
Comment 2 Nick Kew 2008-05-17 06:47:55 UTC
This is actually a fix of Bug 39727.  Yes, we know the core code testing for conditional responses still needs work.
Comment 3 John Siracusa 2008-06-24 07:32:39 UTC
This bug nearly defeats the purpose of mod_deflate.  I turned mod_deflate on with the intent of saving bandwidth when sending JavaScript and CSS files.  Instead, I'm using *more* bandwidth for every request but the first.  (Previously, subsequent requests would result in small, no-content 304 responses.  Now the compressed JS and CSS files are sent every time.)

BTW, the change that appends "-gzip" to the etag (revision 607219, I think) is a bit wacky in its own right in that it appends outside the double quotes.  IOW, if the original header is:

    Etag: "5954c6-10f4-449d11713aac0"

the modified header ends up as:

    Etag: "5954c6-10f4-449d11713aac0"-gzip

when it probably should be:

    Etag: "5954c6-10f4-449d11713aac0-gzip"

with the "-gzip" inside the quotes.  I altered the code to do this, but it had no affect on this 304 bug.  I just wanted to note it here so the situation is avoided when this bug is fixed for real.
Comment 4 rahul 2008-08-19 05:36:44 UTC
There is a workaround, 
If you are serving from a location say /js
You can use a configuration like below to switch ETag: $1-gzip to $1 

<Location /js>
RequestHeader  edit "If-None-Match" "^(.*)-gzip$" "$1"
Header  edit "ETag" "^(.*[^g][^z][^i][^p])$" "$1-gzip"
</Location>

(Perhaps this should be done from mod_deflate as input filter
with the condition that if Client requests with Accept-Encoding: gzip
only then modify If*match headers to their un-gzip values.)

(Also the current value of ETag (+gzip) is not RFC compliant, since RFC mandates
a quoted string as the value of ETag. As the noted by john, -gzip should be inside quotes.)
Comment 5 rahul 2008-08-19 06:41:51 UTC
Created attachment 22453 [details]
A POC slightly better than the above conf lines,

Adds an input filter ETAG that can be used like below, along with DEFLATE
to let apache recognize ($1)-gzip as $1

AddInputFilter ETAG .txt
AddOutputFilterByType DEFLATE text/plain
Comment 6 John Siracusa 2008-08-19 07:47:26 UTC
Both the workaround in comment #4 and the patch in comment #5 appear to work.  Neither are real "fixes" for the bug, however.  (But I'm glad I have something to work with until a real fix arrives, so thanks rahul!)
Comment 7 rahul 2008-08-21 08:13:07 UTC
Created attachment 22468 [details]
use ap_hook_post_read_request instead of input filter

Incorporate suggested changes by Nick
Comment 8 Sidharth Kshatriya 2008-09-18 08:23:02 UTC
Hello!

I'm also facing the same problem -- I had been scratching my head for sometime before I found this bug report. I have to reduce my page size urgently and not having gzip on is not really an option for me.

I'm not really comfortable having a customized compiled version of apache2 on my system and the regular expression "hack" will also slow things down (will it?).

Any alternative ideas? When might this bug be fixed?

Thanks,

Sidharth
Comment 9 Sidharth Kshatriya 2008-09-18 08:27:57 UTC
I had a look at ap_hook_post_read_request attachment...seems like I can get away with just compiling mod_deflate.c

I'm using Apache 2.2.8. Can anyone help me on where I can get the correct source for the corresponding mod_deflate and how I can compile mod_deflate?

Thanks for your help!

Sidharth
Comment 10 Sidharth Kshatriya 2008-09-18 09:19:28 UTC
Hi, 

Sorry for flooding this mailing list.

In this thread there is Rahul's mod_deflate patch and there are some patches in this bug thread:

https://issues.apache.org/bugzilla/show_bug.cgi?id=39727
(mostly by Nick Kew)

Which one is the best solution?

Thanks,

Sidharth

Comment 11 rahul 2008-09-18 10:27:15 UTC
You will need both. This patch expects the other to have been applied (this is already applied on trunk).
Also please use the dev@httpd.apache.org for questions.

Comment 12 Sidharth Kshatriya 2008-09-18 11:03:49 UTC
Thanks for your response Rahul.

Unfortunately I am new to all of this. I applied Nick's patch to my mod_deflate.c but I got patching errors...his patch is assuming a slightly different mod_deflate.c from mine.

Would really appreciate if you were able to direct me to the place (or even better just gave me the patched mod_deflate.c version) that I could just compile.

[ASIDE: I am using apxs2 -c mod_deflate.c

with LDFLAGS="-lz" set in /usr/bin/aprconfig

I notice that my original mod_deflate.so depends on libpthread. When I apply your patch to my file and compile, I don't get dependency with libpthread.]

Please tell me if I would need to do any additional transformations to the file you would provide me (assuming you can :-) ).

Thanks a Ton!

Sidharth
[Attaching the mod_deflate that I have. This I obtained by issuing
sudo apt-get install apache2-src 
on ubuntu. This is version 2.2.8 Apache]
Comment 13 Sidharth Kshatriya 2008-09-18 11:10:32 UTC
Created attachment 22605 [details]
mod_deflate as found in apache2-src package on ubuntu version 2.2.8-1ubuntu0.3
Comment 14 Roy T. Fielding 2009-04-03 18:41:47 UTC
This bug was added in 2.2.8 in a failed attempt to address Bug 39727.
Since a real fix will require extensive changes, I have reverted the
change in 

  http://svn.apache.org/viewvc?view=rev&revision=761835

Note: if you are patching a released version of the source code,
then extract the patch from the original change

  http://svn.apache.org/viewvc?view=rev&revision=608849

and use patch -R to apply it in reverse.

Meanwhile, the original bug will still be tracked as issue 39727.
Comment 15 hackeron 2009-04-13 11:25:53 UTC
I'm still experiencing this with 2.2.11-2ubuntu2 -- but adding these 2 lines are suggested by rahul works:

RequestHeader  edit "If-None-Match" "^(.*)-gzip$" "$1"
Header  edit "ETag" "^(.*[^g][^z][^i][^p])$" "$1-gzip"
Comment 16 Eric Covener 2013-06-23 17:41:44 UTC
Was this ever resolved in trunk/2.4?

Alias /deflate /home/covener/SRC/httpd-trunk/built/htdocs/index.html
<Location /deflate>
SetOutputFilter DEFLATE
</location>


$ wget  --header="Accept-Encoding: gzip" -S http://localhost/deflate -O/dev/null 2>&1 |grep ETag
  ETag: "58c5-4b26c6f28ce80-gzip"

$ wget --header='If-None-Match: "58c5-4b26c6f28ce80-gzip"'  --header="Accept-Encoding: gzip" -S http://localhost/deflate -O/dev/null 2>&1
--2013-06-23 13:39:47--  http://localhost/deflate
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sun, 23 Jun 2013 17:39:47 GMT
  Server: Apache/2.5.0-dev (Unix) OpenSSL/1.0.1c
  Last-Modified: Wed, 23 Nov 2011 20:04:58 GMT
  ETag: "58c5-4b26c6f28ce80-gzip"
  Accept-Ranges: bytes
  Vary: Accept-Encoding
  Content-Encoding: gzip
  Content-Length: 206
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html

covener@cov-w520:~/SRC/httpd-trunk$ wget --header='If-None-Match: "58c5-4b26c6f28ce80"'  --header="Accept-Encoding: gzip" -S http://localhost/deflate -O/dev/null 2>&1|grep HTTP/1.1
  HTTP/1.1 304 Not Modified
Comment 17 Eric Covener 2014-04-11 02:43:20 UTC
It's a copout, but I've made this configurable in r1586542 with a default for now of maintaining the 2.4 behavior.
Comment 18 Yann Ylavic 2014-04-16 11:27:53 UTC
*** Bug 56354 has been marked as a duplicate of this bug. ***
Comment 19 Jesús Cea 2014-04-23 21:27:46 UTC
Eric, I am graceful for the new directive, but why not implementing a better ETAG verification instead of altering the ETAG generation?.

If you get an etag with a "-gzip" suffix, you just verify the etag without that suffix. You could link this to the "accept-encoding" header, maybe.
Comment 20 Eric Covener 2014-04-23 21:38:26 UTC
(In reply to Jesús Cea from comment #19)
> Eric, I am graceful for the new directive, but why not implementing a better
> ETAG verification instead of altering the ETAG generation?.

My immediate concern is the unnecessary difference between 2.2 and 2.4, allowing the 2.2 behavior in 2.4 is a lot simpler to tackle then adding some third behavior (I admitted before it was a copout)
Comment 21 Martin Heide 2015-02-13 08:10:40 UTC
(In reply to Eric Covener from comment #17)
> It's a copout, but I've made this configurable in r1586542 with a default
> for now of maintaining the 2.4 behavior.

Hi Eric,
will your workaround of r1586542 make it into some Apache 2.4 release? I saw that it 
is integrated in 2.5, but it is not mentioned in the 2.4 docs.
I tried the DeflateAlterETag directive, but it does not seem to available in Ubuntu 14.04 LTS (Apache 2.4.7).

Since you only added a directive, this cannot break any existing server configurations, so it should be available in 2.4, too!
Comment 22 joost.dekeijzer 2015-11-08 22:00:51 UTC
Am I correct to read in http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 that the If-None-Match header can contain mulitple ETAG values?

If so, maybe an alternative version of rahuls workaround could be:

RequestHeader edit "If-None-Match" '^"((.*)-gzip)"$' '"$1", "$2"'

With above line, I think there is no need to modify the outgoing Header and the modefied RequestHeader works in with or without deflation