Bug 61176 - Content-Encoding gzip will prevent cache manager to cache embedded resources
Summary: Content-Encoding gzip will prevent cache manager to cache embedded resources
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 3.2
Hardware: PC All
: P2 enhancement with 1 vote (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-06-10 21:14 UTC by Thomas Hoffmann
Modified: 2017-07-26 19:57 UTC (History)
2 users (show)



Attachments
JMeter with working cache (34.42 KB, image/png)
2017-06-12 12:21 UTC, Thomas Hoffmann
Details
JMeter with non-working cache because of gzip (41.90 KB, image/png)
2017-06-12 12:22 UTC, Thomas Hoffmann
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Hoffmann 2017-06-10 21:14:20 UTC
Hello,
I tracked down an issue with version 3.2
I use Cache Manager and a GET request within a thread group. In the advanced settings of the http request, fetching the embedded resources is checked.
If the included CSS or JS files are gzip compressed, the Cache is not working any more. The internal method "setCache" is not called for these files.

"Content-Encoding:gzip" in the http response header will disable caching and prevent the Cache Manager to work.
In my opinion it should cache despite compression as compression is recommended for speeding up pages.

When I disable mod_deflate, JMeter works as expected and caches the static files which are included in the HTML page.

If you need further information, I can set up a sample page.

Kind regards,
Thomas
Comment 1 Philippe Mouawad 2017-06-10 22:50:26 UTC
Hello,
Thanks for report.
Are you sure that your issue is not due to presence of Vary header ?
Would it be possible to provide a Sample request and response ?

Thanks
Comment 2 Thomas Hoffmann 2017-06-12 12:21:49 UTC
Created attachment 35046 [details]
JMeter with working cache
Comment 3 Thomas Hoffmann 2017-06-12 12:22:09 UTC
Created attachment 35047 [details]
JMeter with non-working cache because of gzip
Comment 4 Thomas Hoffmann 2017-06-12 12:25:09 UTC
Hello,

I set up a test environment with two URLs:
http://52.59.21.204/jmeter.php
http://52.59.21.204/jmeter2.php

The second link contains a JS which is compressed.
This causes JMeter to fetch the JS again and again.
I also attached screenshots of wireshark which shows the behaviour.

In the screenshots you can see the first time the PHP and included CSS and JS are fetched.
Further calls will fetch the JS file if it was delivered with "Content-Encoding:gzip"

Hope this helps to figure out the reason.

Thank you in advance,
Thomas
Comment 5 Thomas Hoffmann 2017-06-12 12:31:48 UTC
Addendum: Request and Response

http://52.59.21.204/jmeter2.js

Request-Headers:
================
Accept:*/*
Accept-Encoding:gzip, deflate, sdch
Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Cache-Control:no-cache
Connection:keep-alive
Host:52.59.21.204
Pragma:no-cache
Referer:http://52.59.21.204/jmeter2.php
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36

Response-Headers:
=================
Accept-Ranges:bytes
Cache-Control:max-age=2419197
Connection:Keep-Alive
Content-Encoding:gzip
Content-Length:58
Content-Type:application/javascript
Date:Mon, 12 Jun 2017 12:11:23 GMT
ETag:"b6-551c23791451c-gzip"
Expires:Mon, 10 Jul 2017 12:11:21 GMT
Keep-Alive:timeout=5, max=100
Last-Modified:Mon, 12 Jun 2017 12:11:21 GMT
Server:Apache/2.4.18 (Ubuntu)
Vary:Accept-Encoding
Comment 6 Felix Schumacher 2017-06-12 15:13:47 UTC
(In reply to Thomas Hoffmann from comment #5)
> Addendum: Request and Response
> 
> http://52.59.21.204/jmeter2.js
> 
> Request-Headers:
> ================
> Accept:*/*
> Accept-Encoding:gzip, deflate, sdch
> Accept-Language:de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
...
> 
> Response-Headers:
> =================
> Accept-Ranges:bytes
> Cache-Control:max-age=2419197
...
> Vary:Accept-Encoding

JMeter will not cache responses with "Vary" header in the current releases.

The same behaviour would be seen with the php pages, as both jmeter.php and jmeter2.php have the "Vary" header.

Feel free to discuss the current behaviour on the mailing list. Currently I would see this as an enhancement request rather than a bug.
Comment 7 Thomas Hoffmann 2017-06-12 15:31:08 UTC
Hello,

thanks for the analysis.

I have checked the documentation of mod_deflate:
http://httpd.apache.org/docs/2.0/mod/mod_deflate.html

"The mod_deflate module sends a Vary: Accept-Encoding HTTP response header to alert proxies that a cached response should be sent only to clients that send the appropriate Accept-Encoding request header. This prevents compressed content from being sent to a client that will not understand it."

So when using the mod_deflate module, caching in JMeter doesn't work any more.
As this module is very common, I consider this feature as quite important.

According to https://w3techs.com/technologies/details/ce-gzipcompression/all/all
over 70% of the websites are using compression. This means that JMeter caching is not working for over 70% of the websites (without modifying server settings).

Prevent caching was introduced with this ticket:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58079
Comment 8 Felix Schumacher 2017-07-13 18:26:46 UTC
@thomas, could you test, if current trunk works for you?

This will be included in JMeter version 3.3.

Date: Thu Jul 13 17:43:15 2017
New Revision: 1801854

URL: http://svn.apache.org/viewvc?rev=1801854&view=rev
Log:
We are not not doing any real work here and it simplifies our code.

In preparation of Bugzilla Id: 61176 (and github pr #298)

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java

Date: Thu Jul 13 17:44:57 2017
New Revision: 1801855

URL: http://svn.apache.org/viewvc?rev=1801855&view=rev
Log:
Use log string templates instead of string concatenation.

This should make the code a bit more readable.

In preparation of Bugzilla Id: 61176 (and github pr #298)

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java
Date: Thu Jul 13 17:51:49 2017
New Revision: 1801856

URL: http://svn.apache.org/viewvc?rev=1801856&view=rev
Log:
Extract code into local private methods.

Extract the various methods for calculation of the expiration date into
smaller methods in an attempt to make code more readable.

In preparation of Bugzilla Id: 61176 (and github pr #298)

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java

Date: Thu Jul 13 17:53:29 2017
New Revision: 1801857

URL: http://svn.apache.org/viewvc?rev=1801857&view=rev
Log:
Rename local variable to comply with java naming conventions.

In preparation of Bugzilla Id: 61176 (and github pr #298)

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java

Date: Thu Jul 13 17:54:29 2017
New Revision: 1801858

URL: http://svn.apache.org/viewvc?rev=1801858&view=rev
Log:
Reformat to make it easier to read.

In preparation of Bugzilla Id: 61176 (and github pr #298)

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java
Date: Thu Jul 13 18:23:05 2017
New Revision: 1801859

URL: http://svn.apache.org/viewvc?rev=1801859&view=rev
Log:
Implement caching of requests with "vary" header.

Bugzilla Id: 61176 (closes #298 on github)

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPJavaImpl.java
    jmeter/trunk/test/src/org/apache/jmeter/protocol/http/control/TestCacheManagerBase.java
    jmeter/trunk/xdocs/changes.xml
    jmeter/trunk/xdocs/usermanual/component_reference.xml
Comment 9 Felix Schumacher 2017-07-26 19:57:17 UTC
Date: Wed Jul 26 19:56:08 2017
New Revision: 1803103

URL: http://svn.apache.org/viewvc?rev=1803103&view=rev
Log:
Guard debug log messages that call a function with Logger#isDebugEnabled

Bugzilla Id: 61176

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java