Bug 52073 - Embedded Resources Parallel download : Improve performances by avoiding shutdown of ThreadPoolExecutor at each sample
Summary: Embedded Resources Parallel download : Improve performances by avoiding shutd...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: HTTP (show other bugs)
Version: 2.5.1
Hardware: All All
: P2 enhancement with 1 vote (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on: 58099
Blocks:
  Show dependency tree
 
Reported: 2011-10-23 21:02 UTC by Philippe Mouawad
Modified: 2016-03-18 09:35 UTC (History)
1 user (show)



Attachments
BROKEN PATCH to illustrate dev list discussion (15.10 KB, patch)
2014-09-01 19:15 UTC, Philippe Mouawad
Details | Diff
patch (24.67 KB, text/plain)
2016-02-21 19:57 UTC, benoit.wiart
Details
Benchmark test plan (27.27 KB, application/xml)
2016-03-08 22:49 UTC, Philippe Mouawad
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philippe Mouawad 2011-10-23 21:02:59 UTC
Related to 51919.
Currently for each download of resources, HttpSamplerBase will create a pool of Threads, do download and shutdown pool.
This results in lots of costly creations/destruction of Threads.
It would be better to create Pool once for Thread Group and make Samplers reuse it (as proposed by sebb)
Comment 1 Philippe Mouawad 2014-09-01 19:15:32 UTC
Created attachment 31959 [details]
BROKEN PATCH to illustrate dev list discussion
Comment 2 benoit.wiart 2016-02-19 16:05:10 UTC
PR https://github.com/apache/jmeter/pull/132

Please test and review carefully

A shared thread pool is used for all the parallel downloads.
A throttle mechanism is used per user to ensure the max concurrent
downloads setting is not exceeded.
A new parameter httpsampler.parallel_download_thread_keepalive_inseconds
is used to set the thread TTL when idle (default 60s).
the parameter httpsampler.await_termination_timeout is not used anymore
and has been removed.

TODO : ResourcesDownloader#shrink should be called at the end of a test
to accelerate the release of the thread pool idle threads (not
mandatory)
Comment 3 benoit.wiart 2016-02-21 19:54:51 UTC
I've closed the PR as it's hard to have both Bug 59034 and this one.
I will retry after the jmeter 3.0 release
Comment 4 benoit.wiart 2016-02-21 19:57:52 UTC
Created attachment 33580 [details]
patch

attach the pr 132 as patch
Comment 5 benoit.wiart 2016-03-04 08:32:35 UTC
PR https://github.com/apache/jmeter/pull/155

Back dans les bacs à chaque attaque, contact !

new version of the PR which should play nicely with Bug 59034.
the cache manager is now correctly shared between the threads of a given user.

Description :

Embedded Resources concurrent download : Improve performances
by avoiding shutdown of ThreadPoolExecutor at each sample.

A shared thread pool is used for all the parallel downloads.
A throttle mechanism is used per user to ensure the max concurrent
downloads setting is not exceeded.
A new parameter httpsampler.parallel_download_thread_keepalive_inseconds
is used to set the thread TTL when idle (default 60s).
the parameter httpsampler.await_termination_timeout is not used anymore
and has been removed.

TODO : ResourcesDownloader#shrink should be called at the end of a test
to accelerate the release of the thread pool idle threads (not
mandatory)

the httpclient is shared between the threads of a given virtual user.
the cache is shared between the threads of a given virtual user.
Comment 6 Philippe Mouawad 2016-03-08 22:49:20 UTC
Created attachment 33651 [details]
Benchmark test plan
Comment 7 Philippe Mouawad 2016-03-08 22:50:48 UTC
Hello,
I benchmarked the PR 155 by Benoit W.(that he updated this evening), the gain of Throughtput with 15 threads is 1.65 bigger than the throughput without it.

I will be commiting it tomorrow, unless there is opposition.

Regards

Benchmark conditions:
- Tomcat with session timeout 1 minute
- Keepalive on connector: 2s

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               enableLookups="false"
               keepAliveTimeout="2000"
               minSpareThreads="50"
               acceptorThreadCount="2"
               compression="off"
               redirectPort="8443" />

- Log level to WARN

JMeter in Non GUI mode:
-server -Xms1024m -Xmx1024m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xss256k -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false  -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42  -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause  -Xloggc:/data/jmeter/verbose-3.0.log -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Addresses=false
Comment 8 Philippe Mouawad 2016-03-09 12:26:27 UTC
Author: pmouawad
Date: Wed Mar  9 12:23:31 2016
New Revision: 1734228

URL: http://svn.apache.org/viewvc?rev=1734228&view=rev
Log:
[Bug 52073] Embedded Resources Parallel download : Improve performances by avoiding shutdown of ThreadPoolExecutor at each sample
Based on PR by Benoit Wiart + the addition (blame me) of JMeterPoolingClientConnectionManager  (see mailing list mail I will send)
Bugzilla Id: 52073

Added:
    jmeter/trunk/src/protocol/http/org/apache/http/
    jmeter/trunk/src/protocol/http/org/apache/http/impl/
    jmeter/trunk/src/protocol/http/org/apache/http/impl/conn/
    jmeter/trunk/src/protocol/http/org/apache/http/impl/conn/JMeterPoolingClientConnectionManager.java   (with props)
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/ResourcesDownloader.java   (with props)
Modified:
    jmeter/trunk/bin/jmeter.properties
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/HTTPSamplerBase.java
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/sampler/MeasuringConnectionManager.java
    jmeter/trunk/xdocs/changes.xml
Comment 9 Felix Schumacher 2016-03-18 09:35:23 UTC
Date: Fri Mar 18 09:25:58 2016
New Revision: 1735560

URL: http://svn.apache.org/viewvc?rev=1735560&view=rev
Log:
Followup to r17734228. No need for temporary variable. Return CacheManager directly.

Bugzilla Id: 52073

Modified:
    jmeter/trunk/src/protocol/http/org/apache/jmeter/protocol/http/control/CacheManager.java