Bug 58786 - JDBC Sampler : Replace Excalibur DataSource by more up to date library commons-dbcp2
Summary: JDBC Sampler : Replace Excalibur DataSource by more up to date library common...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 2.13
Hardware: All All
: P2 enhancement (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-12-30 18:31 UTC by Philippe Mouawad
Modified: 2016-03-05 20:24 UTC (History)
1 user (show)



Attachments
Implementation of Bug (29.37 KB, patch)
2015-12-30 18:46 UTC, Philippe Mouawad
Details | Diff
Patch using DBCP2 (30.08 KB, patch)
2015-12-30 21:49 UTC, Philippe Mouawad
Details | Diff
Test plan used for comparison (6.85 KB, application/xml)
2015-12-30 21:53 UTC, Philippe Mouawad
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Philippe Mouawad 2015-12-30 18:31:02 UTC
As per previous discussion on dev mailing list:
- http://mail-archives.apache.org/mod_mbox/jmeter-dev/201401.mbox/%3CCAH9fUpa%2BJ636TXiWJYM1%2B7nvO3v9pi0xbdAsH_L1bpMiutsZ_A%40mail.gmail.com%3E

- http://mail-archives.apache.org/mod_mbox/jmeter-dev/201201.mbox/%3CCAH9fUpYWBNH4bMzw0SwZEwN_GKKy1qe42k-i%3DZb6_kZo2FLLXQ%40mail.gmail.com%3E

I propose to switch to Tomcat Pool for the following reasons:
- It is used by Tomcat so useful for developers who want to test the pool options
- It is up to date while Excalibur is in the ATTIC
- It supports JDBC 4 , not excalibur
- It appears to have nice performances
Comment 1 Philippe Mouawad 2015-12-30 18:46:30 UTC
Created attachment 33387 [details]
Implementation of Bug
Comment 2 Felix Schumacher 2015-12-30 18:58:38 UTC
tomcat-jdbc is probably not the best choice. A better choice would be dbcp2 from commons.

That db pool is included in tomcat (8 and higher) also and is the default implementation there, so it should have a higher test coverage by real users.
Comment 3 Philippe Mouawad 2015-12-30 21:49:02 UTC
Created attachment 33388 [details]
Patch using DBCP2
Comment 4 Philippe Mouawad 2015-12-30 21:52:45 UTC
Hi Felix,
I attached a new patch using DBCP2.
BUT , comparing performances, it is strange as it appears DBCP2 is slower than Excalibur (when using Max Number of Connections > 0) , throughput is divided by 2.

WHILE Tomcat-Pool has a much higher throughput than excalibur and DBCP2.


Maybe I made a mistake somewhere, so an external eye is welcome.

Regards

Philippe
Comment 5 Philippe Mouawad 2015-12-30 21:53:11 UTC
Created attachment 33389 [details]
Test plan used for comparison
Comment 6 Philippe Mouawad 2015-12-30 22:28:38 UTC
Update:

I made a benchmark based on attached test plan and I am wrong in my last comment:
DBCP:
Generate Summary Results =  50000 in 00:00:06 = 8860.5/s Avg:     4 Min:     0 Max:   417 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:05 = 9109.1/s Avg:     4 Min:     0 Max:   411 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:06 = 9053.1/s Avg:     4 Min:     0 Max:   416 Err:     0 (0.00%)

TOMCAT POOL:
Generate Summary Results =  50000 in 00:00:04 = 12242.9/s Avg:     3 Min:     0 Max:   591 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:03 = 15216.1/s Avg:     2 Min:     0 Max:   422 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:03 = 14654.2/s Avg:     2 Min:     0 Max:   412 Err:     0 (0.00%)


EXCALIBUR
Generate Summary Results =  50000 in     2s = 30921.5/s Avg:     0 Min:     0 Max:   214 Err:     0 (0.00%)
Generate Summary Results =  50000 in     2s = 29360.0/s Avg:     0 Min:     0 Max:   252 Err:     0 (0.00%)
Generate Summary Results =  50000 in     2s = 31645.6/s Avg:     0 Min:     0 Max:   186 Err:     0 (0.00%)



So if my benchmark is not wrong:
Excalibur is the fastest, then Tomcat JDBC and finally DBCP2

Look strange to me.
Comment 7 Steven Swor 2015-12-31 05:19:53 UTC
Hi Phillippe,

Have you evaluated C3P0 or Commons-Pool as possible alternatives? It would be nice to see some throughput comparison of those pooling implementations as well (not just from a JMeter perspective, but for the J2EE developer community as a whole).
Comment 8 Steven Swor 2015-12-31 05:33:17 UTC
(In reply to Steven Swor from comment #7)
> Hi Phillippe,
> 
> Have you evaluated C3P0 or Commons-Pool as possible alternatives? It would
> be nice to see some throughput comparison of those pooling implementations
> as well (not just from a JMeter perspective, but for the J2EE developer
> community as a whole).

I just stumbled upon https://github.com/brettwooldridge/HikariCP which claims to be a really high performing pooling implementation. Probably worth a look as well.
Comment 9 Pascal Schumacher 2015-12-31 07:51:46 UTC
HikariCP would be my first choice. It is great, apache licensed and has very helpful documentation, e.g. https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-doesn't-show-a-difference.%22
Comment 10 Milamber 2015-12-31 08:44:01 UTC
HikariCP have very impressive performances against the other pool managers! Probably a good choice to replace Excalibur (but need to be confirm via a performance comparison)
Comment 11 Philippe Mouawad 2015-12-31 16:25:31 UTC
Hi,
I agree perf seem impressive but there are other factors that are important and one that I don't find great is that I see 1 main contributor vs many for dbcp2 or tomcat-jdbc.

Anyway I find it strange that perf of dbcp and tomcat-jdbc are not better than excalibur.
Comment 12 Philippe Mouawad 2015-12-31 17:48:23 UTC
Results with HikariCP for the attached JMX:
Generate Summary Results =  50000 in 00:00:03 = 18063.6/s Avg:     1 Min:     0 Max:    61 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:03 = 18254.8/s Avg:     1 Min:     0 Max:    68 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:03 = 18382.4/s Avg:     1 Min:     0 Max:    80 Err:     0 (0.00%)
Comment 13 Philippe Mouawad 2015-12-31 17:54:10 UTC
Using dataSourceConfig.setDataSourceClassName("org.postgresql.ds.PGSimpleDataSource")
instead of : dataSource.setDriverClassName(getDriver());

As per :
- https://github.com/brettwooldridge/HikariCP#configuration-knobs-baby

Results are the best:

Generate Summary Results =  50000 in 00:00:01 = 36231.9/s Avg:     0 Min:     0 Max:   225 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:01 = 35663.3/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:01 = 35612.5/s Avg:     0 Min:     0 Max:   227 Err:     0 (0.00%)
Comment 14 Philippe Mouawad 2015-12-31 18:01:35 UTC
I am not fair in comparison, org.postgresql.ds.PGSimpleDataSource is a postgres class, wait for more
Comment 15 Felix Schumacher 2015-12-31 19:00:06 UTC
You could even use h2 as a jdbc driver with inbuilt db.

I tried the three different sets excalibur, tomcat-jdbc and dbcp2. When using the jdbc uri jdbc:h2:~/test;TRACE_LEVEL_FILE=2 (with 5 threads, pool-size and loops) it will generate a trace file under ~/test.trace.db which shows, that tomcat-jdbc and dbcp2 will create 5 connections, but excalibur seems to create only one connection.
Comment 16 Philippe Mouawad 2016-01-01 11:40:19 UTC
Hi Felix,
I made further tests:
1/ I don't see what you notice
2/ I was wrong in my previous approach as my machine was overloaded at 50 threads so results were meaningless


I made a new set of tests:
5 threads
DBCP2:
Generate Summary Results =  50000 in 00:00:02 = 22212.4/s Avg:     0 Min:     0 Max:    50 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:02 = 22341.4/s Avg:     0 Min:     0 Max:    45 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:02 = 22431.6/s Avg:     0 Min:     0 Max:    45 Err:     0 (0.00%)


Excalibur:
Generate Summary Results =  50000 in 00:00:02 = 22716.9/s Avg:     0 Min:     0 Max:     9 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:02 = 22779.0/s Avg:     0 Min:     0 Max:    10 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:02 = 22573.4/s Avg:     0 Min:     0 Max:     9 Err:     0 (0.00%)

Tomcat JDBC:
Generate Summary Results =  50000 in 00:00:02 = 23946.4/s Avg:     0 Min:     0 Max:     1 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:02 = 24108.0/s Avg:     0 Min:     0 Max:     1 Err:     0 (0.00%)
Generate Summary Results =  50000 in 00:00:02 = 24084.8/s Avg:     0 Min:     0 Max:     1 Err:     0 (0.00%)




10 Threads:
Excalibur
Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future
Writing log file to:/data/decathlon/workspace/jmeter/bin/jmeter.log
Generate Summary Results +      1 in 00:00:00 =    5.0/s Avg:    65 Min:    65 Max:    65 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 29841.5/s Avg:     0 Min:     0 Max:   105 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:04 = 28161.1/s Avg:     0 Min:     0 Max:   105 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 34916.2/s Avg:     0 Min:     0 Max:    10 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 35112.4/s Avg:     0 Min:     0 Max:    10 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    9.2/s Avg:     1 Min:     1 Max:     1 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 36179.1/s Avg:     0 Min:     0 Max:     9 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 34794.7/s Avg:     0 Min:     0 Max:     9 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    9.3/s Avg:     0 Min:     0 Max:     0 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 36562.7/s Avg:     0 Min:     0 Max:    11 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 35161.7/s Avg:     0 Min:     0 Max:    11 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 35174.1/s Avg:     0 Min:     0 Max:     9 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    9.5/s Avg:     0 Min:     0 Max:     0 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 36429.5/s Avg:     0 Min:     0 Max:    11 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 35087.7/s Avg:     0 Min:     0 Max:    11 Err:     0 (0.00%)



Tomcat jdbc:
Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future
Writing log file to:/data/decathlon/workspace/jmeter/bin/jmeter.log
Generate Summary Results +  40486 in 00:00:02 = 20826.1/s Avg:     0 Min:     0 Max:    79 Err:     0 (0.00%) Active: 10 Started: 10 Finished: 0
Generate Summary Results +  59514 in 00:00:01 = 43728.1/s Avg:     0 Min:     0 Max:     5 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 30257.2/s Avg:     0 Min:     0 Max:    79 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 36589.8/s Avg:     0 Min:     0 Max:     2 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 37050.8/s Avg:     0 Min:     0 Max:     3 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    8.8/s Avg:     1 Min:     1 Max:     1 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 38008.0/s Avg:     0 Min:     0 Max:     8 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 36429.9/s Avg:     0 Min:     0 Max:     8 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 37119.5/s Avg:     0 Min:     0 Max:     2 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    9.0/s Avg:     1 Min:     1 Max:     1 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 38925.3/s Avg:     0 Min:     0 Max:     6 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 37313.4/s Avg:     0 Min:     0 Max:     6 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 36954.9/s Avg:     0 Min:     0 Max:     2 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 36859.6/s Avg:     0 Min:     0 Max:     2 Err:     0 (0.00%)


DBCP2:
Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the future
Writing log file to:/data/decathlon/workspace/jmeter/bin/jmeter.log
Generate Summary Results = 100000 in 00:00:04 = 26246.7/s Avg:     0 Min:     0 Max:   312 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 33233.6/s Avg:     0 Min:     0 Max:   210 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 33670.0/s Avg:     0 Min:     0 Max:   172 Err:     0 (0.00%)
Generate Summary Results +  99877 in 00:00:03 = 33403.7/s Avg:     0 Min:     0 Max:   162 Err:     0 (0.00%) Active: 1 Started: 10 Finished: 9
Generate Summary Results +    123 in 00:00:00 = 8200.0/s Avg:     0 Min:     0 Max:     1 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 33266.8/s Avg:     0 Min:     0 Max:   162 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.2/s Avg:   170 Min:   170 Max:   170 Err:     0 (0.00%) Active: 2 Started: 2 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 38520.4/s Avg:     0 Min:     0 Max:    69 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 34411.6/s Avg:     0 Min:     0 Max:   170 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 34387.9/s Avg:     0 Min:     0 Max:   158 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.3/s Avg:   157 Min:   157 Max:   157 Err:     0 (0.00%) Active: 2 Started: 2 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 37864.1/s Avg:     0 Min:     0 Max:    55 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 34002.0/s Avg:     0 Min:     0 Max:   157 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.4/s Avg:   160 Min:   160 Max:   160 Err:     0 (0.00%) Active: 2 Started: 2 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 36576.1/s Avg:     0 Min:     0 Max:    59 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 33025.1/s Avg:     0 Min:     0 Max:   160 Err:     0 (0.00%)

HikariCP:
Generate Summary Results = 100000 in 00:00:04 = 26617.0/s Avg:     0 Min:     0 Max:   320 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 33079.7/s Avg:     0 Min:     0 Max:   176 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 33806.6/s Avg:     0 Min:     0 Max:   174 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 34223.1/s Avg:     0 Min:     0 Max:   162 Err:     0 (0.00%)
Generate Summary Results +  95982 in 00:00:03 = 35430.8/s Avg:     0 Min:     0 Max:   161 Err:     0 (0.00%) Active: 4 Started: 10 Finished: 6
Generate Summary Results +   4018 in 00:00:00 = 18775.7/s Avg:     0 Min:     0 Max:     1 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 34211.4/s Avg:     0 Min:     0 Max:   161 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.4/s Avg:   160 Min:   160 Max:   160 Err:     0 (0.00%) Active: 2 Started: 2 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 37849.7/s Avg:     0 Min:     0 Max:    61 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 34059.9/s Avg:     0 Min:     0 Max:   160 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.4/s Avg:   161 Min:   161 Max:   161 Err:     0 (0.00%) Active: 2 Started: 2 Finished: 0
Generate Summary Results +  99999 in 00:00:03 = 37749.7/s Avg:     0 Min:     0 Max:    61 Err:     0 (0.00%) Active: 0 Started: 10 Finished: 10
Generate Summary Results = 100000 in 00:00:03 = 33955.9/s Avg:     0 Min:     0 Max:   161 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 33978.9/s Avg:     0 Min:     0 Max:   156 Err:     0 (0.00%)
Generate Summary Results = 100000 in 00:00:03 = 33602.2/s Avg:     0 Min:     0 Max:   158 Err:     0 (0.00%)


20 Threads:
Excalibur
Generate Summary Results = 200000 in 00:00:05 = 39223.4/s Avg:     0 Min:     0 Max:   197 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 42176.3/s Avg:     0 Min:     0 Max:   126 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40225.3/s Avg:     0 Min:     0 Max:   125 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40858.0/s Avg:     0 Min:     0 Max:   127 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 41042.5/s Avg:     0 Min:     0 Max:   126 Err:     0 (0.00%)
Generate Summary Results +  64952 in 00:00:02 = 35319.2/s Avg:     0 Min:     0 Max:   126 Err:     0 (0.00%) Active: 20 Started: 20 Finished: 0
Generate Summary Results + 135048 in 00:00:03 = 45302.9/s Avg:     0 Min:     0 Max:   145 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 41493.8/s Avg:     0 Min:     0 Max:   145 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 41382.2/s Avg:     0 Min:     0 Max:   126 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 41849.8/s Avg:     0 Min:     0 Max:   140 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40891.4/s Avg:     0 Min:     0 Max:   130 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40783.0/s Avg:     0 Min:     0 Max:   125 Err:     0 (0.00%)


Tomcat jdbc:
Generate Summary Results + 177333 in 00:00:04 = 45798.8/s Avg:     0 Min:     0 Max:   131 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 43177.9/s Avg:     0 Min:     0 Max:   131 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 43994.7/s Avg:     0 Min:     0 Max:   127 Err:     0 (0.00%)
Generate Summary Results + 113334 in 00:00:03 = 44288.4/s Avg:     0 Min:     0 Max:   126 Err:     0 (0.00%) Active: 20 Started: 20 Finished: 0
Generate Summary Results +  86666 in 00:00:02 = 47828.9/s Avg:     0 Min:     0 Max:   125 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:04 = 45756.1/s Avg:     0 Min:     0 Max:   126 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    8.9/s Avg:     1 Min:     1 Max:     1 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results + 199999 in 00:00:04 = 46082.7/s Avg:     0 Min:     0 Max:   245 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:04 = 44923.6/s Avg:     0 Min:     0 Max:   245 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:04 = 45987.6/s Avg:     0 Min:     0 Max:   156 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    9.7/s Avg:     1 Min:     1 Max:     1 Err:     0 (0.00%) Active: 1 Started: 1 Finished: 0
Generate Summary Results + 199999 in 00:00:04 = 46739.7/s Avg:     0 Min:     0 Max:   125 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:04 = 45630.8/s Avg:     0 Min:     0 Max:   125 Err:     0 (0.00%)


DBCP2:
Generate Summary Results = 200000 in 00:00:05 = 39062.5/s Avg:     0 Min:     0 Max:   173 Err:     0 (0.00%)
Generate Summary Results +  89785 in 00:00:02 = 37582.7/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%) Active: 20 Started: 20 Finished: 0
Generate Summary Results + 110215 in 00:00:03 = 42768.7/s Avg:     0 Min:     0 Max:   128 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 40273.9/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.8/s Avg:   154 Min:   154 Max:   154 Err:     0 (0.00%) Active: 4 Started: 4 Finished: 0
Generate Summary Results + 199999 in 00:00:05 = 42462.6/s Avg:     0 Min:     0 Max:   207 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 40225.3/s Avg:     0 Min:     0 Max:   207 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40625.6/s Avg:     0 Min:     0 Max:   164 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 39666.8/s Avg:     0 Min:     0 Max:   160 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40379.6/s Avg:     0 Min:     0 Max:   161 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40184.9/s Avg:     0 Min:     0 Max:   158 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40176.8/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.6/s Avg:   159 Min:   159 Max:   159 Err:     0 (0.00%) Active: 4 Started: 4 Finished: 0
Generate Summary Results + 199999 in 00:00:05 = 42899.8/s Avg:     0 Min:     0 Max:   131 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 40502.2/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40404.0/s Avg:     0 Min:     0 Max:   165 Err:     0 (0.00%)

HikariCP:
Generate Summary Results + 198611 in 00:00:05 = 40278.0/s Avg:     0 Min:     0 Max:   167 Err:     0 (0.00%) Active: 2 Started: 20 Finished: 18
Generate Summary Results +   1389 in 00:00:00 = 15433.3/s Avg:     0 Min:     0 Max:     1 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 39832.7/s Avg:     0 Min:     0 Max:   167 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.6/s Avg:   170 Min:   170 Max:   170 Err:     0 (0.00%) Active: 4 Started: 4 Finished: 0
Generate Summary Results + 199999 in 00:00:05 = 42982.8/s Avg:     0 Min:     0 Max:   141 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 40535.1/s Avg:     0 Min:     0 Max:   170 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 40461.3/s Avg:     0 Min:     0 Max:   162 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 39904.2/s Avg:     0 Min:     0 Max:   164 Err:     0 (0.00%)
Generate Summary Results + 190298 in 00:00:05 = 39869.7/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%) Active: 9 Started: 20 Finished: 11
Generate Summary Results +   9702 in 00:00:00 = 23434.8/s Avg:     0 Min:     0 Max:    26 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 38550.5/s Avg:     0 Min:     0 Max:   159 Err:     0 (0.00%)
Generate Summary Results +      1 in 00:00:00 =    3.7/s Avg:   163 Min:   163 Max:   163 Err:     0 (0.00%) Active: 4 Started: 4 Finished: 0
Generate Summary Results + 199999 in 00:00:05 = 40766.2/s Avg:     0 Min:     0 Max:   150 Err:     0 (0.00%) Active: 0 Started: 20 Finished: 20
Generate Summary Results = 200000 in 00:00:05 = 38639.9/s Avg:     0 Min:     0 Max:   163 Err:     0 (0.00%)
Generate Summary Results = 200000 in 00:00:05 = 38956.0/s Avg:     0 Min:     0 Max:   152 Err:     0 (0.00%)
Comment 17 Felix Schumacher 2016-01-01 12:06:50 UTC
Those results seem to be more correct. If you look at profiling data (I obtained them with java 8 and jmc), you will see, that the actual time spent in the db pool is minimal compared to the time spent in the db or in other parts of jmeter.

I used 

 JVM_ARGS=-"XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true,dumponexitpath=/tmp/flightrecording.jfr" ./bin/jmeter -n -t ~/BUG_58786.jmx

to record the profiling data and used jmc to look at them.

Looking at your numbers I don't see a reason to update the implementation for its speed.

If we want to swap implementation because of excaliburs age (and given it is in attic), then I would favour dbcp2, even if it is a bit slower, because it has more developers than tomcat-jdbc and hikari.
Comment 18 Felix Schumacher 2016-01-01 12:20:40 UTC
(In reply to Philippe Mouawad from comment #16)
> Hi Felix,
> I made further tests:
> 1/ I don't see what you notice

In my tests I found the following:

1. excalibur seems to build the connections in the pool on demand and since demand was low, it only build and used one connection.

2. tomcat-jdbc and dbcp2 build all connections up front, but used only 2 out of 5

3. dbcp2 made one connection before the pooled connections where built. Probably to check, if it could reach the database.

[To check for the number of opened connection just grep "getConnection" in the trace file.]
Comment 19 Philippe Mouawad 2016-01-01 18:04:19 UTC
Author: pmouawad
Date: Fri Jan  1 17:59:04 2016
New Revision: 1722514

URL: http://svn.apache.org/viewvc?rev=1722514&view=rev
Log:
Bug 58786 - JDBC Sampler : Replace Excalibur DataSource by more up to date library commons-dbcp2
Bugzilla Id: 58786

Modified:
    jmeter/trunk/build.properties
    jmeter/trunk/build.xml
    jmeter/trunk/eclipse.classpath
    jmeter/trunk/lib/   (props changed)
    jmeter/trunk/lib/aareadme.txt
    jmeter/trunk/licenses/bin/README.txt
    jmeter/trunk/res/maven/ApacheJMeter_parent.pom
    jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/config/DataSourceElement.java
    jmeter/trunk/xdocs/changes.xml
    jmeter/trunk/xdocs/usermanual/component_reference.xml
Comment 20 Philippe Mouawad 2016-01-01 18:07:20 UTC
Author: pmouawad
Date: Fri Jan  1 18:06:55 2016
New Revision: 1722520

URL: http://svn.apache.org/viewvc?rev=1722520&view=rev
Log:
Bug 58786 - JDBC Sampler : Replace Excalibur DataSource by more up to date library commons-dbcp2
Bugzilla Id: 58786

Modified:
    jmeter/trunk/build.properties
Comment 21 Philippe Mouawad 2016-02-23 12:53:19 UTC
Author: pmouawad
Date: Tue Feb 23 12:51:03 2016
New Revision: 1731847

URL: http://svn.apache.org/viewvc?rev=1731847&view=rev
Log:
Bug 58786 - JDBC Sampler : Replace Excalibur DataSource by more up to date library commons-dbcp2
Update documentation and rename properties to match DBCP names (and more recent pools)
Bugzilla Id: 58786

Modified:
    jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/config/DataSourceElementResources.properties
    jmeter/trunk/xdocs/usermanual/component_reference.xml
Comment 22 Sebb 2016-02-28 21:20:33 UTC
Screenshots need updating
Comment 23 Philippe Mouawad 2016-03-05 20:24:12 UTC
(In reply to Sebb from comment #22)
> Screenshots need updating

Hi Sebb, I suggest we fix this within release preparation.
There are a lot of screenshots everywhere to update as many guis have changed.

So closing it as I created Bug 59126

Regards