Bug 60085 - AbstractJDBCTest Element exhausts available memory caching all PoolGuardConnectionWrappers
Summary: AbstractJDBCTest Element exhausts available memory caching all PoolGuardConne...
Status: RESOLVED FIXED
Alias: None
Product: JMeter
Classification: Unclassified
Component: Main (show other bugs)
Version: 3.0
Hardware: PC Linux
: P2 normal (vote)
Target Milestone: ---
Assignee: JMeter issues mailing list
URL:
Keywords:
: 60749 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-09-05 15:29 UTC by Justin McCartney
Modified: 2017-02-25 14:13 UTC (History)
1 user (show)



Attachments
Heap dump analysis (89.38 KB, image/png)
2016-09-05 15:31 UTC, Justin McCartney
Details
Example Jmeter script with memory leak (12.62 KB, application/xml)
2016-09-06 09:42 UTC, Justin McCartney
Details
Remove perConnMap as it is useless and has a memory leak (7.13 KB, patch)
2016-09-06 18:23 UTC, Felix Schumacher
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Justin McCartney 2016-09-05 15:29:52 UTC
So in our test where we encountered the error we have 20 threads, executing a mix of HTTP and JDBC requests.

We noticed that after a period of time (2.5 hours) the response times would appear to slow down and it was found that this was due to Jmeter, rather than the application under test.

The JDBC connection config has Max Number of connections = 0, in order to assign a single pool to each thread with one connection in it.

Jmeter was responding to memory pressure by performing full GCs and this was skewing the response time results.

Taking a heap dump after running a test for 2 hours it was noticed that there were 212,185 AbstractJDBCTestElements in the heap and also the same number of org.apache.commons.dbcp2.PoolingDataSource$PoolGuardConnectionWrapper classes.

The PoolingDataSource for DBCP (v 2.2.1), which appears bundled with Jmeter, returns an instance of PoolingDataSource$PoolGuardConnectionWrapper for each call to getConnection().  The PoolGuardConnectionWrapper does not override the default hashcode/equals.

Therefore, as the test runs the static map perConnCache maintained by the AbstractJDBCTestElements class fills up as each time we hit getConnection, we are receiving a new object.  This behaviour is unaffected by the LRU for the prepared statements as the map sees a new connection each time.

For a reason I have not found yet, this also causes a link to be held to the AbstractJDBCTestElements.

This will cause Jmeter to run out of memory and breaks our long duration tests.

I haven't tested this with previous versions of Jmeter, as this is a new application under test and I don't have time to back port the tests.
Comment 1 Justin McCartney 2016-09-05 15:31:28 UTC
Created attachment 34200 [details]
Heap dump analysis

Added a screenshot from the heap dump
Comment 2 Felix Schumacher 2016-09-06 06:00:36 UTC
What type of sql statements do you use and are you keeping the samplers? Could you provide a minimal test that exhibits the described behaviour?
Comment 3 Justin McCartney 2016-09-06 09:42:54 UTC
Created attachment 34204 [details]
Example Jmeter script with memory leak

The problem occurs with prepared statements, as these involve the cache of prepared statements that AbstractJdbcTestElement maintains (perConnCache).  As discussed above each time a connection is pulled from the pool, it has a different Java identity and therefore the map grows unbounded.

I have attached a simplified test that shows the problem.  This test looks for Postgres to be on localhost@5432 with a DB called jmetertest.

I tried this with Postgres 9.5 on Ubuntu.

I just used the postgres user.

The following table needs created:

CREATE TABLE public.test
(
  id integer NOT NULL DEFAULT nextval('test_id_seq'::regclass),
  text character varying(64),
  CONSTRAINT "PK_ID" PRIMARY KEY (id)
)
WITH (
  OIDS=FALSE
);
ALTER TABLE public.test
  OWNER TO postgres;


If you run the test for a minute then take a heap dump you will see an example of the problem it appears immediately.

The count of instances of org.apache.commons.dbcp2.PoolingDataSource$PoolGuardConnectionWrapper and org.apache.jmeter.protocol.jdbc.AbstractJDBCTestElement$1 will be the number of executions of the sampler.
Comment 4 Felix Schumacher 2016-09-06 18:23:00 UTC
Created attachment 34211 [details]
Remove perConnMap as it is useless and has a memory leak

Remove pool for prepared statements, as it didn't work with the current jdbc pool implementation and newer jdbc drivers should support pooling themselves.
Comment 5 Felix Schumacher 2016-09-07 19:28:58 UTC
Date: Wed Sep  7 19:26:17 2016
New Revision: 1759668

URL: http://svn.apache.org/viewvc?rev=1759668&view=rev
Log:
Remove cache for prepared statements, as it didn't work with the current
jdbc pool implementation and current jdbc drivers should support caching
of prepared statements themselves.

Bugzilla Id: 60085

Modified:
    jmeter/trunk/bin/jmeter.properties
    jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java
    jmeter/trunk/xdocs/changes.xml
    jmeter/trunk/xdocs/usermanual/properties_reference.xml
Comment 6 Felix Schumacher 2016-09-10 16:21:30 UTC
Date: Sat Sep 10 16:16:01 2016
New Revision: 1760194

URL: http://svn.apache.org/viewvc?rev=1760194&view=rev
Log:
Followup to r1759668. Close PreparedStatements as they are no longer cached and reused.

Bugzilla Id: 60085

Modified:
    jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java
Comment 7 Philippe Mouawad 2017-02-25 14:13:26 UTC
*** Bug 60749 has been marked as a duplicate of this bug. ***