|Summary:||AbstractJDBCTest Element exhausts available memory caching all PoolGuardConnectionWrappers|
|Product:||JMeter||Reporter:||Justin McCartney <be_strew>|
|Component:||Main||Assignee:||JMeter issues mailing list <issues>|
Heap dump analysis
Example Jmeter script with memory leak
Remove perConnMap as it is useless and has a memory leak
Description Justin McCartney 2016-09-05 15:29:52 UTC
So in our test where we encountered the error we have 20 threads, executing a mix of HTTP and JDBC requests. We noticed that after a period of time (2.5 hours) the response times would appear to slow down and it was found that this was due to Jmeter, rather than the application under test. The JDBC connection config has Max Number of connections = 0, in order to assign a single pool to each thread with one connection in it. Jmeter was responding to memory pressure by performing full GCs and this was skewing the response time results. Taking a heap dump after running a test for 2 hours it was noticed that there were 212,185 AbstractJDBCTestElements in the heap and also the same number of org.apache.commons.dbcp2.PoolingDataSource$PoolGuardConnectionWrapper classes. The PoolingDataSource for DBCP (v 2.2.1), which appears bundled with Jmeter, returns an instance of PoolingDataSource$PoolGuardConnectionWrapper for each call to getConnection(). The PoolGuardConnectionWrapper does not override the default hashcode/equals. Therefore, as the test runs the static map perConnCache maintained by the AbstractJDBCTestElements class fills up as each time we hit getConnection, we are receiving a new object. This behaviour is unaffected by the LRU for the prepared statements as the map sees a new connection each time. For a reason I have not found yet, this also causes a link to be held to the AbstractJDBCTestElements. This will cause Jmeter to run out of memory and breaks our long duration tests. I haven't tested this with previous versions of Jmeter, as this is a new application under test and I don't have time to back port the tests.
Comment 1 Justin McCartney 2016-09-05 15:31:28 UTC
Created attachment 34200 [details] Heap dump analysis Added a screenshot from the heap dump
Comment 2 Felix Schumacher 2016-09-06 06:00:36 UTC
What type of sql statements do you use and are you keeping the samplers? Could you provide a minimal test that exhibits the described behaviour?
Comment 3 Justin McCartney 2016-09-06 09:42:54 UTC
Created attachment 34204 [details] Example Jmeter script with memory leak The problem occurs with prepared statements, as these involve the cache of prepared statements that AbstractJdbcTestElement maintains (perConnCache). As discussed above each time a connection is pulled from the pool, it has a different Java identity and therefore the map grows unbounded. I have attached a simplified test that shows the problem. This test looks for Postgres to be on localhost@5432 with a DB called jmetertest. I tried this with Postgres 9.5 on Ubuntu. I just used the postgres user. The following table needs created: CREATE TABLE public.test ( id integer NOT NULL DEFAULT nextval('test_id_seq'::regclass), text character varying(64), CONSTRAINT "PK_ID" PRIMARY KEY (id) ) WITH ( OIDS=FALSE ); ALTER TABLE public.test OWNER TO postgres; If you run the test for a minute then take a heap dump you will see an example of the problem it appears immediately. The count of instances of org.apache.commons.dbcp2.PoolingDataSource$PoolGuardConnectionWrapper and org.apache.jmeter.protocol.jdbc.AbstractJDBCTestElement$1 will be the number of executions of the sampler.
Comment 4 Felix Schumacher 2016-09-06 18:23:00 UTC
Created attachment 34211 [details] Remove perConnMap as it is useless and has a memory leak Remove pool for prepared statements, as it didn't work with the current jdbc pool implementation and newer jdbc drivers should support pooling themselves.
Comment 5 Felix Schumacher 2016-09-07 19:28:58 UTC
Date: Wed Sep 7 19:26:17 2016 New Revision: 1759668 URL: http://svn.apache.org/viewvc?rev=1759668&view=rev Log: Remove cache for prepared statements, as it didn't work with the current jdbc pool implementation and current jdbc drivers should support caching of prepared statements themselves. Bugzilla Id: 60085 Modified: jmeter/trunk/bin/jmeter.properties jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java jmeter/trunk/xdocs/changes.xml jmeter/trunk/xdocs/usermanual/properties_reference.xml
Comment 6 Felix Schumacher 2016-09-10 16:21:30 UTC
Date: Sat Sep 10 16:16:01 2016 New Revision: 1760194 URL: http://svn.apache.org/viewvc?rev=1760194&view=rev Log: Followup to r1759668. Close PreparedStatements as they are no longer cached and reused. Bugzilla Id: 60085 Modified: jmeter/trunk/src/protocol/jdbc/org/apache/jmeter/protocol/jdbc/AbstractJDBCTestElement.java