Bug 53367 - Database failure may cause pool to hang
Database failure may cause pool to hang
Status: RESOLVED FIXED
Product: Tomcat Modules
Classification: Unclassified
Component: jdbc-pool
unspecified
PC All
: P2 critical (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
: 56660 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2012-06-06 01:01 UTC by Filip Hanik
Modified: 2015-06-04 17:12 UTC (History)
4 users (show)



Attachments
Test demonstrating jdbc-pool connection leakage (4.84 KB, text/x-java)
2014-07-22 01:36 UTC, Anton
Details
Unit test demonstrating fair queue not working properly (5.19 KB, text/plain)
2014-07-22 09:56 UTC, Filip Hanik
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Filip Hanik 2012-06-06 01:01:10 UTC
during the 
 con.connect(); 
call in the method 

protected PooledConnection borrowConnection(long now, PooledConnection con, String username, String password) in ConnectionPool.java

There is no counting down the size of the pool. This means, if a connection failure happens here, the pool size remains the same. This means that 'size' will show the pool as full, but in reality the pool is empty
Comment 1 Filip Hanik 2012-06-06 01:03:16 UTC
Fixed in r1346691
Comment 2 monteslu 2012-07-07 16:55:04 UTC
I don't believe this is fixed.  Pulled down from svn trunk, buit, and can still easily reproduce.
Comment 3 Filip Hanik 2012-07-09 15:55:27 UTC
I'll take a look today
Comment 4 Filip Hanik 2012-07-09 16:22:19 UTC
Please post your configuration. Reran all tests and I'm unable to reproduce this issue.
Comment 5 Filip Hanik 2012-07-09 17:43:30 UTC
Unable to reproduce after ticket has been reopened. Please reopen with at a minimum your configuration.
Comment 6 Avinash 2014-03-20 06:45:08 UTC
I'm able to reproduce the error on tomact 7.0.37

Exception: [http-bio-8080-exec-41] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:120; busy:119; idle:0; lastwait:30000].
Exception: [http-bio-8080-exec-33] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:120; busy:119; idle:0; lastwait:30000].
Exception: [http-bio-8080-exec-3] Timeout: Pool empty. Unable to fetch a connection in 30 seconds, none available[size:120; busy:119; idle:0; lastwait:30000].
Comment 7 Christopher Schultz 2014-03-20 19:14:34 UTC
How about Tomcat 7.0.52 -- the latest?

Also, you haven't provided any indication that this is a problem with the pool... it could just be misuse of the pool. You'll need configuration, thread dumps, etc. to justify reopening this bug.
Comment 8 Filip Hanik 2014-03-21 14:29:01 UTC
I think you may experience a different scenario here, than the actual bug reported.
http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/modules/jdbc-pool/src/main/java/org/apache/tomcat/jdbc/pool/ConnectionPool.java?view=markup

Line
throw new PoolExhaustedException("[" + Thread.currentThread().getName()+"] " +
                        "Timeout: Pool empty. Unable to fetch a connection in " + (maxWait / 1000) +
                        " seconds, none available[size:"+size.get() +"; busy:"+busy.size()+"; idle:"+idle.size()+"; lastwait:"+timetowait+"].");

The statement
  busy.size()
is not just a counter, it actually measures the number of connections that are placed in the busy queue.

Here is what I would do

1. set logAbandoned="true"
//this will print a log entry of places in your code that are leaking connections (ie failing to call Connection.close())

With the log output from 1, we should be able to resolve all your problems.
Comment 9 Anton 2014-07-22 01:36:19 UTC
Created attachment 31838 [details]
Test demonstrating jdbc-pool connection leakage

I wrote a test which demonstrates how connections can be leaked. To execute the test you need MySQL server running on port 3306.
Comment 10 Filip Hanik 2014-07-22 09:56:36 UTC
Created attachment 31840 [details]
Unit test demonstrating fair queue not working properly

This tests correctly fails (assertion fails) when it is supposed to properly show the bug.
Comment 11 Filip Hanik 2014-07-22 09:57:54 UTC
(In reply to Anton from comment #9)
> Created attachment 31838 [details]
> Test demonstrating jdbc-pool connection leakage
> 
> I wrote a test which demonstrates how connections can be leaked. To execute
> the test you need MySQL server running on port 3306.

Anton, thank you for your test. I do confirm the bug you are seeing.
Here is what I got

1. work around is to set poolProperties.setFairQueue(false)
2. I did modify the test to properly fail on assertion to demonstrate the actual bug

I will get this fixed.
Comment 12 Anton 2014-07-22 16:40:46 UTC
Great, thank you!
Comment 13 Filip Hanik 2014-07-24 13:44:24 UTC
Fixed in trunk at r1613123
Comment 14 Filip Hanik 2014-07-24 14:02:13 UTC
Fixed in r1613137 of Tomcat 7 branch
Comment 15 Filip Hanik 2014-07-24 14:02:54 UTC
Flag correction in trunk r1613135
Comment 16 Filip Hanik 2015-06-04 17:12:21 UTC
*** Bug 56660 has been marked as a duplicate of this bug. ***