Bug 54537 - StatementFinalizer closeInvoked is too slow for large batch jobs.
StatementFinalizer closeInvoked is too slow for large batch jobs.
Status: RESOLVED FIXED
Product: Tomcat Modules
Classification: Unclassified
Component: jdbc-pool
unspecified
All All
: P2 critical (vote)
: ---
Assigned To: Tomcat Developers Mailing List
:
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2013-02-07 12:36 UTC by kristo.kuuskull
Modified: 2015-06-23 19:04 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description kristo.kuuskull 2013-02-07 12:36:48 UTC
We had a transaction with 6 million statements. In the end of transaction org.apache.tomcat.jdbc.pool.interceptor.closeInvoked was called and it ran for couple of hours before we terminated it.

The problem is that code below is basically O(n^2) and takes too much time with large amount of statements.

Solution: use more suitable Collection type for statements or change the cycle logic (for i=...).

while (this.statements.size() > 0) {
      WeakReference ws = (WeakReference)this.statements.remove(0)
Comment 1 Filip Hanik 2014-07-28 18:28:19 UTC
Ok, so if we used a linked list, it should be O(n)
Comment 2 Filip Hanik 2014-08-07 21:47:27 UTC
Fixed in r1616602
Comment 3 Rahul Singh 2015-06-23 18:16:10 UTC
Can't we get keep ArrayList as it and instead of removing the object from the list (first line of StatementFinalizer.closeInvoked() method), just get that Statement reference and close that.

At the end of loop when all Statements have been closed, we can just call clear() on the list. It will nullify all the references and those will GCed latter.

Finding a particular entry in a linked list having millions of objects may also have some cost.

With this code method closeInvoked() should look like
public void closeInvoked()
  {
    for (WeakReference ws : this.statements) {
      Statement st = (Statement)ws.get();
      if (st == null) continue;
      try {
        st.close();
      } catch (Exception ignore) {
        if (log.isDebugEnabled())
          log.debug("Unable to closed statement upon connection close.", ignore);
      }
    }
   this.statements.clear();
  }

Hope I am correct!!!
Comment 4 Christopher Schultz 2015-06-23 18:37:41 UTC
(In reply to Rahul Singh from comment #3)
> Can't we get keep ArrayList as it and instead of removing the object from
> the list (first line of StatementFinalizer.closeInvoked() method), just get
> that Statement reference and close that.

Why do you specifically want to use an ArrayList? Why are you commenting on a bug that was closed almost one year ago?

> At the end of loop when all Statements have been closed, we can just call
> clear() on the list. It will nullify all the references and those will GCed
> latter.
> 
> Finding a particular entry in a linked list having millions of objects may
> also have some cost.

When do you need to index into the middle of the list?
Comment 5 Rahul Singh 2015-06-23 18:48:30 UTC
Have no love with ArrayList. Now noticed that it will always remove 0th index so that will not have any cost. I guessed this fix has not been released yet..sorry for commenting on FIXED thread.
Comment 6 Christopher Schultz 2015-06-23 19:04:10 UTC
This fix was released long ago.

It also has the advantage of being able to partially-process a list and remove those items processed. With ArrayList.clear, it's all or nothing.