This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 43278 - [perf] Deleting folder with files should be made faster
Summary: [perf] Deleting folder with files should be made faster
Alias: None
Product: platform
Classification: Unclassified
Component: -- Other -- (show other bugs)
Version: 4.x
Hardware: PC Windows ME/2000
: P2 blocker (vote)
Assignee: rmatous
: 41154 (view as bug list)
Depends on: 44003
  Show dependency tree
Reported: 2004-05-14 14:58 UTC by Antonin Nebuzelsky
Modified: 2008-12-22 19:39 UTC (History)
4 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:

AIOOBE exception (3.68 KB, text/plain)
2004-05-14 15:00 UTC, Antonin Nebuzelsky
OptimizeIt snapshot of the second part of delete action (when files' nodes disappeared but folder's node is still visible) (49.17 KB, application/octet-stream)
2004-06-08 13:54 UTC, Antonin Nebuzelsky
exception (1.96 KB, text/plain)
2005-07-18 15:29 UTC, Jiri Prox
BTW, here is a snapshot from OptimizeIt 6 (113.71 KB, application/octet-stream)
2005-07-27 11:54 UTC, Antonin Nebuzelsky

Note You need to log in before you can comment on or make changes to this bug.
Description Antonin Nebuzelsky 2004-05-14 14:58:38 UTC
Steps to reproduce:
- copy a new folder with 1000 java files into the
open project (in a external file manager)
- refresh the project in Files view
- expand the newly added folder w/ 1000 javas
- delete the folder w/ 1000 javas

Dell Precision 220, PIII 800MHz, 512MB RAM,
Windows 2000, JDK 1.4.2_04

trunk 200405131800: 6406ms, 6375ms, 6125ms
refactoring 040513: 15750ms, 15031ms, 15109ms

With the refactoring build the contents of the
folder is repainted as deleted in Files view after
approx. 6 seconds but an exception is raised (see
the attachment) and the delete action continues
for next 9 seconds and only then the folder's node
is deleted in the Files view. The whole delete
action thus takes 15 seconds.
Comment 1 Antonin Nebuzelsky 2004-05-14 15:00:05 UTC
Created attachment 14870 [details]
AIOOBE exception
Comment 2 Jan Becicka 2004-05-14 16:07:43 UTC
AIOOBE filed as a separate issue 43280.

Regarding your numbers:
Deleting java file will take longer in refactoring build, because we
must delete also Resource from java metamodel to ensure consistency.

Comment 3 _ ttran 2004-05-14 16:25:54 UTC
suggestion: perhaps you can quickly mark the deleted entries as
invalid and do the real update in the background later.  
Comment 4 Jan Becicka 2004-05-17 08:13:21 UTC
OK. I move metamodel update to background thread.
Comment 5 Jan Becicka 2004-05-26 16:44:42 UTC
Metamodel update moved to background thread.
Tondo, can you provide fresh numbers?

Checking in;
new revision:; previous revision:
Comment 6 Antonin Nebuzelsky 2004-05-28 18:17:57 UTC
The numbers are now affected by a trunk regression (see issue 44003).
Please, keep this bug open until 44003 is fixed and the numbers can be
compared again.
Comment 7 Jan Becicka 2004-06-04 10:43:27 UTC
Tondo, can you compare it again? Thanks
Comment 8 Antonin Nebuzelsky 2004-06-04 14:23:37 UTC
The same configuration and procedure as before. I have done more
measurements this time because the behavior was quite unstable (see
the numbers below). Each invocation of delete on the folder of 1000
java files is listed as two numbers (A/B) where A is the time when the
nodes of the 1000 java files disappeared in the explorer and B is the
time when also the node of the folder disappeared. Both A and B are
measured from the beginning of delete action.

trunk 200406031800: 6828ms/7406ms, 7765ms/12000ms, 7719ms/8531ms,
6484ms/10359ms, 6734ms/6921ms

You can see that especially the very end of the delete action (time B)
varies a lot. This behavior should be investigated in a profiler and
Comment 9 Antonin Nebuzelsky 2004-06-08 13:54:33 UTC
Created attachment 15559 [details]
OptimizeIt snapshot of the second part of delete action (when files' nodes disappeared but folder's node is still visible)
Comment 10 Antonin Nebuzelsky 2004-06-08 13:59:35 UTC
One possible improvement found. Quite a lot time is spent in$RootsListener.processEvent().
Tomas is going to integrate an improvement which minimizes the time
spent in this method for deleted files other than the first one in the
Comment 11 Antonin Nebuzelsky 2004-06-08 14:19:03 UTC
*** Issue 41154 has been marked as a duplicate of this issue. ***
Comment 12 Antonin Nebuzelsky 2004-06-08 14:21:20 UTC
Changing summary of this bug.
Comment 13 Tomas Zezula 2004-06-15 12:50:12 UTC
The trunk regression is fixed.

Checking in api/nbproject/;
/cvs/java/api/nbproject/,v  <--
new revision: 1.4; previous revision: 1.3
Processing log script arguments...
More commits to come...
Checking in api/src/org/netbeans/api/java/classpath/;
new revision: 1.19; previous revision: 1.18
Processing log script arguments...
More commits to come...
Checking in
new revision: 1.4; previous revision: 1.3
Comment 14 Jan Becicka 2004-06-16 11:07:23 UTC
Tondo, please close this bug, or reassign it to proper module. It does
not look like refactoring issue any more. Thanks
Comment 15 Jan Becicka 2004-06-18 06:14:05 UTC
Tomas fixed it. Please reopen this issue if deleting is still slower
than your expactation.
Comment 16 Jiri Prox 2005-07-18 15:19:31 UTC
I tried it for 1800 files on dev build 20050718 and it took several minutes. 
I'm using Linux FC2, on P4 1GB ram. 

I concurently watched the folder in MC and the files was beeing deleted slowly
about 3 in a second, but the cpu usage was only about 50%. During the deleting
it can be worked with IDE but it had long responses. 

When i tried it second time the exception occures:
java.lang.Error: org.openide.loaders.DataObjectNotFoundException:

Please try it also on some performance computer, and if the response time is ok
feel free to close it again.
Comment 17 Jiri Prox 2005-07-18 15:29:03 UTC
Created attachment 23141 [details]
Comment 18 Jan Becicka 2005-07-22 10:32:01 UTC
Can you attach few thread dumps for me to see what is going on during delete
process? Thanks
Comment 19 Jan Becicka 2005-07-22 15:13:47 UTC
This bug is hardly refactoring issue. Tondo, can you reevaluate this issue and
attach fresh performance team opinion? Thanks
Comment 20 Antonin Nebuzelsky 2005-07-26 14:09:14 UTC
With a folder with 1000 files I measured 35s on local filesystem and 69s on my
NFS network home.

OptimizeIt shows that majority of the time is spent in

|- 40% - org.openide.loaders.MultiDataObject$Entry.takeLock()
|- 35% - org.netbeans.modules.masterfs.MasterFileObject.delete()
|- 21% - org.netbeans.modules.masterfs.Delegate$FileLockImpl.releaseLock()

Radku, please have a look on this and try to optimize deleting and the
operations with the locks needed for that.
Comment 21 Antonin Nebuzelsky 2005-07-27 11:54:15 UTC
Created attachment 23323 [details]
BTW, here is a snapshot from OptimizeIt 6
Comment 22 rmatous 2005-09-12 11:28:26 UTC
Locking is slower since 4.1 which affect delete bulk operations For every locked
FileObject is created temporary file which is deleted after releasing of lock.
This is probably the cause. This concept was introduced because of reliability
of locking, preventing from data loss in editor and was approved by DevRev. 

I don't recommend to fix it for 5.0 (definitely not for beta) because of
stability of the whole product. On the other hand I would vote to solve this
issue as a prority issue for next release. 
Comment 23 rmatous 2005-09-12 11:38:41 UTC
I think that lock on folder  could mean locking of the whole subtree, which
would bring semantical API change that we could afford. 
Comment 24 Antonin Nebuzelsky 2005-11-14 10:15:10 UTC
Radku, please plan for the API change soon enough for the next release. For 5.0,
after the discussion we had, I agree we have to waive this issue.
Comment 25 rmatous 2005-11-16 16:21:43 UTC
/cvs/openide/loaders/src/org/openide/loaders/,v  <--  new
revision: 1.41; previous revision: 1.40

/cvs/openide/masterfs/src/org/netbeans/modules/masterfs/,v  <-- 
new revision: 1.20; previous revision: 1.19

 new revision: 1.19; previous revision: 1.18

 new revision: 1.16; previous revision: 1.15

 new revision: 1.15; previous revision: 1.14

 new revision: 1.10; previous revision: 1.9

 new revision: 1.3; previous revision: 1.2

 new revision: 1.8; previous revision: 1.7

 new revision: 1.4; previous revision: 1.3

Although not very nice I hope its fixed.
Jarda, please review.
Comment 26 Antonin Nebuzelsky 2005-11-18 13:39:08 UTC
Verified. It now takes <5s on my local filesystem to delete a folder with 1000
java files.