This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Created attachment 136344 [details] nps snapshot When I open a git project that contains many images (binary files) then Netbeans eats up a lot of CPU for a very long time (many minutes). It doesn't help that I keep those images in ignored folders - both in project settings and in .gitignore. Usually, these are images that are part of a web site's database like pictures of products - they are not in git repo for obvious reasons (they are ignored) but are there in project folders so that the web application works well on my localhost. Symptoms: 1. Just after opening a project the CPU usage goes up to >50% on my dual core machine and stays like that even if I don't do anything and even if I close the project. The only way to stop it is to close Netbeans or wait many minutes. For example, if there are 4 folders with 26803 jpg images all taking 459 MB then Netbeans stops consuming CPU after about 9 minutes. If I move away the image folders from the project then CPU usage goes down to zero within a few seconds after opening the project. 2. During the high CPU load the files that are changed are not highlighted in blue colour on the tabs and in the file tree panel. 3. During the high CPU load I don't notice any excessive disk activity. I don't know if Netbeans is reading all the images or not but if it is then it must be doing it pretty slowly. 4. When I "profile the IDE" during the high CPU load I notice a "GitRefresh" method that seems to operate all the time. Also, when I choose "Git -> Show Changes -> Refresh Statuses" the operation locks up. I've attached profile snapshot in this report. I have found the following actions stop the CPU problem from occuring: 1. Disabling the Git plugin. 2. Moving the large image folders away from the project folder. This problem makes using Netbeans painful for projects that need many binary files in them, which is often the case with online shops, online image databases, etc. The correct behaviour of Netbeans (or the Git plugin) should be to really ignore the files and folders which are ignored but currently this is not the case. Ignored files and folders should have no effect on the time Netbeans/Git scans the project. Ideally, both .gitignore and project's ignored folders should be respected but if at least only one of them is respected then this would be really beneficial. BTW, this problem does not occur in svn projects.
interesting, there is no I/O in the snapshot, all time is spent in java.lang.String.toLowerCase() which is really weird. Please could you make another snapshot running for a longer time (about a minute) ? And also please attach the messages.log (http://wiki.netbeans.org/FaqLogMessagesFile)
btw, i just tried and in my case Git skips ignored files and does not care about them. Are you sure the folder is really ignored? Can you share the content of .gitignore file with me and describe the folder structure a bit?
Created attachment 136345 [details] 1-minute snapshot
Created attachment 136346 [details] messages.log There seem to be nothing new in messages.log when I opened the problematic project but there are some earlier messages.
Created attachment 136347 [details] .gitignore
In .gitignore you can see the last 4 folders (/www/images/produkty/...) are the ones where these images are kept. TortoiseGit recognizes this .gitignore without a problem and the images are ignored there correctly. In Netbeans it makes a lot of difference whether these 4 folders are filled with images or empty - as far as CPU usage is concerned. The Projects pane doesn't list those ignored files but under the hood they seem to be visible by the Git plugin somehow causing the delay. This is a PHP project. I have two of them that exhibit the same problem and in both cases many images in ignored folders seems to be the cause. I don't know how many of them need to be there to become troublesome but anything above 500 MB in quantity of many thousand files cause a very significantly long CPU usage in my case.
> /www/images/produkty/large/* > /www/images/produkty/normal/* > /www/images/produkty/small/* > /www/images/produkty/thumbs/* i guess the ignored images are directly under /www/images/produkty/large (normal, small) i.e. /www/images/produkty/large/image.png, right? That is probably the thing, you're not ignoring their parent folder (/www/images/produkty/large/) but the images themselves. The IDE then gets the information for all those files being ignored instead of just one single message about their parent folder be ignored. Could you please try and change the .gitignore file so it looks as follows? > /www/images/produkty/large/ > /www/images/produkty/normal/ > /www/images/produkty/small/ > /www/images/produkty/thumbs/ restart the IDE and let me know if it helped. This way the IDE will recognize the folders as ignored and will not enter into them and consider everything inside as ignored.
Yes, the changes you suggested in .gitignore stopped the problem from occurring. However, this can be an acceptable workaround in some cases but it's not a solution to the problem because I don't want to ignore the folders, I just want to ignore the files in them. The existence of the folders is often crucial for the application to work, however the files in them may be ignored from version control because they are user-submitted large binary files (and can be treated like database data, which is outside version control, too). Also, my example was the most simplified use case. Sometimes I may want to version certain files in those folders, like documentation files, README.TXT, a php script or javascript file, or an .htaccess file while ignoring the binary files. Then I will not be able to use the workaround you suggest and I'll have to do something like this: /www/images/produkty/large/*.jpg /www/images/produkty/normal/*.jpg /www/images/produkty/small/*.jpg /www/images/produkty/thumbs/*.jpg or: /www/images/produkty/large/* !/www/images/produkty/large/.htaccess /www/images/produkty/normal/* !/www/images/produkty/normal/.htaccess /www/images/produkty/small/* !/www/images/produkty/small/.htaccess /www/images/produkty/thumbs/* !/www/images/produkty/thumbs/.htaccess in such cases I'm back to the same problem again. It seems strange to me that simply ignoring a few thousand files has to take so long. TortoiseGit is able to filter the ignored files in less than a second while the git plugin in Netbeans needs 9 minutes for the same thing. I wouldn't want to give up the power that git provides me with .gitignore configuration just to accommodate one tool (Netbeans) that doesn't cope with it very well. I hope there is a chance for improving the way it works now.
making some performance improvements, you can try it and let me know if it helped. If it's still insufficient, you can completely disable the relevant code (not sure what regressions it may cause however) by running the IDE with -J-Dversioning.git.noignoreindex=true (add this option to etc\netbeans.conf to netbeans_default_options). fix: http://hg.netbeans.org/core-main/rev/95f9dbdf456e
Thanks, sorry for the question but how do I test this fix? I'm not a Netbeans developer and I don't think I'll cope with compiling Netbeans from source code...
Wait a few days, an automatic comment will be posted to this issue with a link to the daily build containing the fix
Ok, thanks.
Integrated into 'main-silver', will be available in build *201306272300* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/95f9dbdf456e User: Ondrej Vrabec <ovrabec@netbeans.org> Log: #231851 - Long high CPU usage on git projects with many images in ignored folders updating the cache index in batch instead of updating one file after the other
Thanks, Ondrej, it's definitely a big improvement - from >9 minutes to 26 seconds when opening this project until Netbeans stops consuming CPU. With the image folders ignored completely (not individual files by *) the project opens in 10 seconds. I have the same project under svn with the same ignore patterns (because I'm migrating to git) and opening the project in Netbeans takes 10 seconds - there seems to be no delay caused by ignoring large numbers of files. With git Netbeans opens the project in 10 seconds, then there is a 3-4 second zero-CPU period and then a 13-second 50% CPU period (taken up by the git plugin, I suppose). This second 50% CPU period seems to be proportional to the number and size of the ignored files. So I think there is still room for improvement (to bring the performance to match the svn plugin) but this fix is very good and perfectly acceptable. I have only tested project opening times because this build seems to have some serious problems of some keys not working (like Enter, Delete, etc.) so I can't use it for real work. -J-Dversioning.git.noignoreindex=true doesn't seem to do anything for me performance wise.
Please disregard my comment about keys not working in the dev build - it turned out to be a different problem after importing key shortcuts from another computer via the zip file. Netbeans created a "NetBeans_hidden" file in the config\Editors\Keybindings folder and this affected both the 7.3.1 and the dev installations (because I imported my settings fro, 7.3.1 to dev). This looks like a bug unrelated to what we are discussing here. After deleting the problematic file the keyboard started to work again as it should.
Netbeans 7.3.1 on my 8 gig dual core i7 64 bit windows 7 machine causes the machine to slow considerably. There's o appreciable I/O, but the CPU usage gets pegged and the fan goes bananas. How do I submit a VisualVM profile as a bug report? I am assuming that this is the proper bug to file under or shall I create a new one? Thanx in adv. -Jake
(In reply to jakemono from comment #16) > Netbeans 7.3.1 The fix for this issue is not part of 7.3.1, so do not reopen this unless you're able to reproduce with 7.4 or a dev build. > How do I submit a VisualVM profile as a bug report? I am assuming that this is > the proper bug to file under or shall I create a new one? Thanx in adv. -Jake http://wiki.netbeans.org/FitnessViaPartnership