This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
We are seeing very very slow download speeds from netbeans.org. I am currently downloading the beta 2 win.exe file from a machine outside our firewall, at between 0.5 and 4K/s, ETA almost 5 hours. I normally see hundreds of K/s from that host (in fact I think I recently timed downloads of NB as part of our performance issues, and was seeing 300K+/s downloads, but I can't find that data now). This has already been independently reported by a user on nbusers, as well as verified by other Sun employees with external accounts. We've just released beta2 today, so this is a very major P1 for us.
Got a call from Jack that the downloads on the site are slow. The site is fast enough but the downloads take a very long time. Starting to look into the issue. Thanks, Karishma-Helpdesk
When I tried at ~7:40am Pacific time from my webbrowser using Bay area located webcache, I got d/l speed 10KB/s, when I tried SCP download on machine in Newark,CA through socks proxy in Newark,CA I got approx. download speed of 16 Mbits/s (yes, sixteen megabits per second) as I got 100 MB large file in 47 seconds to disk on that machine. That leads me to idea to setup special httpd running on different port and servings just downloads. We can redirect big file downloads to new server within a few minutes. This could help with load balancing. Jack, are there any unacceptable consequences of such a solution?
updating status whiteboard and keywords. Thanks, Karishma-Helpdesk
Rudo, sounds possible ... though lets wait and see if there is a simpler fix. Download speed without caches, before the performance problems, was still acceptable.
How does the staticisation handle the download/ dir ? That dir is not under CVS, so perhaps is not staticised ? However most of the content, like these download bits, almost never changes, so might be a good candidate for some kind of caching.
Jack, The download speeds seems to be fine now. There were some processes accessing the download repeatedly. The processes had to be killed and it is working fine now. Can you please check now and confirm the same. Thanks, Karishma-Helpdesk
Thanks - I am seeing ~300K/s from within Sun's firewall, and up to 300K/s from outside. So, postmortem - what happened, and how do we make sure it wont happen again ?
Reducing the priority as the download speed is now fine. Will check with ops regarding the reason and how it could be prevented in the future and will get back to you on this. Thanks, Karishma-Helpdesk
FYI: The download area is also cached; however the file size limit for cache is set to 110 MB. Can you please verify on your end that it is working as intended? 'wget --server-response (url for download)' should help you determine this; if X-Cache is in the headers returned, and it says HIT, then you know it was cached.
Hi Jack Just got off the phone with you,the site seems to be fine,tried accessing most of the tabs and the response was fast,did a cvs checkout and that was quick too.Any way I have asked my engineers to check on this,I tried reaching you back # 42-0602 683 489 and left a voice message.Please feel free to contact support for any clarifications. Thanks Karthik-helpdesk
Karthik, I'm now trying to download file http://www.netbeans.org/download/5_0/beta2/200511141730/netbeans-5_0-beta2-src-ide_sources.zip and even I'm on 1Mbit DSL line I'm getting this file at ~1KB/s. I'll do another test of SCP download and will consider upping the priority again.
The issues is back, please fix it and keep an eye on the server. My home line downloads at 1-2 KB/s. Machine in Santa Clara downloads at 1.8 KB/s. SCP downloads at 2000 KB/s (and it's encrypted data transfer).
Updating on Behalf of Gary Thompson. The download speed in www.netbeans.org is very low,I am trying to download from the site and the download speed clocks 10kb/sec (2.5 hours remaning),could you please look into this ASAP. Regards Karthik-Helpdesk (On behalf of Gary Thompson)
Upping priority again after mid-air collission with comment by Karthik on behalf of Gary Thompson.
ccing Gary Thompson Regards Karthik-helpdesk
Hi Rudolf The engineers have restarted the http,could you please try to download a file greater than 120-130 MB and check the download speed. Regards Karthik-helpdesk
Hi Rudlof We have ordered additional memory for our cache servers. We will be adding this memory on Friday. This should increase the site performance. Regards Karthik-Helpdesk
A quick update. Apparently the Squid server is not known for it's speed when the files are cached in the disk, rather than in-memory. Hence file sizes less than 110 MB which are cached (as per Squid config settings) are taking longer to download. Any file larger than that size 110 MB and above should not take that long to download. That being said, Ops will add more memory to the cache servers by Friday. In the interim, checking if we can set the upper size limit to a low value on the Squid/cache server end so that it does not cache binary files larger than 5/10 MB and directly provides it/them from the download area.
Aniruddha, I've staged one file in /download/delivery (do not want to disclose it's filename here). It's 550 MB in size (577,054,922 B). Download on machine in Santa Clara,CA timed out and retries are at ~1KB/s. Download to my notebook over VPN to Sun and then through Sun's proxy in Europe is at 5.5 KB/s (I'm on 1Mbit DSL). SCP download through socks proxy in Newark,CA gave me new cool throughput 18 Mbits. I treat this as serious performance issue of the Squid server. I built this opionion on top of fact that in the same time I do http download on two machines and it's terribly slow (1-10 KB/s), I can get encrypted data transfer more than 200 times faster. The same machine has time to serve I/O operations and encryption of 18Mbits/s data stream. That said the machine is not stucked with I/O operations and also aggregate processor load is not near to 100%. Thus I think something is wrong in Squid's code or configuration.
Gary Thompson had called to report that the download speeds were extremely slow. Have updated the ops regarding this and they are working on this. Will get back to you with further updates. Thanks, Karishma-Helpdsk
We have made some changes in the cache server and download speeds have improved.I just downloaded the following from Netbeans site NetBeans IDE 5.0 Beta 2 + Application Server 8.1 Bundle Installer sjsas_pe-8_1_02_2005Q2-nb-5_0-beta2-bin-win.exe (100.87 MB) MD5: 5eb4fbfe132a8d73965b63979fc135e0 and I was getting download speeds upto 300kb/sec,please verify from your end and get back to us for more clarifications. Regards Karthik-Helpdesk
Thanks for update. I've checked and seen faster downloads. Machine in Santa Clara,CA got large file at 1.1 MB/s. My home DSL line got the same file at ~20 KB/s (that's better than before and there is huge latency on Above.net' network between UK and US).
Why did this issue just appear ? The cache servers have been in place for some weeks or close to a month now. Beta 2 files are not significantly larger than beta 1 was, and well under 110MB. What changed, what caused this ?
I'm not sure I understand this comment : > Apparently the Squid server is not known for it's speed when the files are > cached in the disk, rather than in-memory. Hence file sizes less than 110 MB > which are cached (as per Squid config settings) are taking longer to download. > Any file larger than that size 110 MB and above should not take that long to > download. So files less than 110MB are cached on disk, not memory ? Only very big files get put in memory? That doesn't sound right ? FWIW the downloads are *not* cached according to wget headers, but I now get good download speed (avg 350K/s), outside Sun firewall : [jak@jem]: ~ $ wget -S http://www.netbeans.org/download/5_0/beta2/200511141730/netbeans-5_0-beta2-bin-windows.exe --11:03:21-- http://www.netbeans.org/download/5_0/beta2/200511141730/netbeans-5_0-beta2-bin-windows.exe => `netbeans-5_0-beta2-bin-windows.exe' Resolving www.netbeans.org... 64.125.133.209 Connecting to www.netbeans.org[64.125.133.209]:80... connected. HTTP request sent, awaiting response... 1 HTTP/1.0 200 OK 2 Date: Fri, 18 Nov 2005 11:03:21 GMT 3 Server: Apache/2.0.47 (Unix) mod_auth_svn/0.1 SVN/0.23.0 mod_jk/1.2.0 mod_auth_mda/2.0 DAV/2 mod_ssl/2.0.47 OpenSSL/0.9.7 4 Vary: Host 5 Last-Modified: Mon, 14 Nov 2005 19:02:36 GMT 6 ETag: "ee7f2f-39880b7-1d3d6b00" 7 Accept-Ranges: bytes 8 Content-Length: 60326071 9 Content-Type: application/octet-stream 10 X-Cache: MISS from cache4.sjc.collab.net 11 Connection: keep-alive 100%[==============================================================>] 60,326,071 315.17K/s ETA 00:00 11:06:10 (350.46 KB/s) - `netbeans-5_0-beta2-bin-windows.exe' saved [60326071/60326071]
Jack, A quick clarification on this issue. The new cache servers which replaced the original cache servers did not have the necessary memory in the boxes. Realizing that CN operations team ordered and installed more memory on the system on Fri, Nov 18, 2005. However, during the period when the memory was low, the caching being turned on for binary files 110 MB and below, caused these downloads to be cached in the proxy servers; hence these went to disk cache(standard swapping) rather than in-memory. As we started getting complaints from netbeans that the download speeds were slow, it was realized that the disk caching from the cache servers (that Squid was handling) was causing the delays. Hence as an interim measure, the max- object size on the cache servers were reduced to a low value so that all the binaries will be served directly from the download(s) area (on the netbeans server). Subsequently, after adding the memories, the cache configuration was turned on back to the max-object size limit of 110 MB. Your test must have taken place before this change happened. I see on my end that this is now cached. ====================================== [asarkar@localhost nb]$ wget -S http://www.netbeans.org/download/5_0/beta2/200511141730/netbeans-5_0-beta2-bin- windows.exe --17:10:29-- http://www.netbeans.org/download/5_0/beta2/200511141730/netbeans- 5_0-beta2-bin-windows.exe => `netbeans-5_0-beta2-bin-windows.exe' Resolving www.netbeans.org... done. Connecting to www.netbeans.org[64.125.133.209]:80... connected. HTTP request sent, awaiting response... 1 HTTP/1.0 200 OK 2 Date: Mon, 21 Nov 2005 23:02:12 GMT 3 Server: Apache/2.0.47 (Unix) mod_auth_svn/0.1 SVN/0.23.0 mod_jk/1.2.0 mod_auth_mda/2.0 DAV/2 mod_ssl/2.0.47 OpenSSL/0.9.7 4 Vary: Host 5 Last-Modified: Mon, 14 Nov 2005 19:02:36 GMT 6 ETag: "ee7f2f-39880b7-1d3d6b00" 7 Accept-Ranges: bytes 8 Content-Length: 60326071 9 Content-Type: application/octet-stream 10 Age: 946 11 X-Cache: HIT from cache5.sjc.collab.net 12 Connection: keep-alive 100%[====================================>] 60,326,071 3.05M/s ETA 00:00 17:10:48 (3.05 MB/s) - `netbeans-5_0-beta2-bin-windows.exe' saved [60326071/60326071] =============== So to answer your earlier questions: 1. Change in the cache servers which had lesser memory had caused this slow download speeds. 2. The size limit of a binary file to be cached is set to 110 MB. This is verified to be an optimum settings per the verifiation and testing on the CN end. Based on the current performance I am reducing the priority of this issue. Please escalate this if you are still seeing slower downloads on your end.
Thanks; I think this is resolved now.
With this evening's release of NB 5.0 RC1 this problem seems to be back. Symptoms are similar, though fluctuate wildly. I am testing from the same account outside Sun's firewall. Downloading the IDE now at 6K/s, ETA over 2hrs. As I write this, speed leaps up to 320-380K/s. However the download has been going for almost 15min at 6K/s, and with an ETA of 2 or 3 hrs most ppl would cancel out of that download. Do the config changes on teh cache servers need to be done again ? Pls investigate urgently. Thanks.
I am getting consistent good speeds (300+K/s) now. So what *was* the problem ? ~2 hours after the files went live we were still seeing very very low, and erratic, transfer speeds.
Hi Jack, We will look into this ASAP! Thanks for reporting. BTW, we were completely surprised by Jidth's e-mail that the download was not working properly even before we saw any issue updates and/or gotten any calls to the helpdesk on this P1.
Accidentally changed the priority after Jack reduced it to P2. Currently Operations engineers are looking into this issue (especially the reported fluctuations in speed).
Apparently the cache servers are not efficient in serving big objects fast enough; hence for larger downloads we have by passed the cache for now. The download speeds are now very fast. Operations team will keep an eye if this indirectly causes the backend server to overload under ceratin load conditions, in which case they will adjust the cache parameters again. The permanent solution will be to have a separate download server in place.
Seperate download server is up; havent' seen this with recent downloads so I think it is fixed.
This was a temporary issue and was fixed. Marking as verified
We recently moved out from Collabnet's infrastructure