Apache OpenOffice (AOO) Bugzilla – Issue 1840
website very slow from 11pm PDT for several hours
Last modified: 2003-12-06 14:52:32 UTC
the website www.openoffice.org (login, navigation, IssueZilla) is very slow at German main buisiness hour between 8am until ~12pm. Effectivly the site is not usable at this time. monitoring.collab.net/openoffice/load.html display a higher load than in other times for this period. same you can see for network traffice monitoring.collab.net/openoffice/network.html.
I agree. I have already raised this with Kat... louis
Hi, A request to change the run time of some rebuild scripts to lighten the load has been entered with our operations department. We will kepp you updated on this issue. Thank you Kat
the performance of the cvs server is worse than ever. If cvs performance if related to this issue, I will raise priority to one, otherwise we have to set up a new issue.
times when site is slow have changed, but things get not better. When can we expect that the site performs better ?
Hi, I will check on other operational issues with our ops department and the impact on performance. Thank you Kat
Hi, Ops has reported there is a possibility of increasing the amount of RAM on the server, as well as other script run time changes to increase performance. The earliest any of these options could be considered due to personnel and hardware constraints is at least one week from now. Thank you Kat
I have been reported today by developers that a login last upto 20 minutes, cvs log commands lasts upto 3 minutes. I think those values are not acceptable.
Our operations department would like to try something to reduce the load starting tonight. The idea is to disable cvsupd from 11PM-4AM, pacific time. This is while the backups are running. The reasoning behind this is that cvsupd is a major IO and CPU drain while it is running and they believe it to be causing IO thrashing in the system. Disabling it for a few hours will test this theory and hopefully restore some performance. The downside to this plan is just that the anoncvs box will lag for a few hours during this time. Does this course of action sound acceptable? Thank you Kat
Tracked on internal issue pcn6008
Yes !
Hi Martin, The crontab was changed last night so anoncvs was not synced between 10PM-7AM pacific time. Another script (slocate) which was causing major load on the system, also competing with backups was changed from running daily to weekly. In the process of doing this, it was found that significant drain on the system still existed from a server in Germany running cvsup over an ssh tunnel, every hour sshd 32625 root 4u IPv4 96935313 TCP h66.sny.collab.net:s sh->calzone.stardiv.de:769 The immediate suggestion from our ops department is for OpenOffice to also discontinue cvsup during these hours (10PM-7am), as it is causing drain on the system during the backup window. If the backups can finish sooner because of less contention, we can shorten the window. For example backups for other sites usually take only 3-4 hours. An additional 1G of RAM is also available and can be installed on Thursday, Oct. 25 at 21:30 PDT. This may also help to improve performance of the site. The estimated downtime will be 15 minutes if you choose to go through with this option. Do either, or a combination of these proposals sound acceptable to you? Thank you Kat
I have now reduced our cvsup times to 6.45, 12.45, 18.45 and 23.45 CEST. I'm ok with a downtime Thursday, Oct. 25 at 21:30 PDT for 15 minutes ( I guess the additional RAM has already been tested ;) ).
The downtime slipped to approximately 22:00 PST due to a scheduling conflict. The upgrade lasted for 31 minutes. Thank you Kat
Please see also #2070 (Cricket Data is not up-to-date). Without this we have now feedback during our tests.
there seems to be a problem with cvsup. Every time cvsup is running it tries to do an "SetAttrs" on a set of file (it seems to be every file). The resulting file date then is January 1st 1970. This seems to cause a high load on openoffice.org. We reduced the times of syncing so load got lower. Is the anoncvs server working in a regular way ? if so what is the command line for cvsup you use. we're using Software and protocol version 16.1. then command line is: cvsup -s -1 -d 10 -l LOCK.cvsup -P m -g -L1 supfile
We are upgrading the cvs client and server to 16.1 and will let you know when it has been done. Thank you Kat
> We are upgrading the cvs client and server to 16.1 and ... You're talking about cvsup (not cvs), aren't you? Otherwise we had to veto!
Hi Stefan - Sorry, I mis-typed in the last post. We are upgrading the cvsup on the server to version 16_1e, and would like to request you do the same if not on this version already. Ops feels this may also be causing some of the performance problems we've been seeing recently, as this site explains in more detail. http://people.freebsd.org/~jdp/s1g/
We have upgraded the cvsup client and it looks better now
We have also upgraded the client and server on our end. Kat
Performance reported back to normal on the call - seems cvsup was the root cause. Closing the issue. Kat
As agreed by Louis I will close these resolved fixed kat (support)-owned issues now. If you have trouble with that, please re-open the issue.