Apache OpenOffice (AOO) Bugzilla – Issue 24771
cvs tag commands are bad performing
Last modified: 2009-08-02 14:06:18 UTC
cvs (r)tag commands are performing bad in comparision to usual cvs set up. cvs rtag of sw module last about 20 minutes on openoffice.org, lasts less than 2-3 minutes on a usual 5 year old Sparc machine. the suspicion is that not a local file system is used for cvs operations. our measurements showed that even a 100MBit NFS file system in comparision to 10 MBit is significantly slower than using a local file system. since tag and also diff command are frequently used in OOo development environment we are dependent to fast cvs operations. I assume that diff operation are similar slow as tag operations. mh->hr: can you please also provide measuerement for diff operations ?!
could you also post information on whether the users are using the full path to the cvs repository in their cvs attempts please
I expect most users will use something like cvs -d:pserver:mh@so-cvs-tunnel.germany.sun.com:/shared/data/helm/cvs/repository rtag -r HEAD <mytag> sw and for tag commands the information from CVS/Repository will be used, which maps to sw/sw for the example of the sw - module.
I am not able to replicate this latency. However I've updatd the internal issue with your commentary. I'll update this issue Monday w/ comments from engineering and operations who are both researching this currently.
I've made a few measurements regarding the tag operation which reveals that local disks can have a huge impact on server performance. The tag operation is mostly server bound (low network bandwith required). In all cases I did a checkout from the tested server beforehand. This means that if enough memory is available many (most? all?) CVS archives are probably still cached. I've timed the tagging of the sw module (for local tests taken from our cvsup server). The client was always remote. 1) stone old Ultra Enterprise 3000 (with 6*250 MHz CPUs, 1,5 GBytes memory) Solaris 2.6 CVS server with CVS respository on NFS storage connected via Gigabit-Ethernet. This is the worst scenario which I could come up with. Tag time: 15:27 min 2) CVS server on V880 server with local disks (8*900 MHz CPUs, 16 GBytes memory) Solaris 2.8. This is probably a sensible setup: Tag time: 1:01 min 3) CVS server on Linux, local disks with resiserfs (1.8 MHz Pentium IV, 1 GBytes memory), SuSE 8.2 Tag time: 0:15 min(!) 4) CVS server on the OOo site: Tag time: 20:00 min (and we had no httpd locks during that time) As you can see there huge differences in tag times, depending on the setup. CVS servers with NFS mounted repositories are really bad. The difference between 3) and 4) might be explained by the fact that Solaris UFS is usually operated in synchronous mode whereas Linux reiserfs is usually operated in asynchronous mode. To really exclude any network bandwith influence I just did an additional rtag check with vs. the OO server. Tag time (rtag): 14:02 min My conclusion: CVS servers with NFS mounted repositories are at least one magnitude slower than with the repository on the local disk.
Engineering's recommends that perhaps you need to take some of the same measures other big sites do: separate the committers from the source downloaders from the site itself.
I going to loose patience. cvs read and write access has been separated for OpenOffice.org from the beginning of the project. also other downloads are already separated from the OOo site. We are talking only about authorized committers to the projects. I don't know about the architecture of viewcvs within sourcecast. If this Interface is bound also to the main site, it is up to you to separate these accesses from the cvs repository.
This issue is closed internally are you still having problems with the rtag commands? Engineering has been unable to find any abnormal performance issues related to this.
this problem still persist. is engineering able to confirm or to disprove the data we provided ? Are you able to answer the question we faced up to you ? * Is the cvs repository located on a local drive ? * Are the viewcvs accesses separated from the cvs read/write accesses ?
reassigning to mh to examine the performance post-hardware upgrade.
tag times are now by factor 2 - 2.5 times better, but performance in comparision to the scenario Heiner provide are still worse. please provide answers to questions we put into issue on 2004/02/05.
postupgrade data has been very favorable. we've sent some preliminary performance data to Eric @ Sun. after another week we'll send another report and will move to close out these issues now that things are moving much faster.
This issue will remain open pending further cvs difficulties. Once established in this issue, CN support will immediately pass the information on to the internal engineers assigned to research and investigate any cvs problems on the OOO site.
st/mh: when/if there's a long commit time: 1) please reassign the issue to support@ 2) the filename/directory of the file 3) the time of the attempt 4) copies of the user's errors (if they have cvs output and timestamps it'd be beneficial) 5) any other relevant information once this is done, support will pass this information on to our response engineers who will research the problem immediately
reassigning to Martin...
since february 5th we wait for an answer to these questions: >is engineering able to confirm or to disprove the data we provided ? >Are you able to answer the question we faced up to you ? >* Is the cvs repository located on a local drive ? >* Are the viewcvs accesses separated from the cvs read/write accesses ?
St: (As per the discussion with Drew on call) Can you have the user perform the one "too slow" rtag operation using the cvs "-t" flag: cvs -t -d :pserver:..... rtag .... and let us know the date/time this was performed, so the engineer can check logs for more info. The point of the "-t" in this case is that it also turns on some server-side logs. 1) please reassign the issue to support@ 2) the filename/directory of the file 3) the time of the attempt 4) copies of the user's errors (if they have cvs output and timestamps it'd be beneficial) 5) any other relevant information
reassigning to Martin
I made some more measurement using the module 'test' available on OOo. for the test on the other configurations the same set of cvs data from the cvsup mirror has been used. cvs rtag command using OpenOffice.org pserver via tunnel mh@so-cvsup(127):/home/usr/mh/work/cvstest/tunnel/test> time cvs -d:pserver:mh@so-cvs-tunnel:/shared/data/helm/cvs/repository rtag -b test1 test 0.010u 0.000s 0:28.20 0.0% 0+0k 0+0io 188pf+0w cvs rtag command on local machine with repository on NFS home mh@so-cvsup(150):/home/usr/mh/work/cvstest/local-nfs-home> time cvs rtag -b test1 test cvs rtag: Tagging test 0.050u 0.300s 0:38.17 0.9% 0+0k 0+0io 165pf+0w cvs rtag command on local machine with repository on local file system (Linux) 0.020u 0.130s 0:00.14 107.1% 0+0k 0+0io 841pf+0w Solaris pserver with repository on local disk cvstest/pool1> time cvs rtag test2 test 0.018u 0.007s 0:03.34 0.2% 0+0k 0+0io 246pf+0w Solaris pserver with repository on NFS 0.016u 0.004s 0:29.37 0.0% 0+0k 0+0io 246pf+0w these data again indicates that the usage of NFS for storing the cvs repository causes significant slowdown of the rtag command.
I have updated the internal issue with your data input here. I will keep monitoring the internal issue & update this issue further from here. Eric
Hi Martin, Here is the engineering response. Can OOo (a) do an rtag _with_ "-t" (b) confirm that this particular rtag run is "slow," by whatever definition is being used. (c) provide date, time, and project/directory so tagged Please provide copies of the user's errors (if they have cvs output and timestamps it'd bebeneficial)or any other relevant information. Then we can check the resulting log for some other possible kinds of slowness. Thanks, Eric
Assigning to Martin to gather the requested information. Eric
>Can OOo >(a) do an rtag _with_ "-t" no OOo can't do that since there is no "-t" option available in the cvs rtag command. >(b) confirm that this particular rtag run is "slow," by whatever definition is >being used. n/a > (c) provide date, time, and project/directory so tagged time cvs -d:pserver:mh@so-cvs-tunnel:/shared/data/helm/cvs/repository rtag -b test4 test ;date real 0m19.651s user 0m0.017s sys 0m0.005s Thu Apr 22 12:48:20 CEST 2004 > Please provide copies of the user's errors (if they have cvs output and >timestamps it'd bebeneficial)or any other relevant information. there are no errors, we're complaining that the rtag command execute some magnitudes higher compared to a regular cvspserver installation. Please provide copies of the user's errors (if they have cvs output and timestamps it'd bebeneficial)or any other relevant information.
Hi Martin, I request the customer run a "cvs -t rtag" run, so that I can get the logging info on the server side that I need for this process. <snip> The ordering of the '-t' matters. Try to run this cvs rtag commandwith the -t as above here & give us a copy of the cvs command used & the output. -t is a cvs option like rtag, so this option is available for cvs commands using the rtag option. The -t will give us the tracing information in the log files that our engineering needs to pinpoint any possible problems here. We suggest the customer perform (a), (b) and (c) as previously requested
Created attachment 14721 [details] log of cvs -t rtag command
reassigned to support, see previous attached log file.
Thanks Martin, I have updated the internal issue with your update. Eric
Hi Martin, Analysis of the log files result in the following. These log files definitely show that there was no XML-RPC problem during this particular experiment. If the customer is satisfied that this experiment was one of those that are "too slow" by their definition, then XML-RPC slow-down is not the cause. Other information here, it seems to make it clear that their reports of slowness are not about heavy-load situations, but about the performance of rtag under the best possible circumstances. It also appears that they are _not_ reporting about a performance change over the upgrade, but something that's been true all along. And, finally, their analysis agrees with ours: there is no practical way to improve this for CVS. Sorry, but this issue is beyond our ability to remedy." Eric
Is it possible to get the three questions answered I entered in this issue at Feb. 5th ? Are you sure that the last comment you entered is related to this issue ? We never spoke before about so called "XML-RPC" issue so I have no idea what you are speaking of.
Sorry, We initially thought this had something to do with XML-RPC & thought that may have been the culprit here in the rtag operation slowness, and now we appear to have ruled that out as a source. I will look into getting those 4 questions answered. Eric
Hi Martin, > Is the cvs repository located on a local drive ? It's a mix of local and NFS. NFS is slower than local disk for some operations. Never the less, there is no way at all we can move this storage off NFS. We use NFS for reliability, fail-over, dynamic sizing, and backup. Comparing local to NFS performance is not really relevant, because local storage is not relevant. Customer's storage was on NFS before the upgrade as well. NFS is also used so we can share the data between the SourceCast box, where ViewCVS runs, and the CVS box, where CVS runs too. > Are the viewcvs accesses separated from the cvs read/write accesses ? I/O wise, if someone requests a specific ,v file, it comes from the same place no matter what program requests it. We realize running standalone CVS on a dedicated box using local disks for storage is going to provide the fastest performance, but SourceCast is a lot more than just CVS. The production environment for SourceCast is designed to accomodate all of its needs and as a result takes advantage of both local and networked storage in a way that is both efficient and managable, not only for the Sun sites but for all of our customers. Access to CVS uses the same I/O paths and is not different for viewcvs than it is for other client access. > The information provided has not indicated a loss in performance since the upgrade or that there are any problems here. Please provide us with the feedback for the following: What are the specific performance needs that are not being met currently? How fast do rtag operations have to be on the OpenOffice site for what size repositories (number of files, file sizes) in order to make the environment useful? Eric
We need more information regarding your expectations. There were a few questions asked that have not been addressed, specifically: 1. What are the specific performance needs that are not being met currently? 2. How fast do rtag operations have to be on the OpenOffice site for what size repositories (number of files, file sizes) in order to make the environment useful? Also, were the systems which produced the previously stated metrics under stress anything like the production OOo system, ~15,000 concurrent active user sessions?
to mh for response.
This is a _very_ substantial problem, wrt. the community - it takes _far_ _far_ too long to tag OO.o, this operation being the basis of the child-workspace method of working. ie. before I can commit my changes, I need to do at least 2 cvs tags, of an anchor point, then a branch to do the changes on. Of course; this taking a huge amount of time is really painful in terms of developer productivity; and - worse - encourages people to work outside of CVS with patching mechanisms, which in turn discourages people from working in OO.o CVS / getting their work up-stream, and (presumably) also discorages Sun developers from integrating fixes, since they have to go through this ugly slowness pain.
Furthermore, I see no really good reason why you can't run the CVS server on the same machine as your NFS server. There are _lots_ of different backup architectures that could be used; what possible advantage does NFS give over rsync ? - [ unless perhaps you're going to some BSD box with a snapshotting file-system or something, but I doubt that. ]
Martin, Stefan, I'd like to see about raising the priority on this issue, if you agree. Louis
louis, I don't think it's a matter of priority, the issue is just assigned to me with questions which make me desparate: >We need more information regarding your expectations. There were a few >questions asked that have not been addressed, specifically: >1. What are the specific performance needs that are not being met currently? make cvs as performant as possible, our expectations are described above. >2. How fast do rtag operations have to be on the OpenOffice site for what size >repositories (number of files, file sizes) in order to make the environment >useful? I really don't know what the question is. collab.net should have all information about size repository(number of files, file sizes) as they are hosting the actual repository and can monitor all the actions which are currently bad performing. >Also, were the systems which produced the previously stated metrics under >stress >anything like the production OOo system, ~15,000 concurrent active user >sessions? we are speaking of maximum 100-200 developers which have write access to the cvs repository. If you're saying that the coupling with source cast is the problem then we have to solve this. I expect cvs performing with a cvs repository hosted on a local filesystem and typically no more than 10-20 users silultaneausly accessing it. since this is no data loss issue, I have no problem to stay with P2, but I'd like to see a plan how to resolve this issue in an appropriate time frame.
PB-NOT-DEF: The scope of this issue is in discussion internally by engineering and support. An update will be provided in the next few weeks.
A solution is offered from CollabNet. Currently OO and CN are reviewing this.
reassign issue to myself to look for more investigation for improvement on OOo first.
mark issue as wontfix, CVS is not longer default SCM for OpenOffice.org code.
wontfix -> closed