Apache OpenOffice (AOO) Bugzilla – Issue 104244
Modify robots.txt to allow Google to index the issue tracker
Last modified: 2017-05-20 10:28:58 UTC
Bugzilla's built-in search lacks PageRank, stemming[1], and many other features. I hear bugzilla 3.0 is faster (usually runs on mod_perl) so performance is no longer such an issue. Would you be so kind as to do an experiment? 1. use the Google Webmaster Tools at https://www.google.com/webmasters/tools to reduce Google's crawl speed on the site to "Slow" 2. add the following to the top of http://bugzilla.mozilla.org/robots.txt: User-agent: Googlebot Allow: /issues/ 3. see if everyone's happier now. ^ [1]. For example, searching for the word "crash" will also find the words "crashes" and "crashing".
Btw have you seen that Mozilla's bugzilla blocks robots too... I changed the setting according to your idea for qa.openoffice.org/issues Let's see if anyone is happier now
From time to time we see some strange crawling activities. But googlebot seems to work fine...