Issue 104244

Summary:	Modify robots.txt to allow Google to index the issue tracker
Product:	Infrastructure	Reporter:	jasonspiro <jasonspiro4>
Component:	Bugzilla	Assignee:	stx123
Status:	CLOSED FIXED	QA Contact:	issues@www <issues>
Severity:	Trivial
Priority:	P3	CC:	issues
Version:	current
Target Milestone:	---
Hardware:	Unknown
OS:	All
Issue Type:	ENHANCEMENT	Latest Confirmation in:	---
Developer Difficulty:	---

Description jasonspiro 2009-08-15 01:25:25 UTC

Bugzilla's built-in search lacks PageRank, stemming[1], and many other features.

I hear bugzilla 3.0 is faster (usually runs on mod_perl) so performance is no
longer such an issue.

Would you be so kind as to do an experiment?

1. use the Google Webmaster Tools at https://www.google.com/webmasters/tools to
reduce Google's crawl speed on the site to "Slow"
2. add the following to the top of http://bugzilla.mozilla.org/robots.txt:
     User-agent: Googlebot
     Allow: /issues/
3. see if everyone's happier now.

^  [1].  For example, searching for the word "crash" will also find the words
"crashes" and "crashing".

Comment 1 stx123 2009-08-21 15:05:01 UTC

Btw have you seen that Mozilla's bugzilla blocks robots too...

I changed the setting according to your idea for
qa.openoffice.org/issues

Let's see if anyone is happier now

Comment 2 stx123 2009-11-24 17:53:22 UTC

From time to time we see some strange crawling activities. But googlebot seems
to work fine...