Issue 118940 - Tune robots.txt for website
Summary: Tune robots.txt for website
Alias: None
Product: Infrastructure
Classification: Infrastructure
Component: Website general issues (show other issues)
Version: current
Hardware: All All
: P3 Normal (vote)
Target Milestone: ---
Assignee: AOO issues mailing list
QA Contact:
Depends on:
Reported: 2012-02-20 23:06 UTC by Rob Weir
Modified: 2017-05-20 10:45 UTC (History)
1 user (show)

See Also:
Issue Type: TASK
Latest Confirmation in: ---
Developer Difficulty: Easy


Note You need to log in before you can comment on or make changes to this issue.
Description Rob Weir 2012-02-20 23:06:59 UTC
We currently have:

User-agent: daumoa
Disallow: /

User-Agent: Sosospider
Disallow: /

User-Agent: Snoopy
Disallow: /

User-Agent: *
Crawl-delay: 5
Disallow: /nonav/
Disallow: /welcome/
Disallow: /white_papers/

Some of this doesn't make sense.  For example, there is nothing at /nonav, /welcome seems to have a redirect issue that we should just fix.  And /white-papers/ has reasonable content that we should not be hiding from index spiders.
Comment 1 Marcus 2013-09-11 20:53:51 UTC
When we have done our first browser tests for MSIE versions with "" I've updated the robots.txt already. IMHO this task is done and I would close this issue as fixed.
Comment 2 Marcus 2017-05-20 10:45:16 UTC
Reset the assignee to the default "".