mirrors
/
rails
mirror of https://github.com/rails/rails
8
0
Fork 0
Code Issues Projects Releases Wiki Activity

In robots.txt, User-agent should be all downcase except for the first 'U', according with http://en.wikipedia.org/wiki/Robots_exclusion_standard

This commit is contained in:
Enrico Carlesso 2012-05-11 11:50:15 +02:00
parent 9b4514c3b8
commit 1abf15657b
2 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
guides/code/getting_started/public/robots.txt
View File

@ -1,5 +1,5 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# User-agent: *
# Disallow: /

2
railties/lib/rails/generators/rails/app/templates/public/robots.txt
View File

@ -1,5 +1,5 @@
# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# User-agent: *
# Disallow: /