git.shiar.nl
/
sheet.git
/ commitdiff
commit
grep
author
committer
pickaxe
?
search:
re
summary
|
shortlog
|
log
|
commit
| commitdiff |
tree
raw
|
patch
| inline |
side by side
(parent:
77ea108
)
robots: disallow crawlers to sources of perl modules
author
Mischa POSLAWSKY
<perl@shiar.org>
Sun, 22 Nov 2009 06:18:12 +0000
(06:18 +0000)
committer
Mischa POSLAWSKY
<perl@shiar.org>
Wed, 2 Dec 2009 20:58:37 +0000
(20:58 +0000)
As far as they support * in Disallow.
Advantage of indexing all related modules is limited: source is (better)
available at official repository site.
robots.txt
patch
|
blob
|
history
diff --git
a/robots.txt
b/robots.txt
index 4bfd238d54d71ad187b19e9cec605ece97f38119..4ba41488a278f873796b611151ab7564588f1ab8 100644
(file)
--- a/
robots.txt
+++ b/
robots.txt
@@
-1
+1,4
@@
+User-agent: *
+Disallow: /source/*::*
+
Sitemap: http://sheet.shiar.nl/sitemap.xml