git.shiar.nl / sheet.git / commitdiff
? search:
summary | shortlog | log | commit | commitdiff | tree
raw | patch | inline | side by side (from parent 1: 77ea108)
robots: disallow crawlers to sources of perl modules
authorMischa POSLAWSKY <perl@shiar.org>
Sun, 22 Nov 2009 06:18:12 +0000 (06:18 +0000)
committerMischa POSLAWSKY <perl@shiar.org>
Wed, 2 Dec 2009 20:58:37 +0000 (20:58 +0000)
As far as they support * in Disallow.
Advantage of indexing all related modules is limited: source is (better)
available at official repository site.

robots.txt patch | blob | history

diff --git a/robots.txt b/robots.txt
index 4bfd238d54d71ad187b19e9cec605ece97f38119..4ba41488a278f873796b611151ab7564588f1ab8 100644 (file)
--- a/robots.txt
+++ b/robots.txt
@@ -1 +1,4 @@
+User-agent: *
+Disallow: /source/*::*
+
 Sitemap: http://sheet.shiar.nl/sitemap.xml
 Sitemap: http://sheet.shiar.nl/sitemap.xml
sheet.shiar.nl - software cheat sheets