From: Mischa POSLAWSKY <perl@shiar.org>
Date: Sun, 22 Nov 2009 06:18:12 +0000 (+0000)
Subject: robots: disallow crawlers to sources of perl modules
X-Git-Tag: v1.3~82
X-Git-Url: http://git.shiar.nl/sheet.git/commitdiff_plain/706180618dd0dd43d6569359cb35bca84bc5ce51

robots: disallow crawlers to sources of perl modules

As far as they support * in Disallow.
Advantage of indexing all related modules is limited: source is (better)
available at official repository site.
---

diff --git a/robots.txt b/robots.txt
index 4bfd238..4ba4148 100644
--- a/robots.txt
+++ b/robots.txt
@@ -1 +1,4 @@
+User-agent: *
+Disallow: /source/*::*
+
 Sitemap: http://sheet.shiar.nl/sitemap.xml