From 706180618dd0dd43d6569359cb35bca84bc5ce51 Mon Sep 17 00:00:00 2001
From: Mischa POSLAWSKY <perl@shiar.org>
Date: Sun, 22 Nov 2009 06:18:12 +0000
Subject: [PATCH] robots: disallow crawlers to sources of perl modules

As far as they support * in Disallow.
Advantage of indexing all related modules is limited: source is (better)
available at official repository site.
---
 robots.txt | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/robots.txt b/robots.txt
index 4bfd238..4ba4148 100644
--- a/robots.txt
+++ b/robots.txt
@@ -1 +1,4 @@
+User-agent: *
+Disallow: /source/*::*
+
 Sitemap: http://sheet.shiar.nl/sitemap.xml
-- 
2.30.0