From: Mischa POSLAWSKY <perl@shiar.org>
Date: Tue, 2 May 2017 21:35:02 +0000 (+0200)
Subject: robots: host directive to reduce mirror crawling
X-Git-Tag: v1.10~19
X-Git-Url: http://git.shiar.nl/sheet.git/commitdiff_plain/ff54b2f125dada0899ac2b12bf4fa9f496d03762?hp=03275d12a6e5b198a3751318743a6fdd338583ff

robots: host directive to reduce mirror crawling

Preferred domain to indicate main site for at least Yandex bots:
<https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml#host>
---

diff --git a/robots.txt b/robots.txt
index 4ba4148..15b6ecb 100644
--- a/robots.txt
+++ b/robots.txt
@@ -2,3 +2,4 @@ User-agent: *
 Disallow: /source/*::*
 
 Sitemap: http://sheet.shiar.nl/sitemap.xml
+Host: sheet.shiar.nl