robots: host directive to reduce mirror crawling
authorMischa POSLAWSKY <perl@shiar.org>
Tue, 2 May 2017 21:35:02 +0000 (23:35 +0200)
committerMischa POSLAWSKY <perl@shiar.org>
Mon, 29 May 2017 17:23:12 +0000 (19:23 +0200)
Preferred domain to indicate main site for at least Yandex bots:
<https://yandex.com/support/webmaster/controlling-robot/robots-txt.xml#host>

robots.txt

index 4ba4148..15b6ecb 100644 (file)
@@ -2,3 +2,4 @@ User-agent: *
 Disallow: /source/*::*
 
 Sitemap: http://sheet.shiar.nl/sitemap.xml
+Host: sheet.shiar.nl