Pārlūkot izejas kodu

Block search engines via robots.txt (#631)

Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
minus 4 gadi atpakaļ
vecāks
revīzija
c543a1df8c
1 mainītis faili ar 2 papildinājumiem un 0 dzēšanām
  1. 2 0
      public/robots.txt

+ 2 - 0
public/robots.txt

@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /